[PATCH 01/70] cpufreq: interactive: New 'interactive' governor

[PATCH 01/70] cpufreq: interactive: New 'interactive' governor

[Bálint Czobor]

From: Mike Chan <mike@android.com>

This governor is designed for latency-sensitive workloads, such as
interactive user interfaces.  The interactive governor aims to be
significantly more responsive to ramp CPU quickly up when CPU-intensive
activity begins.

Existing governors sample CPU load at a particular rate, typically
every X ms.  This can lead to under-powering UI threads for the period of
time during which the user begins interacting with a previously-idle system
until the next sample period happens.

The 'interactive' governor uses a different approach. Instead of sampling
the CPU at a specified rate, the governor will check whether to scale the
CPU frequency up soon after coming out of idle.  When the CPU comes out of
idle, a timer is configured to fire within 1-2 ticks.  If the CPU is very
busy from exiting idle to when the timer fires then we assume the CPU is
underpowered and ramp to MAX speed.

If the CPU was not sufficiently busy to immediately ramp to MAX speed, then
the governor evaluates the CPU load since the last speed adjustment,
choosing the highest value between that longer-term load or the short-term
load since idle exit to determine the CPU speed to ramp to.

A realtime thread is used for scaling up, giving the remaining tasks the
CPU performance benefit, unlike existing governors which are more likely to
schedule rampup work to occur after your performance starved tasks have
completed.

The tuneables for this governor are:
/sys/devices/system/cpu/cpufreq/interactive/min_sample_time:
	The minimum amount of time to spend at the current frequency before
	ramping down. This is to ensure that the governor has seen enough
	historic CPU load data to determine the appropriate workload.
	Default is 80000 uS.
/sys/devices/system/cpu/cpufreq/interactive/go_maxspeed_load
	The CPU load at which to ramp to max speed.  Default is 85.

Change-Id: Ib2b362607c62f7c56d35f44a9ef3280f98c17585
Signed-off-by: Mike Chan <mike@android.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Bug: 3152864
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 Documentation/cpu-freq/governors.txt  |   37 ++
 drivers/cpufreq/Kconfig               |   27 ++
 drivers/cpufreq/Makefile              |    1 +
 drivers/cpufreq/cpufreq_interactive.c |  706 +++++++++++++++++++++++++++++++++
 include/linux/cpufreq.h               |    3 +
 5 files changed, 774 insertions(+)
 create mode 100644 drivers/cpufreq/cpufreq_interactive.c

diff --git a/Documentation/cpu-freq/governors.txt b/Documentation/cpu-freq/governors.txt
index c15aa75..b262c53 100644
--- a/Documentation/cpu-freq/governors.txt
+++ b/Documentation/cpu-freq/governors.txt
@@ -28,6 +28,7 @@ Contents:
 2.3  Userspace
 2.4  Ondemand
 2.5  Conservative
+2.6  Interactive
 
 3.   The Governor Interface in the CPUfreq Core
 
@@ -218,6 +219,42 @@ a decision on when to decrease the frequency while running in any
 speed. Load for frequency increase is still evaluated every
 sampling rate.
 
+2.6 Interactive
+---------------
+
+The CPUfreq governor "interactive" is designed for latency-sensitive,
+interactive workloads. This governor sets the CPU speed depending on
+usage, similar to "ondemand" and "conservative" governors.  However,
+the governor is more aggressive about scaling the CPU speed up in
+response to CPU-intensive activity.
+
+Sampling the CPU load every X ms can lead to under-powering the CPU
+for X ms, leading to dropped frames, stuttering UI, etc.  Instead of
+sampling the cpu at a specified rate, the interactive governor will
+check whether to scale the cpu frequency up soon after coming out of
+idle.  When the cpu comes out of idle, a timer is configured to fire
+within 1-2 ticks.  If the cpu is very busy between exiting idle and
+when the timer fires then we assume the cpu is underpowered and ramp
+to MAX speed.
+
+If the cpu was not sufficiently busy to immediately ramp to MAX speed,
+then governor evaluates the cpu load since the last speed adjustment,
+choosing the highest value between that longer-term load or the
+short-term load since idle exit to determine the cpu speed to ramp to.
+
+The tuneable values for this governor are:
+
+min_sample_time: The minimum amount of time to spend at the current
+frequency before ramping down. This is to ensure that the governor has
+seen enough historic cpu load data to determine the appropriate
+workload.  Default is 80000 uS.
+
+go_maxspeed_load: The CPU load at which to ramp to max speed.  Default
+is 85.
+
+timer_rate: Sample rate for reevaluating cpu load when the system is
+not idle.  Default is 30000 uS.
+
 3. The Governor Interface in the CPUfreq Core
 =============================================
 
diff --git a/drivers/cpufreq/Kconfig b/drivers/cpufreq/Kconfig
index 659879a..6e099e5 100644
--- a/drivers/cpufreq/Kconfig
+++ b/drivers/cpufreq/Kconfig
@@ -102,6 +102,16 @@ config CPU_FREQ_DEFAULT_GOV_CONSERVATIVE
 	  Be aware that not all cpufreq drivers support the conservative
 	  governor. If unsure have a look at the help section of the
 	  driver. Fallback governor will be the performance governor.
+
+config CPU_FREQ_DEFAULT_GOV_INTERACTIVE
+	bool "interactive"
+	select CPU_FREQ_GOV_INTERACTIVE
+	help
+	  Use the CPUFreq governor 'interactive' as default. This allows
+	  you to get a full dynamic cpu frequency capable system by simply
+	  loading your cpufreq low-level hardware driver, using the
+	  'interactive' governor for latency-sensitive workloads.
+
 endchoice
 
 config CPU_FREQ_GOV_PERFORMANCE
@@ -159,6 +169,23 @@ config CPU_FREQ_GOV_ONDEMAND
 
 	  If in doubt, say N.
 
+config CPU_FREQ_GOV_INTERACTIVE
+	tristate "'interactive' cpufreq policy governor"
+	help
+	  'interactive' - This driver adds a dynamic cpufreq policy governor
+	  designed for latency-sensitive workloads.
+
+	  This governor attempts to reduce the latency of clock
+	  increases so that the system is more responsive to
+	  interactive workloads.
+
+	  To compile this driver as a module, choose M here: the
+	  module will be called cpufreq_interactive.
+
+	  For details, take a look at linux/Documentation/cpu-freq.
+
+	  If in doubt, say N.
+
 config CPU_FREQ_GOV_CONSERVATIVE
 	tristate "'conservative' cpufreq governor"
 	depends on CPU_FREQ
diff --git a/drivers/cpufreq/Makefile b/drivers/cpufreq/Makefile
index 4134038..98667c8 100644
--- a/drivers/cpufreq/Makefile
+++ b/drivers/cpufreq/Makefile
@@ -11,6 +11,7 @@ obj-$(CONFIG_CPU_FREQ_GOV_POWERSAVE)	+= cpufreq_powersave.o
 obj-$(CONFIG_CPU_FREQ_GOV_USERSPACE)	+= cpufreq_userspace.o
 obj-$(CONFIG_CPU_FREQ_GOV_ONDEMAND)	+= cpufreq_ondemand.o
 obj-$(CONFIG_CPU_FREQ_GOV_CONSERVATIVE)	+= cpufreq_conservative.o
+obj-$(CONFIG_CPU_FREQ_GOV_INTERACTIVE)	+= cpufreq_interactive.o
 obj-$(CONFIG_CPU_FREQ_GOV_COMMON)		+= cpufreq_governor.o
 
 obj-$(CONFIG_CPUFREQ_DT)		+= cpufreq-dt.o
diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
new file mode 100644
index 0000000..8fc147c
--- /dev/null
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -0,0 +1,706 @@
+/*
+ * drivers/cpufreq/cpufreq_interactive.c
+ *
+ * Copyright (C) 2010 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * Author: Mike Chan (mike@android.com)
+ *
+ */
+
+#include <linux/cpu.h>
+#include <linux/cpumask.h>
+#include <linux/cpufreq.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/sched.h>
+#include <linux/sched/rt.h>
+#include <linux/tick.h>
+#include <linux/time.h>
+#include <linux/timer.h>
+#include <linux/workqueue.h>
+#include <linux/kthread.h>
+#include <linux/mutex.h>
+
+#include <asm/cputime.h>
+
+static atomic_t active_count = ATOMIC_INIT(0);
+
+struct cpufreq_interactive_cpuinfo {
+	struct timer_list cpu_timer;
+	int timer_idlecancel;
+	u64 time_in_idle;
+	u64 idle_exit_time;
+	u64 timer_run_time;
+	int idling;
+	u64 freq_change_time;
+	u64 freq_change_time_in_idle;
+	struct cpufreq_policy *policy;
+	struct cpufreq_frequency_table *freq_table;
+	unsigned int target_freq;
+	int governor_enabled;
+};
+
+static DEFINE_PER_CPU(struct cpufreq_interactive_cpuinfo, cpuinfo);
+
+/* Workqueues handle frequency scaling */
+static struct task_struct *up_task;
+static struct workqueue_struct *down_wq;
+static struct work_struct freq_scale_down_work;
+static cpumask_t up_cpumask;
+static spinlock_t up_cpumask_lock;
+static cpumask_t down_cpumask;
+static spinlock_t down_cpumask_lock;
+static struct mutex set_speed_lock;
+
+/* Hi speed to bump to from lo speed when load burst (default max) */
+static u64 hispeed_freq;
+
+/* Go to hi speed when CPU load at or above this value. */
+#define DEFAULT_GO_HISPEED_LOAD 95
+static unsigned long go_hispeed_load;
+
+/*
+ * The minimum amount of time to spend at a frequency before we can ramp down.
+ */
+#define DEFAULT_MIN_SAMPLE_TIME 20 * USEC_PER_MSEC
+static unsigned long min_sample_time;
+
+/*
+ * The sample rate of the timer used to increase frequency
+ */
+#define DEFAULT_TIMER_RATE 20 * USEC_PER_MSEC
+static unsigned long timer_rate;
+
+static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
+		unsigned int event);
+
+#ifndef CONFIG_CPU_FREQ_DEFAULT_GOV_INTERACTIVE
+static
+#endif
+struct cpufreq_governor cpufreq_gov_interactive = {
+	.name = "interactive",
+	.governor = cpufreq_governor_interactive,
+	.max_transition_latency = 10000000,
+	.owner = THIS_MODULE,
+};
+
+static void cpufreq_interactive_timer(unsigned long data)
+{
+	unsigned int delta_idle;
+	unsigned int delta_time;
+	int cpu_load;
+	int load_since_change;
+	u64 time_in_idle;
+	u64 idle_exit_time;
+	struct cpufreq_interactive_cpuinfo *pcpu =
+		&per_cpu(cpuinfo, data);
+	u64 now_idle;
+	unsigned int new_freq;
+	unsigned int index;
+	unsigned long flags;
+
+	smp_rmb();
+
+	if (!pcpu->governor_enabled)
+		goto exit;
+
+	/*
+	 * Once pcpu->timer_run_time is updated to >= pcpu->idle_exit_time,
+	 * this lets idle exit know the current idle time sample has
+	 * been processed, and idle exit can generate a new sample and
+	 * re-arm the timer.  This prevents a concurrent idle
+	 * exit on that CPU from writing a new set of info at the same time
+	 * the timer function runs (the timer function can't use that info
+	 * until more time passes).
+	 */
+	time_in_idle = pcpu->time_in_idle;
+	idle_exit_time = pcpu->idle_exit_time;
+	now_idle = get_cpu_idle_time_us(data, &pcpu->timer_run_time);
+	smp_wmb();
+
+	/* If we raced with cancelling a timer, skip. */
+	if (!idle_exit_time)
+		goto exit;
+
+	delta_idle = (unsigned int)(now_idle - time_in_idle);
+	delta_time = (unsigned int)(pcpu->timer_run_time - idle_exit_time);
+
+	/*
+	 * If timer ran less than 1ms after short-term sample started, retry.
+	 */
+	if (delta_time < 1000)
+		goto rearm;
+
+	if (delta_idle > delta_time)
+		cpu_load = 0;
+	else
+		cpu_load = 100 * (delta_time - delta_idle) / delta_time;
+
+	delta_idle = (unsigned int)(now_idle - pcpu->freq_change_time_in_idle);
+	delta_time = (unsigned int)(pcpu->timer_run_time - pcpu->freq_change_time);
+
+	if ((delta_time == 0) || (delta_idle > delta_time))
+		load_since_change = 0;
+	else
+		load_since_change =
+			100 * (delta_time - delta_idle) / delta_time;
+
+	/*
+	 * Choose greater of short-term load (since last idle timer
+	 * started or timer function re-armed itself) or long-term load
+	 * (since last frequency change).
+	 */
+	if (load_since_change > cpu_load)
+		cpu_load = load_since_change;
+
+	if (cpu_load >= go_hispeed_load) {
+		if (pcpu->policy->cur == pcpu->policy->min)
+			new_freq = hispeed_freq;
+		else
+			new_freq = pcpu->policy->max * cpu_load / 100;
+	} else {
+		new_freq = pcpu->policy->cur * cpu_load / 100;
+	}
+
+	if (cpufreq_frequency_table_target(pcpu->policy, pcpu->freq_table,
+					   new_freq, CPUFREQ_RELATION_H,
+					   &index)) {
+		pr_warn_once("timer %d: cpufreq_frequency_table_target error\n",
+			     (int) data);
+		goto rearm;
+	}
+
+	new_freq = pcpu->freq_table[index].frequency;
+
+	if (pcpu->target_freq == new_freq)
+		goto rearm_if_notmax;
+
+	/*
+	 * Do not scale down unless we have been at this frequency for the
+	 * minimum sample time.
+	 */
+	if (new_freq < pcpu->target_freq) {
+		if (pcpu->timer_run_time - pcpu->freq_change_time
+		    < min_sample_time)
+			goto rearm;
+	}
+
+	if (new_freq < pcpu->target_freq) {
+		pcpu->target_freq = new_freq;
+		spin_lock_irqsave(&down_cpumask_lock, flags);
+		cpumask_set_cpu(data, &down_cpumask);
+		spin_unlock_irqrestore(&down_cpumask_lock, flags);
+		queue_work(down_wq, &freq_scale_down_work);
+	} else {
+		pcpu->target_freq = new_freq;
+		spin_lock_irqsave(&up_cpumask_lock, flags);
+		cpumask_set_cpu(data, &up_cpumask);
+		spin_unlock_irqrestore(&up_cpumask_lock, flags);
+		wake_up_process(up_task);
+	}
+
+rearm_if_notmax:
+	/*
+	 * Already set max speed and don't see a need to change that,
+	 * wait until next idle to re-evaluate, don't need timer.
+	 */
+	if (pcpu->target_freq == pcpu->policy->max)
+		goto exit;
+
+rearm:
+	if (!timer_pending(&pcpu->cpu_timer)) {
+		/*
+		 * If already at min: if that CPU is idle, don't set timer.
+		 * Else cancel the timer if that CPU goes idle.  We don't
+		 * need to re-evaluate speed until the next idle exit.
+		 */
+		if (pcpu->target_freq == pcpu->policy->min) {
+			smp_rmb();
+
+			if (pcpu->idling)
+				goto exit;
+
+			pcpu->timer_idlecancel = 1;
+		}
+
+		pcpu->time_in_idle = get_cpu_idle_time_us(
+			data, &pcpu->idle_exit_time);
+		mod_timer(&pcpu->cpu_timer,
+			  jiffies + usecs_to_jiffies(timer_rate));
+	}
+
+exit:
+	return;
+}
+
+static void cpufreq_interactive_idle_start(void)
+{
+	struct cpufreq_interactive_cpuinfo *pcpu =
+		&per_cpu(cpuinfo, smp_processor_id());
+	int pending;
+
+	if (!pcpu->governor_enabled)
+		return;
+
+	pcpu->idling = 1;
+	smp_wmb();
+	pending = timer_pending(&pcpu->cpu_timer);
+
+	if (pcpu->target_freq != pcpu->policy->min) {
+#ifdef CONFIG_SMP
+		/*
+		 * Entering idle while not at lowest speed.  On some
+		 * platforms this can hold the other CPU(s) at that speed
+		 * even though the CPU is idle. Set a timer to re-evaluate
+		 * speed so this idle CPU doesn't hold the other CPUs above
+		 * min indefinitely.  This should probably be a quirk of
+		 * the CPUFreq driver.
+		 */
+		if (!pending) {
+			pcpu->time_in_idle = get_cpu_idle_time_us(
+				smp_processor_id(), &pcpu->idle_exit_time);
+			pcpu->timer_idlecancel = 0;
+			mod_timer(&pcpu->cpu_timer,
+				  jiffies + usecs_to_jiffies(timer_rate));
+		}
+#endif
+	} else {
+		/*
+		 * If at min speed and entering idle after load has
+		 * already been evaluated, and a timer has been set just in
+		 * case the CPU suddenly goes busy, cancel that timer.  The
+		 * CPU didn't go busy; we'll recheck things upon idle exit.
+		 */
+		if (pending && pcpu->timer_idlecancel) {
+			del_timer(&pcpu->cpu_timer);
+			/*
+			 * Ensure last timer run time is after current idle
+			 * sample start time, so next idle exit will always
+			 * start a new idle sampling period.
+			 */
+			pcpu->idle_exit_time = 0;
+			pcpu->timer_idlecancel = 0;
+		}
+	}
+
+}
+
+static void cpufreq_interactive_idle_end(void)
+{
+	struct cpufreq_interactive_cpuinfo *pcpu =
+		&per_cpu(cpuinfo, smp_processor_id());
+
+	pcpu->idling = 0;
+	smp_wmb();
+
+	/*
+	 * Arm the timer for 1-2 ticks later if not already, and if the timer
+	 * function has already processed the previous load sampling
+	 * interval.  (If the timer is not pending but has not processed
+	 * the previous interval, it is probably racing with us on another
+	 * CPU.  Let it compute load based on the previous sample and then
+	 * re-arm the timer for another interval when it's done, rather
+	 * than updating the interval start time to be "now", which doesn't
+	 * give the timer function enough time to make a decision on this
+	 * run.)
+	 */
+	if (timer_pending(&pcpu->cpu_timer) == 0 &&
+	    pcpu->timer_run_time >= pcpu->idle_exit_time &&
+	    pcpu->governor_enabled) {
+		pcpu->time_in_idle =
+			get_cpu_idle_time_us(smp_processor_id(),
+					     &pcpu->idle_exit_time);
+		pcpu->timer_idlecancel = 0;
+		mod_timer(&pcpu->cpu_timer,
+			  jiffies + usecs_to_jiffies(timer_rate));
+	}
+
+}
+
+static int cpufreq_interactive_up_task(void *data)
+{
+	unsigned int cpu;
+	cpumask_t tmp_mask;
+	unsigned long flags;
+	struct cpufreq_interactive_cpuinfo *pcpu;
+
+	while (1) {
+		set_current_state(TASK_INTERRUPTIBLE);
+		spin_lock_irqsave(&up_cpumask_lock, flags);
+
+		if (cpumask_empty(&up_cpumask)) {
+			spin_unlock_irqrestore(&up_cpumask_lock, flags);
+			schedule();
+
+			if (kthread_should_stop())
+				break;
+
+			spin_lock_irqsave(&up_cpumask_lock, flags);
+		}
+
+		set_current_state(TASK_RUNNING);
+		tmp_mask = up_cpumask;
+		cpumask_clear(&up_cpumask);
+		spin_unlock_irqrestore(&up_cpumask_lock, flags);
+
+		for_each_cpu(cpu, &tmp_mask) {
+			unsigned int j;
+			unsigned int max_freq = 0;
+
+			pcpu = &per_cpu(cpuinfo, cpu);
+			smp_rmb();
+
+			if (!pcpu->governor_enabled)
+				continue;
+
+			mutex_lock(&set_speed_lock);
+
+			for_each_cpu(j, pcpu->policy->cpus) {
+				struct cpufreq_interactive_cpuinfo *pjcpu =
+					&per_cpu(cpuinfo, j);
+
+				if (pjcpu->target_freq > max_freq)
+					max_freq = pjcpu->target_freq;
+			}
+
+			if (max_freq != pcpu->policy->cur)
+				__cpufreq_driver_target(pcpu->policy,
+							max_freq,
+							CPUFREQ_RELATION_H);
+			mutex_unlock(&set_speed_lock);
+
+			pcpu->freq_change_time_in_idle =
+				get_cpu_idle_time_us(cpu,
+						     &pcpu->freq_change_time);
+		}
+	}
+
+	return 0;
+}
+
+static void cpufreq_interactive_freq_down(struct work_struct *work)
+{
+	unsigned int cpu;
+	cpumask_t tmp_mask;
+	unsigned long flags;
+	struct cpufreq_interactive_cpuinfo *pcpu;
+
+	spin_lock_irqsave(&down_cpumask_lock, flags);
+	tmp_mask = down_cpumask;
+	cpumask_clear(&down_cpumask);
+	spin_unlock_irqrestore(&down_cpumask_lock, flags);
+
+	for_each_cpu(cpu, &tmp_mask) {
+		unsigned int j;
+		unsigned int max_freq = 0;
+
+		pcpu = &per_cpu(cpuinfo, cpu);
+		smp_rmb();
+
+		if (!pcpu->governor_enabled)
+			continue;
+
+		mutex_lock(&set_speed_lock);
+
+		for_each_cpu(j, pcpu->policy->cpus) {
+			struct cpufreq_interactive_cpuinfo *pjcpu =
+				&per_cpu(cpuinfo, j);
+
+			if (pjcpu->target_freq > max_freq)
+				max_freq = pjcpu->target_freq;
+		}
+
+		if (max_freq != pcpu->policy->cur)
+			__cpufreq_driver_target(pcpu->policy, max_freq,
+						CPUFREQ_RELATION_H);
+
+		mutex_unlock(&set_speed_lock);
+		pcpu->freq_change_time_in_idle =
+			get_cpu_idle_time_us(cpu,
+					     &pcpu->freq_change_time);
+	}
+}
+
+static ssize_t show_hispeed_freq(struct kobject *kobj,
+				 struct attribute *attr, char *buf)
+{
+	return sprintf(buf, "%llu\n", hispeed_freq);
+}
+
+static ssize_t store_hispeed_freq(struct kobject *kobj,
+				  struct attribute *attr, const char *buf,
+				  size_t count)
+{
+	int ret;
+	u64 val;
+
+	ret = strict_strtoull(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+	hispeed_freq = val;
+	return count;
+}
+
+static struct global_attr hispeed_freq_attr = __ATTR(hispeed_freq, 0644,
+		show_hispeed_freq, store_hispeed_freq);
+
+
+static ssize_t show_go_hispeed_load(struct kobject *kobj,
+				     struct attribute *attr, char *buf)
+{
+	return sprintf(buf, "%lu\n", go_hispeed_load);
+}
+
+static ssize_t store_go_hispeed_load(struct kobject *kobj,
+			struct attribute *attr, const char *buf, size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = strict_strtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+	go_hispeed_load = val;
+	return count;
+}
+
+static struct global_attr go_hispeed_load_attr = __ATTR(go_hispeed_load, 0644,
+		show_go_hispeed_load, store_go_hispeed_load);
+
+static ssize_t show_min_sample_time(struct kobject *kobj,
+				struct attribute *attr, char *buf)
+{
+	return sprintf(buf, "%lu\n", min_sample_time);
+}
+
+static ssize_t store_min_sample_time(struct kobject *kobj,
+			struct attribute *attr, const char *buf, size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = strict_strtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+	min_sample_time = val;
+	return count;
+}
+
+static struct global_attr min_sample_time_attr = __ATTR(min_sample_time, 0644,
+		show_min_sample_time, store_min_sample_time);
+
+static ssize_t show_timer_rate(struct kobject *kobj,
+			struct attribute *attr, char *buf)
+{
+	return sprintf(buf, "%lu\n", timer_rate);
+}
+
+static ssize_t store_timer_rate(struct kobject *kobj,
+			struct attribute *attr, const char *buf, size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = strict_strtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+	timer_rate = val;
+	return count;
+}
+
+static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644,
+		show_timer_rate, store_timer_rate);
+
+static struct attribute *interactive_attributes[] = {
+	&hispeed_freq_attr.attr,
+	&go_hispeed_load_attr.attr,
+	&min_sample_time_attr.attr,
+	&timer_rate_attr.attr,
+	NULL,
+};
+
+static struct attribute_group interactive_attr_group = {
+	.attrs = interactive_attributes,
+	.name = "interactive",
+};
+
+static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
+		unsigned int event)
+{
+	int rc;
+	unsigned int j;
+	struct cpufreq_interactive_cpuinfo *pcpu;
+	struct cpufreq_frequency_table *freq_table;
+
+	switch (event) {
+	case CPUFREQ_GOV_START:
+		if (!cpu_online(policy->cpu))
+			return -EINVAL;
+
+		freq_table =
+			cpufreq_frequency_get_table(policy->cpu);
+
+		for_each_cpu(j, policy->cpus) {
+			pcpu = &per_cpu(cpuinfo, j);
+			pcpu->policy = policy;
+			pcpu->target_freq = policy->cur;
+			pcpu->freq_table = freq_table;
+			pcpu->freq_change_time_in_idle =
+				get_cpu_idle_time_us(j,
+					     &pcpu->freq_change_time);
+			pcpu->governor_enabled = 1;
+			smp_wmb();
+		}
+
+		if (!hispeed_freq)
+			hispeed_freq = policy->max;
+
+		/*
+		 * Do not register the idle hook and create sysfs
+		 * entries if we have already done so.
+		 */
+		if (atomic_inc_return(&active_count) > 1)
+			return 0;
+
+		rc = sysfs_create_group(cpufreq_global_kobject,
+				&interactive_attr_group);
+		if (rc)
+			return rc;
+
+		break;
+
+	case CPUFREQ_GOV_STOP:
+		for_each_cpu(j, policy->cpus) {
+			pcpu = &per_cpu(cpuinfo, j);
+			pcpu->governor_enabled = 0;
+			smp_wmb();
+			del_timer_sync(&pcpu->cpu_timer);
+
+			/*
+			 * Reset idle exit time since we may cancel the timer
+			 * before it can run after the last idle exit time,
+			 * to avoid tripping the check in idle exit for a timer
+			 * that is trying to run.
+			 */
+			pcpu->idle_exit_time = 0;
+		}
+
+		flush_work(&freq_scale_down_work);
+		if (atomic_dec_return(&active_count) > 0)
+			return 0;
+
+		sysfs_remove_group(cpufreq_global_kobject,
+				&interactive_attr_group);
+
+		break;
+
+	case CPUFREQ_GOV_LIMITS:
+		if (policy->max < policy->cur)
+			__cpufreq_driver_target(policy,
+					policy->max, CPUFREQ_RELATION_H);
+		else if (policy->min > policy->cur)
+			__cpufreq_driver_target(policy,
+					policy->min, CPUFREQ_RELATION_L);
+		break;
+	}
+	return 0;
+}
+
+static int cpufreq_interactive_idle_notifier(struct notifier_block *nb,
+					     unsigned long val,
+					     void *data)
+{
+	switch (val) {
+	case IDLE_START:
+		cpufreq_interactive_idle_start();
+		break;
+	case IDLE_END:
+		cpufreq_interactive_idle_end();
+		break;
+	}
+
+	return 0;
+}
+
+static struct notifier_block cpufreq_interactive_idle_nb = {
+	.notifier_call = cpufreq_interactive_idle_notifier,
+};
+
+static int __init cpufreq_interactive_init(void)
+{
+	unsigned int i;
+	struct cpufreq_interactive_cpuinfo *pcpu;
+	struct sched_param param = { .sched_priority = MAX_RT_PRIO-1 };
+
+	go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;
+	min_sample_time = DEFAULT_MIN_SAMPLE_TIME;
+	timer_rate = DEFAULT_TIMER_RATE;
+
+	/* Initalize per-cpu timers */
+	for_each_possible_cpu(i) {
+		pcpu = &per_cpu(cpuinfo, i);
+		init_timer(&pcpu->cpu_timer);
+		pcpu->cpu_timer.function = cpufreq_interactive_timer;
+		pcpu->cpu_timer.data = i;
+	}
+
+	up_task = kthread_create(cpufreq_interactive_up_task, NULL,
+				 "kinteractiveup");
+	if (IS_ERR(up_task))
+		return PTR_ERR(up_task);
+
+	sched_setscheduler_nocheck(up_task, SCHED_FIFO, &param);
+	get_task_struct(up_task);
+
+	/* No rescuer thread, bind to CPU queuing the work for possibly
+	   warm cache (probably doesn't matter much). */
+	down_wq = alloc_workqueue("knteractive_down", 0, 1);
+
+	if (!down_wq)
+		goto err_freeuptask;
+
+	INIT_WORK(&freq_scale_down_work,
+		  cpufreq_interactive_freq_down);
+
+	spin_lock_init(&up_cpumask_lock);
+	spin_lock_init(&down_cpumask_lock);
+	mutex_init(&set_speed_lock);
+
+	idle_notifier_register(&cpufreq_interactive_idle_nb);
+
+	return cpufreq_register_governor(&cpufreq_gov_interactive);
+
+err_freeuptask:
+	put_task_struct(up_task);
+	return -ENOMEM;
+}
+
+#ifdef CONFIG_CPU_FREQ_DEFAULT_GOV_INTERACTIVE
+fs_initcall(cpufreq_interactive_init);
+#else
+module_init(cpufreq_interactive_init);
+#endif
+
+static void __exit cpufreq_interactive_exit(void)
+{
+	cpufreq_unregister_governor(&cpufreq_gov_interactive);
+	kthread_stop(up_task);
+	put_task_struct(up_task);
+	destroy_workqueue(down_wq);
+}
+
+module_exit(cpufreq_interactive_exit);
+
+MODULE_AUTHOR("Mike Chan <mike@android.com>");
+MODULE_DESCRIPTION("'cpufreq_interactive' - A cpufreq governor for "
+	"Latency sensitive workloads");
+MODULE_LICENSE("GPL");
diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h
index dca22de..2ea8ab5 100644
--- a/include/linux/cpufreq.h
+++ b/include/linux/cpufreq.h
@@ -491,6 +491,9 @@ extern struct cpufreq_governor cpufreq_gov_ondemand;
 #elif defined(CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE)
 extern struct cpufreq_governor cpufreq_gov_conservative;
 #define CPUFREQ_DEFAULT_GOVERNOR	(&cpufreq_gov_conservative)
+#elif defined(CONFIG_CPU_FREQ_DEFAULT_GOV_INTERACTIVE)
+extern struct cpufreq_governor cpufreq_gov_interactive;
+#define CPUFREQ_DEFAULT_GOVERNOR	(&cpufreq_gov_interactive)
 #endif
 
 /*********************************************************************
-- 
1.7.9.5

[PATCH 02/70] cpufreq interactive governor: event tracing

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Change-Id: Ic13614a3da2faa2d4bd215ca3eb7191614f0cf66
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c      |   19 +++++-
 include/trace/events/cpufreq_interactive.h |   87 ++++++++++++++++++++++++++++
 2 files changed, 105 insertions(+), 1 deletion(-)
 create mode 100644 include/trace/events/cpufreq_interactive.h

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 8fc147c..62e7456 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -30,6 +30,9 @@
 #include <linux/kthread.h>
 #include <linux/mutex.h>
 
+#define CREATE_TRACE_POINTS
+#include <trace/events/cpufreq_interactive.h>
+
 #include <asm/cputime.h>
 
 static atomic_t active_count = ATOMIC_INIT(0);
@@ -182,7 +185,11 @@ static void cpufreq_interactive_timer(unsigned long data)
 	new_freq = pcpu->freq_table[index].frequency;
 
 	if (pcpu->target_freq == new_freq)
+	{
+		trace_cpufreq_interactive_already(data, cpu_load,
+						  pcpu->target_freq, new_freq);
 		goto rearm_if_notmax;
+	}
 
 	/*
 	 * Do not scale down unless we have been at this frequency for the
@@ -190,10 +197,16 @@ static void cpufreq_interactive_timer(unsigned long data)
 	 */
 	if (new_freq < pcpu->target_freq) {
 		if (pcpu->timer_run_time - pcpu->freq_change_time
-		    < min_sample_time)
+		    < min_sample_time) {
+			trace_cpufreq_interactive_notyet(data, cpu_load,
+					 pcpu->target_freq, new_freq);
 			goto rearm;
+		}
 	}
 
+	trace_cpufreq_interactive_target(data, cpu_load, pcpu->target_freq,
+					 new_freq);
+
 	if (new_freq < pcpu->target_freq) {
 		pcpu->target_freq = new_freq;
 		spin_lock_irqsave(&down_cpumask_lock, flags);
@@ -381,6 +394,8 @@ static int cpufreq_interactive_up_task(void *data)
 			pcpu->freq_change_time_in_idle =
 				get_cpu_idle_time_us(cpu,
 						     &pcpu->freq_change_time);
+			trace_cpufreq_interactive_up(cpu, pcpu->target_freq,
+						     pcpu->policy->cur);
 		}
 	}
 
@@ -427,6 +442,8 @@ static void cpufreq_interactive_freq_down(struct work_struct *work)
 		pcpu->freq_change_time_in_idle =
 			get_cpu_idle_time_us(cpu,
 					     &pcpu->freq_change_time);
+		trace_cpufreq_interactive_down(cpu, pcpu->target_freq,
+					       pcpu->policy->cur);
 	}
 }
 
diff --git a/include/trace/events/cpufreq_interactive.h b/include/trace/events/cpufreq_interactive.h
new file mode 100644
index 0000000..3a90d3d
--- /dev/null
+++ b/include/trace/events/cpufreq_interactive.h
@@ -0,0 +1,87 @@
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM cpufreq_interactive
+
+#if !defined(_TRACE_CPUFREQ_INTERACTIVE_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_CPUFREQ_INTERACTIVE_H
+
+#include <linux/tracepoint.h>
+
+DECLARE_EVENT_CLASS(set,
+	TP_PROTO(u32 cpu_id, unsigned long targfreq,
+	         unsigned long actualfreq),
+	TP_ARGS(cpu_id, targfreq, actualfreq),
+
+	TP_STRUCT__entry(
+	    __field(          u32, cpu_id    )
+	    __field(unsigned long, targfreq   )
+	    __field(unsigned long, actualfreq )
+	   ),
+
+	TP_fast_assign(
+	    __entry->cpu_id = (u32) cpu_id;
+	    __entry->targfreq = targfreq;
+	    __entry->actualfreq = actualfreq;
+	),
+
+	TP_printk("cpu=%u targ=%lu actual=%lu",
+	      __entry->cpu_id, __entry->targfreq,
+	      __entry->actualfreq)
+);
+
+DEFINE_EVENT(set, cpufreq_interactive_up,
+	TP_PROTO(u32 cpu_id, unsigned long targfreq,
+	     unsigned long actualfreq),
+	TP_ARGS(cpu_id, targfreq, actualfreq)
+);
+
+DEFINE_EVENT(set, cpufreq_interactive_down,
+	TP_PROTO(u32 cpu_id, unsigned long targfreq,
+	     unsigned long actualfreq),
+	TP_ARGS(cpu_id, targfreq, actualfreq)
+);
+
+DECLARE_EVENT_CLASS(loadeval,
+	    TP_PROTO(unsigned long cpu_id, unsigned long load,
+		     unsigned long curfreq, unsigned long targfreq),
+	    TP_ARGS(cpu_id, load, curfreq, targfreq),
+
+	    TP_STRUCT__entry(
+		    __field(unsigned long, cpu_id    )
+		    __field(unsigned long, load      )
+		    __field(unsigned long, curfreq   )
+		    __field(unsigned long, targfreq  )
+	    ),
+
+	    TP_fast_assign(
+		    __entry->cpu_id = cpu_id;
+		    __entry->load = load;
+		    __entry->curfreq = curfreq;
+		    __entry->targfreq = targfreq;
+	    ),
+
+	    TP_printk("cpu=%lu load=%lu cur=%lu targ=%lu",
+		      __entry->cpu_id, __entry->load, __entry->curfreq,
+		      __entry->targfreq)
+);
+
+DEFINE_EVENT(loadeval, cpufreq_interactive_target,
+	    TP_PROTO(unsigned long cpu_id, unsigned long load,
+		     unsigned long curfreq, unsigned long targfreq),
+	    TP_ARGS(cpu_id, load, curfreq, targfreq)
+);
+
+DEFINE_EVENT(loadeval, cpufreq_interactive_already,
+	    TP_PROTO(unsigned long cpu_id, unsigned long load,
+		     unsigned long curfreq, unsigned long targfreq),
+	    TP_ARGS(cpu_id, load, curfreq, targfreq)
+);
+
+DEFINE_EVENT(loadeval, cpufreq_interactive_notyet,
+	    TP_PROTO(unsigned long cpu_id, unsigned long load,
+		     unsigned long curfreq, unsigned long targfreq),
+	    TP_ARGS(cpu_id, load, curfreq, targfreq)
+);
+#endif /* _TRACE_CPUFREQ_INTERACTIVE_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
-- 
1.7.9.5

[PATCH 03/70] cpufreq: interactive: apply intermediate load to max speed not current

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Evaluate spikes in load (below go_hispeed_load) against the maximum
speed supported by the device, not the current speed (which tends to
make it too difficult to raise speed to intermediate levels until
very busy).

Change-Id: Ib937006abf8bedb60891a739acd733e89b732ae0
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 62e7456..e3bbe82 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -171,7 +171,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 		else
 			new_freq = pcpu->policy->max * cpu_load / 100;
 	} else {
-		new_freq = pcpu->policy->cur * cpu_load / 100;
+		new_freq = pcpu->policy->max * cpu_load / 100;
 	}
 
 	if (cpufreq_frequency_table_target(pcpu->policy, pcpu->freq_table,
-- 
1.7.9.5

[PATCH 04/70] cpufreq: interactive: set at least hispeed when above hispeed load

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

If load is above go_hispeed_load, always go to at least hispeed_freq,
even when reducing speed from a higher speed, not just when jumping
up from minimum speed.  Avoids running at a lower than intended
speed after a burst of even higher load.

Change-Id: I5b9d2a15ba25ce609b21bac7c724265cf6838dee
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index e3bbe82..12345c7b 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -166,10 +166,14 @@ static void cpufreq_interactive_timer(unsigned long data)
 		cpu_load = load_since_change;
 
 	if (cpu_load >= go_hispeed_load) {
-		if (pcpu->policy->cur == pcpu->policy->min)
+		if (pcpu->policy->cur == pcpu->policy->min) {
 			new_freq = hispeed_freq;
-		else
+		} else {
 			new_freq = pcpu->policy->max * cpu_load / 100;
+
+			if (new_freq < hispeed_freq)
+				new_freq = hispeed_freq;
+		}
 	} else {
 		new_freq = pcpu->policy->max * cpu_load / 100;
 	}
-- 
1.7.9.5

[PATCH 05/70] cpufreq: interactive: don't drop speed if recently at higher load

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Apply min_sample_time to the last time the current target speed
was originally requested or re-validated as appropriate for the
current load, not to the time since the current speed was
originally set.  Avoids periodic dips in speed during bursty
loads.

Change-Id: I250bda657985de60373f9897cc41f480664d51a1
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   38 +++++++++++++++------------------
 1 file changed, 17 insertions(+), 21 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 12345c7b..59e7779a 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -44,8 +44,8 @@ struct cpufreq_interactive_cpuinfo {
 	u64 idle_exit_time;
 	u64 timer_run_time;
 	int idling;
-	u64 freq_change_time;
-	u64 freq_change_time_in_idle;
+	u64 target_set_time;
+	u64 target_set_time_in_idle;
 	struct cpufreq_policy *policy;
 	struct cpufreq_frequency_table *freq_table;
 	unsigned int target_freq;
@@ -148,8 +148,9 @@ static void cpufreq_interactive_timer(unsigned long data)
 	else
 		cpu_load = 100 * (delta_time - delta_idle) / delta_time;
 
-	delta_idle = (unsigned int)(now_idle - pcpu->freq_change_time_in_idle);
-	delta_time = (unsigned int)(pcpu->timer_run_time - pcpu->freq_change_time);
+	delta_idle = (unsigned int)(now_idle - pcpu->target_set_time_in_idle);
+	delta_time = (unsigned int)(pcpu->timer_run_time -
+				    pcpu->target_set_time);
 
 	if ((delta_time == 0) || (delta_idle > delta_time))
 		load_since_change = 0;
@@ -188,19 +189,12 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 	new_freq = pcpu->freq_table[index].frequency;
 
-	if (pcpu->target_freq == new_freq)
-	{
-		trace_cpufreq_interactive_already(data, cpu_load,
-						  pcpu->target_freq, new_freq);
-		goto rearm_if_notmax;
-	}
-
 	/*
 	 * Do not scale down unless we have been at this frequency for the
 	 * minimum sample time.
 	 */
 	if (new_freq < pcpu->target_freq) {
-		if (pcpu->timer_run_time - pcpu->freq_change_time
+		if (pcpu->timer_run_time - pcpu->target_set_time
 		    < min_sample_time) {
 			trace_cpufreq_interactive_notyet(data, cpu_load,
 					 pcpu->target_freq, new_freq);
@@ -208,6 +202,15 @@ static void cpufreq_interactive_timer(unsigned long data)
 		}
 	}
 
+	pcpu->target_set_time_in_idle = now_idle;
+	pcpu->target_set_time = pcpu->timer_run_time;
+
+	if (pcpu->target_freq == new_freq) {
+		trace_cpufreq_interactive_already(data, cpu_load,
+						  pcpu->target_freq, new_freq);
+		goto rearm_if_notmax;
+	}
+
 	trace_cpufreq_interactive_target(data, cpu_load, pcpu->target_freq,
 					 new_freq);
 
@@ -394,10 +397,6 @@ static int cpufreq_interactive_up_task(void *data)
 							max_freq,
 							CPUFREQ_RELATION_H);
 			mutex_unlock(&set_speed_lock);
-
-			pcpu->freq_change_time_in_idle =
-				get_cpu_idle_time_us(cpu,
-						     &pcpu->freq_change_time);
 			trace_cpufreq_interactive_up(cpu, pcpu->target_freq,
 						     pcpu->policy->cur);
 		}
@@ -443,9 +442,6 @@ static void cpufreq_interactive_freq_down(struct work_struct *work)
 						CPUFREQ_RELATION_H);
 
 		mutex_unlock(&set_speed_lock);
-		pcpu->freq_change_time_in_idle =
-			get_cpu_idle_time_us(cpu,
-					     &pcpu->freq_change_time);
 		trace_cpufreq_interactive_down(cpu, pcpu->target_freq,
 					       pcpu->policy->cur);
 	}
@@ -575,9 +571,9 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->policy = policy;
 			pcpu->target_freq = policy->cur;
 			pcpu->freq_table = freq_table;
-			pcpu->freq_change_time_in_idle =
+			pcpu->target_set_time_in_idle =
 				get_cpu_idle_time_us(j,
-					     &pcpu->freq_change_time);
+					     &pcpu->target_set_time);
 			pcpu->governor_enabled = 1;
 			smp_wmb();
 		}
-- 
1.7.9.5

[PATCH 06/70] cpufreq: interactive: configurable delay before raising above hispeed

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Change-Id: I4d6ac40b23a3790d48e30c37408284e9f955e8fa
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   41 +++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 59e7779a..154f7bd 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -83,6 +83,13 @@ static unsigned long min_sample_time;
 #define DEFAULT_TIMER_RATE 20 * USEC_PER_MSEC
 static unsigned long timer_rate;
 
+/*
+ * Wait this long before raising speed above hispeed, by default a single
+ * timer interval.
+ */
+#define DEFAULT_ABOVE_HISPEED_DELAY DEFAULT_TIMER_RATE
+static unsigned long above_hispeed_delay_val;
+
 static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		unsigned int event);
 
@@ -174,6 +181,16 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 			if (new_freq < hispeed_freq)
 				new_freq = hispeed_freq;
+
+			if (pcpu->target_freq == hispeed_freq &&
+			    new_freq > hispeed_freq &&
+			    pcpu->timer_run_time - pcpu->target_set_time
+			    < above_hispeed_delay_val) {
+				trace_cpufreq_interactive_notyet(data, cpu_load,
+								 pcpu->target_freq,
+								 new_freq);
+				goto rearm;
+			}
 		}
 	} else {
 		new_freq = pcpu->policy->max * cpu_load / 100;
@@ -515,6 +532,28 @@ static ssize_t store_min_sample_time(struct kobject *kobj,
 static struct global_attr min_sample_time_attr = __ATTR(min_sample_time, 0644,
 		show_min_sample_time, store_min_sample_time);
 
+static ssize_t show_above_hispeed_delay(struct kobject *kobj,
+					struct attribute *attr, char *buf)
+{
+	return sprintf(buf, "%lu\n", above_hispeed_delay_val);
+}
+
+static ssize_t store_above_hispeed_delay(struct kobject *kobj,
+					 struct attribute *attr,
+					 const char *buf, size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = strict_strtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+	above_hispeed_delay_val = val;
+	return count;
+}
+
+define_one_global_rw(above_hispeed_delay);
+
 static ssize_t show_timer_rate(struct kobject *kobj,
 			struct attribute *attr, char *buf)
 {
@@ -540,6 +579,7 @@ static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644,
 static struct attribute *interactive_attributes[] = {
 	&hispeed_freq_attr.attr,
 	&go_hispeed_load_attr.attr,
+	&above_hispeed_delay.attr,
 	&min_sample_time_attr.attr,
 	&timer_rate_attr.attr,
 	NULL,
@@ -660,6 +700,7 @@ static int __init cpufreq_interactive_init(void)
 
 	go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;
 	min_sample_time = DEFAULT_MIN_SAMPLE_TIME;
+	above_hispeed_delay_val = DEFAULT_ABOVE_HISPEED_DELAY;
 	timer_rate = DEFAULT_TIMER_RATE;
 
 	/* Initalize per-cpu timers */
-- 
1.7.9.5

[PATCH 07/70] cpufreq: interactive: adjust code and documentation to match

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Change-Id: If59c668d514a29febe5c35404fd9d01df8548eb1
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 Documentation/cpu-freq/governors.txt  |   16 +++++++++++++---
 drivers/cpufreq/cpufreq_interactive.c |    6 +++---
 2 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/Documentation/cpu-freq/governors.txt b/Documentation/cpu-freq/governors.txt
index b262c53..40894da 100644
--- a/Documentation/cpu-freq/governors.txt
+++ b/Documentation/cpu-freq/governors.txt
@@ -249,11 +249,21 @@ frequency before ramping down. This is to ensure that the governor has
 seen enough historic cpu load data to determine the appropriate
 workload.  Default is 80000 uS.
 
-go_maxspeed_load: The CPU load at which to ramp to max speed.  Default
-is 85.
+hispeed_freq: An intermediate "hi speed" at which to initially ramp
+when CPU load hits the value specified in go_hispeed_load.  If load
+stays high for the amount of time specified in above_hispeed_delay,
+then speed may be bumped higher.  Default is maximum speed.
+
+go_hispeed_load: The CPU load at which to ramp to the intermediate "hi
+speed".  Default is 85%.
+
+above_hispeed_delay: Once speed is set to hispeed_freq, wait for this
+long before bumping speed higher in response to continued high load.
+Default is 20000 uS.
 
 timer_rate: Sample rate for reevaluating cpu load when the system is
-not idle.  Default is 30000 uS.
+not idle.  Default is 20000 uS.
+
 
 3. The Governor Interface in the CPUfreq Core
 =============================================
diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 154f7bd..bf8d772 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -68,19 +68,19 @@ static struct mutex set_speed_lock;
 static u64 hispeed_freq;
 
 /* Go to hi speed when CPU load at or above this value. */
-#define DEFAULT_GO_HISPEED_LOAD 95
+#define DEFAULT_GO_HISPEED_LOAD 85
 static unsigned long go_hispeed_load;
 
 /*
  * The minimum amount of time to spend at a frequency before we can ramp down.
  */
-#define DEFAULT_MIN_SAMPLE_TIME 20 * USEC_PER_MSEC
+#define DEFAULT_MIN_SAMPLE_TIME (80 * USEC_PER_MSEC)
 static unsigned long min_sample_time;
 
 /*
  * The sample rate of the timer used to increase frequency
  */
-#define DEFAULT_TIMER_RATE 20 * USEC_PER_MSEC
+#define DEFAULT_TIMER_RATE (20 * USEC_PER_MSEC)
 static unsigned long timer_rate;
 
 /*
-- 
1.7.9.5

[PATCH 08/70] cpufreq: interactive: base hispeed bump on target freq, not actual

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

For systems that set a common speed for all CPUs, checking current
speed here could bypass the intermediate hispeed bump decision for
this CPU when another CPU was already at hispeed.  This could
result in an overly high setting (for all CPUs) in situations
where all CPUs were about to drop to load levels that map to
hispeed or below.

Change-Id: I186f23dcfc5e2b6336cab8b0327f0c8a9a4482bc
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index bf8d772..b78174f 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -174,7 +174,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 		cpu_load = load_since_change;
 
 	if (cpu_load >= go_hispeed_load) {
-		if (pcpu->policy->cur == pcpu->policy->min) {
+		if (pcpu->target_freq <= pcpu->policy->min) {
 			new_freq = hispeed_freq;
 		} else {
 			new_freq = pcpu->policy->max * cpu_load / 100;
-- 
1.7.9.5

[PATCH 09/70] cpufreq: interactive: Separate speed target revalidate time and initial set time

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Allow speed drop after min_sample_time elapses from last time
the current speed was last re-validated as appropriate for
current load / input boost.

Allow speed bump after min_sample_time (or above_hispeed_delay)
elapses from the time the current speed was originally set.

Change-Id: Ic25687a7a53d25e6544c30c47d7ab6f27a47bee8
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index b78174f..083f790 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -46,6 +46,8 @@ struct cpufreq_interactive_cpuinfo {
 	int idling;
 	u64 target_set_time;
 	u64 target_set_time_in_idle;
+	u64 target_validate_time;
+	u64 target_validate_time_in_idle;
 	struct cpufreq_policy *policy;
 	struct cpufreq_frequency_table *freq_table;
 	unsigned int target_freq;
@@ -208,10 +210,10 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 	/*
 	 * Do not scale down unless we have been at this frequency for the
-	 * minimum sample time.
+	 * minimum sample time since last validated.
 	 */
 	if (new_freq < pcpu->target_freq) {
-		if (pcpu->timer_run_time - pcpu->target_set_time
+		if (pcpu->timer_run_time - pcpu->target_validate_time
 		    < min_sample_time) {
 			trace_cpufreq_interactive_notyet(data, cpu_load,
 					 pcpu->target_freq, new_freq);
@@ -219,8 +221,8 @@ static void cpufreq_interactive_timer(unsigned long data)
 		}
 	}
 
-	pcpu->target_set_time_in_idle = now_idle;
-	pcpu->target_set_time = pcpu->timer_run_time;
+	pcpu->target_validate_time_in_idle = now_idle;
+	pcpu->target_validate_time = pcpu->timer_run_time;
 
 	if (pcpu->target_freq == new_freq) {
 		trace_cpufreq_interactive_already(data, cpu_load,
@@ -230,6 +232,8 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 	trace_cpufreq_interactive_target(data, cpu_load, pcpu->target_freq,
 					 new_freq);
+	pcpu->target_set_time_in_idle = now_idle;
+	pcpu->target_set_time = pcpu->timer_run_time;
 
 	if (new_freq < pcpu->target_freq) {
 		pcpu->target_freq = new_freq;
@@ -614,6 +618,10 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->target_set_time_in_idle =
 				get_cpu_idle_time_us(j,
 					     &pcpu->target_set_time);
+			pcpu->target_validate_time =
+				pcpu->target_set_time;
+			pcpu->target_validate_time_in_idle =
+				pcpu->target_set_time_in_idle;
 			pcpu->governor_enabled = 1;
 			smp_wmb();
 		}
-- 
1.7.9.5

[PATCH 10/70] cpufreq: interactive: Boost frequency on touchscreen input

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Based on previous patches by Tero Kristo <tero.kristo@nokia.com>,
Brian Steuer <bsteuer@codeaurora.org>,
David Ng <dave@codeaurora.org>,
Antti P Miettinen <amiettinen@nvidia.com>, and
Thomas Renninger <trenn@suse.de>

Change-Id: Ic55fedcf6f9310f43a7022fb88e23b0392122769
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 Documentation/cpu-freq/governors.txt       |    3 +
 drivers/cpufreq/cpufreq_interactive.c      |  164 +++++++++++++++++++++++++++-
 include/trace/events/cpufreq_interactive.h |   12 ++
 3 files changed, 178 insertions(+), 1 deletion(-)

diff --git a/Documentation/cpu-freq/governors.txt b/Documentation/cpu-freq/governors.txt
index 40894da..4172cad 100644
--- a/Documentation/cpu-freq/governors.txt
+++ b/Documentation/cpu-freq/governors.txt
@@ -264,6 +264,9 @@ Default is 20000 uS.
 timer_rate: Sample rate for reevaluating cpu load when the system is
 not idle.  Default is 20000 uS.
 
+input_boost: If non-zero, boost speed of all CPUs to hispeed_freq on
+touchscreen activity.  Default is 0.
+
 
 3. The Governor Interface in the CPUfreq Core
 =============================================
diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 083f790..1387cd5 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -29,6 +29,8 @@
 #include <linux/workqueue.h>
 #include <linux/kthread.h>
 #include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/input.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/cpufreq_interactive.h>
@@ -92,6 +94,19 @@ static unsigned long timer_rate;
 #define DEFAULT_ABOVE_HISPEED_DELAY DEFAULT_TIMER_RATE
 static unsigned long above_hispeed_delay_val;
 
+/*
+ * Boost to hispeed on touchscreen input.
+ */
+
+static int input_boost_val;
+
+struct cpufreq_interactive_inputopen {
+	struct input_handle *handle;
+	struct work_struct inputopen_work;
+};
+
+static struct cpufreq_interactive_inputopen inputopen;
+
 static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		unsigned int event);
 
@@ -468,6 +483,125 @@ static void cpufreq_interactive_freq_down(struct work_struct *work)
 	}
 }
 
+static void cpufreq_interactive_boost(void)
+{
+	int i;
+	int anyboost = 0;
+	unsigned long flags;
+	struct cpufreq_interactive_cpuinfo *pcpu;
+
+	trace_cpufreq_interactive_boost(hispeed_freq);
+	spin_lock_irqsave(&up_cpumask_lock, flags);
+
+	for_each_online_cpu(i) {
+		pcpu = &per_cpu(cpuinfo, i);
+
+		if (pcpu->target_freq < hispeed_freq) {
+			pcpu->target_freq = hispeed_freq;
+			cpumask_set_cpu(i, &up_cpumask);
+			pcpu->target_set_time_in_idle =
+				get_cpu_idle_time_us(i, &pcpu->target_set_time);
+			anyboost = 1;
+		}
+
+		/*
+		 * Refresh time at which current (possibly being
+		 * boosted) speed last validated (reset timer for
+		 * allowing speed to drop).
+		 */
+
+		pcpu->target_validate_time_in_idle =
+			get_cpu_idle_time_us(i, &pcpu->target_validate_time);
+	}
+
+	spin_unlock_irqrestore(&up_cpumask_lock, flags);
+
+	if (anyboost)
+		wake_up_process(up_task);
+}
+
+static void cpufreq_interactive_input_event(struct input_handle *handle,
+					    unsigned int type,
+					    unsigned int code, int value)
+{
+	if (input_boost_val && type == EV_SYN && code == SYN_REPORT)
+		cpufreq_interactive_boost();
+}
+
+static void cpufreq_interactive_input_open(struct work_struct *w)
+{
+	struct cpufreq_interactive_inputopen *io =
+		container_of(w, struct cpufreq_interactive_inputopen,
+			     inputopen_work);
+	int error;
+
+	error = input_open_device(io->handle);
+	if (error)
+		input_unregister_handle(io->handle);
+}
+
+static int cpufreq_interactive_input_connect(struct input_handler *handler,
+					     struct input_dev *dev,
+					     const struct input_device_id *id)
+{
+	struct input_handle *handle;
+	int error;
+
+	pr_info("%s: connect to %s\n", __func__, dev->name);
+	handle = kzalloc(sizeof(struct input_handle), GFP_KERNEL);
+	if (!handle)
+		return -ENOMEM;
+
+	handle->dev = dev;
+	handle->handler = handler;
+	handle->name = "cpufreq_interactive";
+
+	error = input_register_handle(handle);
+	if (error)
+		goto err;
+
+	inputopen.handle = handle;
+	queue_work(down_wq, &inputopen.inputopen_work);
+	return 0;
+err:
+	kfree(handle);
+	return error;
+}
+
+static void cpufreq_interactive_input_disconnect(struct input_handle *handle)
+{
+	input_close_device(handle);
+	input_unregister_handle(handle);
+	kfree(handle);
+}
+
+static const struct input_device_id cpufreq_interactive_ids[] = {
+	{
+		.flags = INPUT_DEVICE_ID_MATCH_EVBIT |
+			 INPUT_DEVICE_ID_MATCH_ABSBIT,
+		.evbit = { BIT_MASK(EV_ABS) },
+		.absbit = { [BIT_WORD(ABS_MT_POSITION_X)] =
+			    BIT_MASK(ABS_MT_POSITION_X) |
+			    BIT_MASK(ABS_MT_POSITION_Y) },
+	}, /* multi-touch touchscreen */
+	{
+		.flags = INPUT_DEVICE_ID_MATCH_KEYBIT |
+			 INPUT_DEVICE_ID_MATCH_ABSBIT,
+		.keybit = { [BIT_WORD(BTN_TOUCH)] = BIT_MASK(BTN_TOUCH) },
+		.absbit = { [BIT_WORD(ABS_X)] =
+			    BIT_MASK(ABS_X) | BIT_MASK(ABS_Y) },
+	}, /* touchpad */
+	{ },
+};
+
+static struct input_handler cpufreq_interactive_input_handler = {
+	.event          = cpufreq_interactive_input_event,
+	.connect        = cpufreq_interactive_input_connect,
+	.disconnect     = cpufreq_interactive_input_disconnect,
+	.name           = "cpufreq_interactive",
+	.id_table       = cpufreq_interactive_ids,
+};
+
 static ssize_t show_hispeed_freq(struct kobject *kobj,
 				 struct attribute *attr, char *buf)
 {
@@ -580,12 +714,34 @@ static ssize_t store_timer_rate(struct kobject *kobj,
 static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644,
 		show_timer_rate, store_timer_rate);
 
+static ssize_t show_input_boost(struct kobject *kobj, struct attribute *attr,
+				char *buf)
+{
+	return sprintf(buf, "%u\n", input_boost_val);
+}
+
+static ssize_t store_input_boost(struct kobject *kobj, struct attribute *attr,
+				 const char *buf, size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = strict_strtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+	input_boost_val = val;
+	return count;
+}
+
+define_one_global_rw(input_boost);
+
 static struct attribute *interactive_attributes[] = {
 	&hispeed_freq_attr.attr,
 	&go_hispeed_load_attr.attr,
 	&above_hispeed_delay.attr,
 	&min_sample_time_attr.attr,
 	&timer_rate_attr.attr,
+	&input_boost.attr,
 	NULL,
 };
 
@@ -641,6 +797,11 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		if (rc)
 			return rc;
 
+		rc = input_register_handler(&cpufreq_interactive_input_handler);
+		if (rc)
+			pr_warn("%s: failed to register input handler\n",
+				__func__);
+
 		break;
 
 	case CPUFREQ_GOV_STOP:
@@ -663,6 +824,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		if (atomic_dec_return(&active_count) > 0)
 			return 0;
 
+		input_unregister_handler(&cpufreq_interactive_input_handler);
 		sysfs_remove_group(cpufreq_global_kobject,
 				&interactive_attr_group);
 
@@ -742,7 +904,7 @@ static int __init cpufreq_interactive_init(void)
 	mutex_init(&set_speed_lock);
 
 	idle_notifier_register(&cpufreq_interactive_idle_nb);
-
+	INIT_WORK(&inputopen.inputopen_work, cpufreq_interactive_input_open);
 	return cpufreq_register_governor(&cpufreq_gov_interactive);
 
 err_freeuptask:
diff --git a/include/trace/events/cpufreq_interactive.h b/include/trace/events/cpufreq_interactive.h
index 3a90d3d..19e070b 100644
--- a/include/trace/events/cpufreq_interactive.h
+++ b/include/trace/events/cpufreq_interactive.h
@@ -81,6 +81,18 @@ DEFINE_EVENT(loadeval, cpufreq_interactive_notyet,
 		     unsigned long curfreq, unsigned long targfreq),
 	    TP_ARGS(cpu_id, load, curfreq, targfreq)
 );
+
+TRACE_EVENT(cpufreq_interactive_boost,
+	    TP_PROTO(unsigned long freq),
+	    TP_ARGS(freq),
+	    TP_STRUCT__entry(
+		    __field(unsigned long, freq)
+	    ),
+	    TP_fast_assign(
+		    __entry->freq = freq;
+	    ),
+	    TP_printk("freq=%lu", __entry->freq)
+);
 #endif /* _TRACE_CPUFREQ_INTERACTIVE_H */
 
 /* This part must be outside protection */
-- 
1.7.9.5

[PATCH 11/70] cpufreq: interactive: remove unused target_validate_time_in_idle

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Change-Id: I37c5085b91318242612440dfd775ad762996612f
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 1387cd5..90db2c3 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -49,7 +49,6 @@ struct cpufreq_interactive_cpuinfo {
 	u64 target_set_time;
 	u64 target_set_time_in_idle;
 	u64 target_validate_time;
-	u64 target_validate_time_in_idle;
 	struct cpufreq_policy *policy;
 	struct cpufreq_frequency_table *freq_table;
 	unsigned int target_freq;
@@ -236,7 +235,6 @@ static void cpufreq_interactive_timer(unsigned long data)
 		}
 	}
 
-	pcpu->target_validate_time_in_idle = now_idle;
 	pcpu->target_validate_time = pcpu->timer_run_time;
 
 	if (pcpu->target_freq == new_freq) {
@@ -510,8 +508,7 @@ static void cpufreq_interactive_boost(void)
 		 * allowing speed to drop).
 		 */
 
-		pcpu->target_validate_time_in_idle =
-			get_cpu_idle_time_us(i, &pcpu->target_validate_time);
+		pcpu->target_validate_time = ktime_to_us(ktime_get());
 	}
 
 	spin_unlock_irqrestore(&up_cpumask_lock, flags);
@@ -776,8 +773,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 					     &pcpu->target_set_time);
 			pcpu->target_validate_time =
 				pcpu->target_set_time;
-			pcpu->target_validate_time_in_idle =
-				pcpu->target_set_time_in_idle;
 			pcpu->governor_enabled = 1;
 			smp_wmb();
 		}
-- 
1.7.9.5

[PATCH 12/70] cpufreq: interactive: Add sysfs boost interface for hints from userspace

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

The explicit hint on/off version.

Change-Id: Ibf62b6d45bf6fb8c9c055b9bdaf074ce9374c04f
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 Documentation/cpu-freq/governors.txt       |    3 ++
 drivers/cpufreq/cpufreq_interactive.c      |   48 +++++++++++++++++++++++++---
 include/trace/events/cpufreq_interactive.h |   13 ++++++++
 3 files changed, 60 insertions(+), 4 deletions(-)

diff --git a/Documentation/cpu-freq/governors.txt b/Documentation/cpu-freq/governors.txt
index 4172cad..4fcaa43 100644
--- a/Documentation/cpu-freq/governors.txt
+++ b/Documentation/cpu-freq/governors.txt
@@ -267,6 +267,9 @@ not idle.  Default is 20000 uS.
 input_boost: If non-zero, boost speed of all CPUs to hispeed_freq on
 touchscreen activity.  Default is 0.
 
+boost: If non-zero, immediately boost speed of all CPUs to
+hispeed_freq.  If zero, allow CPU speeds to drop below hispeed_freq.
+
 
 3. The Governor Interface in the CPUfreq Core
 =============================================
diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 90db2c3..07b7caf 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -31,12 +31,11 @@
 #include <linux/mutex.h>
 #include <linux/slab.h>
 #include <linux/input.h>
+#include <asm/cputime.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/cpufreq_interactive.h>
 
-#include <asm/cputime.h>
-
 static atomic_t active_count = ATOMIC_INIT(0);
 
 struct cpufreq_interactive_cpuinfo {
@@ -94,7 +93,7 @@ static unsigned long timer_rate;
 static unsigned long above_hispeed_delay_val;
 
 /*
- * Boost to hispeed on touchscreen input.
+ * Boost pulse to hispeed on touchscreen input.
  */
 
 static int input_boost_val;
@@ -106,6 +105,12 @@ struct cpufreq_interactive_inputopen {
 
 static struct cpufreq_interactive_inputopen inputopen;
 
+/*
+ * Non-zero means longer-term speed boost active.
+ */
+
+static int boost_val;
+
 static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		unsigned int event);
 
@@ -189,7 +194,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	if (load_since_change > cpu_load)
 		cpu_load = load_since_change;
 
-	if (cpu_load >= go_hispeed_load) {
+	if (cpu_load >= go_hispeed_load || boost_val) {
 		if (pcpu->target_freq <= pcpu->policy->min) {
 			new_freq = hispeed_freq;
 		} else {
@@ -517,6 +522,12 @@ static void cpufreq_interactive_boost(void)
 		wake_up_process(up_task);
 }
 
+/*
+ * Pulsed boost on input event raises CPUs to hispeed_freq and lets
+ * usual algorithm of min_sample_time  decide when to allow speed
+ * to drop.
+ */
+
 static void cpufreq_interactive_input_event(struct input_handle *handle,
 					    unsigned int type,
 					    unsigned int code, int value)
@@ -732,6 +743,34 @@ static ssize_t store_input_boost(struct kobject *kobj, struct attribute *attr,
 
 define_one_global_rw(input_boost);
 
+static ssize_t show_boost(struct kobject *kobj, struct attribute *attr,
+			  char *buf)
+{
+	return sprintf(buf, "%d\n", boost_val);
+}
+
+static ssize_t store_boost(struct kobject *kobj, struct attribute *attr,
+			   const char *buf, size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = kstrtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+
+	boost_val = val;
+
+	if (boost_val)
+		cpufreq_interactive_boost();
+	else
+		trace_cpufreq_interactive_unboost(hispeed_freq);
+
+	return count;
+}
+
+define_one_global_rw(boost);
+
 static struct attribute *interactive_attributes[] = {
 	&hispeed_freq_attr.attr,
 	&go_hispeed_load_attr.attr,
@@ -739,6 +778,7 @@ static struct attribute *interactive_attributes[] = {
 	&min_sample_time_attr.attr,
 	&timer_rate_attr.attr,
 	&input_boost.attr,
+	&boost.attr,
 	NULL,
 };
 
diff --git a/include/trace/events/cpufreq_interactive.h b/include/trace/events/cpufreq_interactive.h
index 19e070b..ae6f232 100644
--- a/include/trace/events/cpufreq_interactive.h
+++ b/include/trace/events/cpufreq_interactive.h
@@ -93,6 +93,19 @@ TRACE_EVENT(cpufreq_interactive_boost,
 	    ),
 	    TP_printk("freq=%lu", __entry->freq)
 );
+
+TRACE_EVENT(cpufreq_interactive_unboost,
+	    TP_PROTO(unsigned long freq),
+	    TP_ARGS(freq),
+	    TP_STRUCT__entry(
+		    __field(unsigned long, freq)
+	    ),
+	    TP_fast_assign(
+		    __entry->freq = freq;
+	    ),
+	    TP_printk("freq=%lu", __entry->freq)
+);
+
 #endif /* _TRACE_CPUFREQ_INTERACTIVE_H */
 
 /* This part must be outside protection */
-- 
1.7.9.5

[PATCH 13/70] cpufreq: interactive: set floor for boosted speed

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Allow speed to drop to flooor frequency but not below, don't pin
to speed at last boost.

Change-Id: I0147c2b7a2e61ba16820605af6baaf09570be787
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 07b7caf..fcd5b1d 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -47,10 +47,11 @@ struct cpufreq_interactive_cpuinfo {
 	int idling;
 	u64 target_set_time;
 	u64 target_set_time_in_idle;
-	u64 target_validate_time;
 	struct cpufreq_policy *policy;
 	struct cpufreq_frequency_table *freq_table;
 	unsigned int target_freq;
+	unsigned int floor_freq;
+	u64 floor_validate_time;
 	int governor_enabled;
 };
 
@@ -228,10 +229,10 @@ static void cpufreq_interactive_timer(unsigned long data)
 	new_freq = pcpu->freq_table[index].frequency;
 
 	/*
-	 * Do not scale down unless we have been at this frequency for the
-	 * minimum sample time since last validated.
+	 * Do not scale below floor_freq unless we have been at or above the
+	 * floor frequency for the minimum sample time since last validated.
 	 */
-	if (new_freq < pcpu->target_freq) {
+	if (new_freq < pcpu->floor_freq) {
 		if (pcpu->timer_run_time - pcpu->target_validate_time
 		    < min_sample_time) {
 			trace_cpufreq_interactive_notyet(data, cpu_load,
@@ -240,7 +241,8 @@ static void cpufreq_interactive_timer(unsigned long data)
 		}
 	}
 
-	pcpu->target_validate_time = pcpu->timer_run_time;
+	pcpu->floor_freq = new_freq;
+	pcpu->floor_validate_time = pcpu->timer_run_time;
 
 	if (pcpu->target_freq == new_freq) {
 		trace_cpufreq_interactive_already(data, cpu_load,
@@ -508,12 +510,12 @@ static void cpufreq_interactive_boost(void)
 		}
 
 		/*
-		 * Refresh time at which current (possibly being
-		 * boosted) speed last validated (reset timer for
-		 * allowing speed to drop).
+		 * Set floor freq and (re)start timer for when last
+		 * validated.
 		 */
 
-		pcpu->target_validate_time = ktime_to_us(ktime_get());
+		pcpu->floor_freq = hispeed_freq;
+		pcpu->floor_validate_time = ktime_to_us(ktime_get());
 	}
 
 	spin_unlock_irqrestore(&up_cpumask_lock, flags);
@@ -811,7 +813,8 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->target_set_time_in_idle =
 				get_cpu_idle_time_us(j,
 					     &pcpu->target_set_time);
-			pcpu->target_validate_time =
+			pcpu->floor_freq = pcpu->target_freq;
+			pcpu->floor_validate_time =
 				pcpu->target_set_time;
 			pcpu->governor_enabled = 1;
 			smp_wmb();
-- 
1.7.9.5

[PATCH 14/70] cpufreq: interactive: add boost pulse interface

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Change-Id: Icf1e86d2065cc8f0816ba9c6b065eb056d4e8249
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 Documentation/cpu-freq/governors.txt       |    9 ++++++--
 drivers/cpufreq/cpufreq_interactive.c      |   32 +++++++++++++++++++++++-----
 include/trace/events/cpufreq_interactive.h |   20 ++++++++---------
 3 files changed, 44 insertions(+), 17 deletions(-)

diff --git a/Documentation/cpu-freq/governors.txt b/Documentation/cpu-freq/governors.txt
index 4fcaa43..509b179 100644
--- a/Documentation/cpu-freq/governors.txt
+++ b/Documentation/cpu-freq/governors.txt
@@ -267,8 +267,13 @@ not idle.  Default is 20000 uS.
 input_boost: If non-zero, boost speed of all CPUs to hispeed_freq on
 touchscreen activity.  Default is 0.
 
-boost: If non-zero, immediately boost speed of all CPUs to
-hispeed_freq.  If zero, allow CPU speeds to drop below hispeed_freq.
+boost: If non-zero, immediately boost speed of all CPUs to at least
+hispeed_freq until zero is written to this attribute.  If zero, allow
+CPU speeds to drop below hispeed_freq according to load as usual.
+
+boostpulse: Immediately boost speed of all CPUs to hispeed_freq for
+min_sample_time, after which speeds are allowed to drop below
+hispeed_freq according to load as usual.
 
 
 3. The Governor Interface in the CPUfreq Core
diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index fcd5b1d..fcf0f21 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -495,7 +495,6 @@ static void cpufreq_interactive_boost(void)
 	unsigned long flags;
 	struct cpufreq_interactive_cpuinfo *pcpu;
 
-	trace_cpufreq_interactive_boost(hispeed_freq);
 	spin_lock_irqsave(&up_cpumask_lock, flags);
 
 	for_each_online_cpu(i) {
@@ -534,8 +533,10 @@ static void cpufreq_interactive_input_event(struct input_handle *handle,
 					    unsigned int type,
 					    unsigned int code, int value)
 {
-	if (input_boost_val && type == EV_SYN && code == SYN_REPORT)
+	if (input_boost_val && type == EV_SYN && code == SYN_REPORT) {
+		trace_cpufreq_interactive_boost("input");
 		cpufreq_interactive_boost();
+	}
 }
 
 static void cpufreq_interactive_input_open(struct work_struct *w)
@@ -763,16 +764,36 @@ static ssize_t store_boost(struct kobject *kobj, struct attribute *attr,
 
 	boost_val = val;
 
-	if (boost_val)
+	if (boost_val) {
+		trace_cpufreq_interactive_boost("on");
 		cpufreq_interactive_boost();
-	else
-		trace_cpufreq_interactive_unboost(hispeed_freq);
+	} else {
+		trace_cpufreq_interactive_unboost("off");
+	}
 
 	return count;
 }
 
 define_one_global_rw(boost);
 
+static ssize_t store_boostpulse(struct kobject *kobj, struct attribute *attr,
+				const char *buf, size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = kstrtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+
+	trace_cpufreq_interactive_boost("pulse");
+	cpufreq_interactive_boost();
+	return count;
+}
+
+static struct global_attr boostpulse =
+	__ATTR(boostpulse, 0200, NULL, store_boostpulse);
+
 static struct attribute *interactive_attributes[] = {
 	&hispeed_freq_attr.attr,
 	&go_hispeed_load_attr.attr,
@@ -781,6 +802,7 @@ static struct attribute *interactive_attributes[] = {
 	&timer_rate_attr.attr,
 	&input_boost.attr,
 	&boost.attr,
+	&boostpulse.attr,
 	NULL,
 };
 
diff --git a/include/trace/events/cpufreq_interactive.h b/include/trace/events/cpufreq_interactive.h
index ae6f232..0791f27 100644
--- a/include/trace/events/cpufreq_interactive.h
+++ b/include/trace/events/cpufreq_interactive.h
@@ -83,27 +83,27 @@ DEFINE_EVENT(loadeval, cpufreq_interactive_notyet,
 );
 
 TRACE_EVENT(cpufreq_interactive_boost,
-	    TP_PROTO(unsigned long freq),
-	    TP_ARGS(freq),
+	    TP_PROTO(char *s),
+	    TP_ARGS(s),
 	    TP_STRUCT__entry(
-		    __field(unsigned long, freq)
+		    __field(char *, s)
 	    ),
 	    TP_fast_assign(
-		    __entry->freq = freq;
+		    __entry->s = s;
 	    ),
-	    TP_printk("freq=%lu", __entry->freq)
+	    TP_printk("%s", __entry->s)
 );
 
 TRACE_EVENT(cpufreq_interactive_unboost,
-	    TP_PROTO(unsigned long freq),
-	    TP_ARGS(freq),
+	    TP_PROTO(char *s),
+	    TP_ARGS(s),
 	    TP_STRUCT__entry(
-		    __field(unsigned long, freq)
+		    __field(char *, s)
 	    ),
 	    TP_fast_assign(
-		    __entry->freq = freq;
+		    __entry->s = s;
 	    ),
-	    TP_printk("freq=%lu", __entry->freq)
+	    TP_printk("%s", __entry->s)
 );
 
 #endif /* _TRACE_CPUFREQ_INTERACTIVE_H */
-- 
1.7.9.5

[PATCH 15/70] cpufreq-interactive: Compile fixup

[Bálint Czobor]

From: John Stultz <john.stultz@linaro.org>

Looks like AOSP has a compile bug. Fix it up.

Signed-off-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index fcf0f21..7cb2e64 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -233,7 +233,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	 * floor frequency for the minimum sample time since last validated.
 	 */
 	if (new_freq < pcpu->floor_freq) {
-		if (pcpu->timer_run_time - pcpu->target_validate_time
+		if (pcpu->timer_run_time - pcpu->floor_validate_time
 		    < min_sample_time) {
 			trace_cpufreq_interactive_notyet(data, cpu_load,
 					 pcpu->target_freq, new_freq);
-- 
1.7.9.5

[PATCH 16/70] cpufreq: interactive: restart above_hispeed_delay at each hispeed load

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Change-Id: I2e5b91d45e8806b0ab94ca2301ed671c9af9ab13
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 7cb2e64..a4ed750 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -52,6 +52,7 @@ struct cpufreq_interactive_cpuinfo {
 	unsigned int target_freq;
 	unsigned int floor_freq;
 	u64 floor_validate_time;
+	u64 hispeed_validate_time;
 	int governor_enabled;
 };
 
@@ -206,7 +207,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 			if (pcpu->target_freq == hispeed_freq &&
 			    new_freq > hispeed_freq &&
-			    pcpu->timer_run_time - pcpu->target_set_time
+			    pcpu->timer_run_time - pcpu->hispeed_validate_time
 			    < above_hispeed_delay_val) {
 				trace_cpufreq_interactive_notyet(data, cpu_load,
 								 pcpu->target_freq,
@@ -218,6 +219,9 @@ static void cpufreq_interactive_timer(unsigned long data)
 		new_freq = pcpu->policy->max * cpu_load / 100;
 	}
 
+	if (new_freq <= hispeed_freq)
+		pcpu->hispeed_validate_time = pcpu->timer_run_time;
+
 	if (cpufreq_frequency_table_target(pcpu->policy, pcpu->freq_table,
 					   new_freq, CPUFREQ_RELATION_H,
 					   &index)) {
@@ -505,6 +509,7 @@ static void cpufreq_interactive_boost(void)
 			cpumask_set_cpu(i, &up_cpumask);
 			pcpu->target_set_time_in_idle =
 				get_cpu_idle_time_us(i, &pcpu->target_set_time);
+			pcpu->hispeed_validate_time = pcpu->target_set_time;
 			anyboost = 1;
 		}
 
@@ -838,6 +843,8 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->floor_freq = pcpu->target_freq;
 			pcpu->floor_validate_time =
 				pcpu->target_set_time;
+			pcpu->hispeed_validate_time =
+				pcpu->target_set_time;
 			pcpu->governor_enabled = 1;
 			smp_wmb();
 		}
-- 
1.7.9.5

[PATCH 17/70] cpufreq: interactive: take idle notifications only when active

[Bálint Czobor]

From: Sam Leffler <sleffler@chromium.org>

Register an idle notifier only when the governor is active.  Also
short-circuit work of idle end if the governor is not enabled.

Signed-off-by: Sam Leffler <sleffler@chromium.org>
Change-Id: I4cae36dd2e7389540d337d74745ffbaa0131870f
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   46 ++++++++++++++++++---------------
 1 file changed, 25 insertions(+), 21 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index a4ed750..2038991 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -364,6 +364,9 @@ static void cpufreq_interactive_idle_end(void)
 	struct cpufreq_interactive_cpuinfo *pcpu =
 		&per_cpu(cpuinfo, smp_processor_id());
 
+	if (!pcpu->governor_enabled)
+		return;
+
 	pcpu->idling = 0;
 	smp_wmb();
 
@@ -816,6 +819,26 @@ static struct attribute_group interactive_attr_group = {
 	.name = "interactive",
 };
 
+static int cpufreq_interactive_idle_notifier(struct notifier_block *nb,
+					     unsigned long val,
+					     void *data)
+{
+	switch (val) {
+	case IDLE_START:
+		cpufreq_interactive_idle_start();
+		break;
+	case IDLE_END:
+		cpufreq_interactive_idle_end();
+		break;
+	}
+
+	return 0;
+}
+
+static struct notifier_block cpufreq_interactive_idle_nb = {
+	.notifier_call = cpufreq_interactive_idle_notifier,
+};
+
 static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		unsigned int event)
 {
@@ -869,6 +892,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pr_warn("%s: failed to register input handler\n",
 				__func__);
 
+		idle_notifier_register(&cpufreq_interactive_idle_nb);
 		break;
 
 	case CPUFREQ_GOV_STOP:
@@ -891,6 +915,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		if (atomic_dec_return(&active_count) > 0)
 			return 0;
 
+		idle_notifier_unregister(&cpufreq_interactive_idle_nb);
 		input_unregister_handler(&cpufreq_interactive_input_handler);
 		sysfs_remove_group(cpufreq_global_kobject,
 				&interactive_attr_group);
@@ -909,26 +934,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 	return 0;
 }
 
-static int cpufreq_interactive_idle_notifier(struct notifier_block *nb,
-					     unsigned long val,
-					     void *data)
-{
-	switch (val) {
-	case IDLE_START:
-		cpufreq_interactive_idle_start();
-		break;
-	case IDLE_END:
-		cpufreq_interactive_idle_end();
-		break;
-	}
-
-	return 0;
-}
-
-static struct notifier_block cpufreq_interactive_idle_nb = {
-	.notifier_call = cpufreq_interactive_idle_notifier,
-};
-
 static int __init cpufreq_interactive_init(void)
 {
 	unsigned int i;
@@ -970,7 +975,6 @@ static int __init cpufreq_interactive_init(void)
 	spin_lock_init(&down_cpumask_lock);
 	mutex_init(&set_speed_lock);
 
-	idle_notifier_register(&cpufreq_interactive_idle_nb);
 	INIT_WORK(&inputopen.inputopen_work, cpufreq_interactive_input_open);
 	return cpufreq_register_governor(&cpufreq_gov_interactive);
 
-- 
1.7.9.5

[PATCH 18/70] cpufreq: interactive: keep freezer happy when not current governor

[Bálint Czobor]

From: Sam Leffler <sleffler@chromium.org>

Fix a problem where the hung task mechanism was deeming the interactive
clock boost thread as hung.  This was because the thread is created at
module init but never run/woken up until needed.  If the governor is not
being used this can be forever.  To workaround this explicitly wake up
the thread once all the necessary data structures are initialized.  The
latter required some minor code shuffle.

Signed-off-by: Sam Leffler <sleffler@chromium.org>
Change-Id: Ie2c058dd75dcb6460ea10e7ac997e46baf66b1fe
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 2038991..184140a 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -953,6 +953,10 @@ static int __init cpufreq_interactive_init(void)
 		pcpu->cpu_timer.data = i;
 	}
 
+	spin_lock_init(&up_cpumask_lock);
+	spin_lock_init(&down_cpumask_lock);
+	mutex_init(&set_speed_lock);
+
 	up_task = kthread_create(cpufreq_interactive_up_task, NULL,
 				 "kinteractiveup");
 	if (IS_ERR(up_task))
@@ -968,14 +972,12 @@ static int __init cpufreq_interactive_init(void)
 	if (!down_wq)
 		goto err_freeuptask;
 
-	INIT_WORK(&freq_scale_down_work,
-		  cpufreq_interactive_freq_down);
+	INIT_WORK(&freq_scale_down_work, cpufreq_interactive_freq_down);
+	INIT_WORK(&inputopen.inputopen_work, cpufreq_interactive_input_open);
 
-	spin_lock_init(&up_cpumask_lock);
-	spin_lock_init(&down_cpumask_lock);
-	mutex_init(&set_speed_lock);
+	/* NB: wake up so the thread does not look hung to the freezer */
+	wake_up_process(up_task);
 
-	INIT_WORK(&inputopen.inputopen_work, cpufreq_interactive_input_open);
 	return cpufreq_register_governor(&cpufreq_gov_interactive);
 
 err_freeuptask:
-- 
1.7.9.5

[PATCH 19/70] cpufreq: interactive: handle speed up and down in the realtime task

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Not useful to have a separate, non-realtime workqueue for speed down
events, avoid priority inversion for speed up events.

Change-Id: Iddcd05545245c847aa1bbe0b8790092914c813d2
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c      |  148 +++++++++-------------------
 include/trace/events/cpufreq_interactive.h |    8 +-
 2 files changed, 45 insertions(+), 111 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 184140a..e9c8516 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -58,15 +58,10 @@ struct cpufreq_interactive_cpuinfo {
 
 static DEFINE_PER_CPU(struct cpufreq_interactive_cpuinfo, cpuinfo);
 
-/* Workqueues handle frequency scaling */
-static struct task_struct *up_task;
-static struct workqueue_struct *down_wq;
-static struct work_struct freq_scale_down_work;
-static cpumask_t up_cpumask;
-static spinlock_t up_cpumask_lock;
-static cpumask_t down_cpumask;
-static spinlock_t down_cpumask_lock;
-static struct mutex set_speed_lock;
+/* realtime thread handles frequency scaling */
+static struct task_struct *speedchange_task;
+static cpumask_t speedchange_cpumask;
+static spinlock_t speedchange_cpumask_lock;
 
 /* Hi speed to bump to from lo speed when load burst (default max) */
 static u64 hispeed_freq;
@@ -106,6 +101,7 @@ struct cpufreq_interactive_inputopen {
 };
 
 static struct cpufreq_interactive_inputopen inputopen;
+static struct workqueue_struct *inputopen_wq;
 
 /*
  * Non-zero means longer-term speed boost active.
@@ -259,19 +255,11 @@ static void cpufreq_interactive_timer(unsigned long data)
 	pcpu->target_set_time_in_idle = now_idle;
 	pcpu->target_set_time = pcpu->timer_run_time;
 
-	if (new_freq < pcpu->target_freq) {
-		pcpu->target_freq = new_freq;
-		spin_lock_irqsave(&down_cpumask_lock, flags);
-		cpumask_set_cpu(data, &down_cpumask);
-		spin_unlock_irqrestore(&down_cpumask_lock, flags);
-		queue_work(down_wq, &freq_scale_down_work);
-	} else {
-		pcpu->target_freq = new_freq;
-		spin_lock_irqsave(&up_cpumask_lock, flags);
-		cpumask_set_cpu(data, &up_cpumask);
-		spin_unlock_irqrestore(&up_cpumask_lock, flags);
-		wake_up_process(up_task);
-	}
+	pcpu->target_freq = new_freq;
+	spin_lock_irqsave(&speedchange_cpumask_lock, flags);
+	cpumask_set_cpu(data, &speedchange_cpumask);
+	spin_unlock_irqrestore(&speedchange_cpumask_lock, flags);
+	wake_up_process(speedchange_task);
 
 rearm_if_notmax:
 	/*
@@ -394,7 +382,7 @@ static void cpufreq_interactive_idle_end(void)
 
 }
 
-static int cpufreq_interactive_up_task(void *data)
+static int cpufreq_interactive_speedchange_task(void *data)
 {
 	unsigned int cpu;
 	cpumask_t tmp_mask;
@@ -403,22 +391,23 @@ static int cpufreq_interactive_up_task(void *data)
 
 	while (1) {
 		set_current_state(TASK_INTERRUPTIBLE);
-		spin_lock_irqsave(&up_cpumask_lock, flags);
+		spin_lock_irqsave(&speedchange_cpumask_lock, flags);
 
-		if (cpumask_empty(&up_cpumask)) {
-			spin_unlock_irqrestore(&up_cpumask_lock, flags);
+		if (cpumask_empty(&speedchange_cpumask)) {
+			spin_unlock_irqrestore(&speedchange_cpumask_lock,
+					       flags);
 			schedule();
 
 			if (kthread_should_stop())
 				break;
 
-			spin_lock_irqsave(&up_cpumask_lock, flags);
+			spin_lock_irqsave(&speedchange_cpumask_lock, flags);
 		}
 
 		set_current_state(TASK_RUNNING);
-		tmp_mask = up_cpumask;
-		cpumask_clear(&up_cpumask);
-		spin_unlock_irqrestore(&up_cpumask_lock, flags);
+		tmp_mask = speedchange_cpumask;
+		cpumask_clear(&speedchange_cpumask);
+		spin_unlock_irqrestore(&speedchange_cpumask_lock, flags);
 
 		for_each_cpu(cpu, &tmp_mask) {
 			unsigned int j;
@@ -430,8 +419,6 @@ static int cpufreq_interactive_up_task(void *data)
 			if (!pcpu->governor_enabled)
 				continue;
 
-			mutex_lock(&set_speed_lock);
-
 			for_each_cpu(j, pcpu->policy->cpus) {
 				struct cpufreq_interactive_cpuinfo *pjcpu =
 					&per_cpu(cpuinfo, j);
@@ -444,8 +431,8 @@ static int cpufreq_interactive_up_task(void *data)
 				__cpufreq_driver_target(pcpu->policy,
 							max_freq,
 							CPUFREQ_RELATION_H);
-			mutex_unlock(&set_speed_lock);
-			trace_cpufreq_interactive_up(cpu, pcpu->target_freq,
+			trace_cpufreq_interactive_setspeed(cpu,
+						     pcpu->target_freq,
 						     pcpu->policy->cur);
 		}
 	}
@@ -453,48 +440,6 @@ static int cpufreq_interactive_up_task(void *data)
 	return 0;
 }
 
-static void cpufreq_interactive_freq_down(struct work_struct *work)
-{
-	unsigned int cpu;
-	cpumask_t tmp_mask;
-	unsigned long flags;
-	struct cpufreq_interactive_cpuinfo *pcpu;
-
-	spin_lock_irqsave(&down_cpumask_lock, flags);
-	tmp_mask = down_cpumask;
-	cpumask_clear(&down_cpumask);
-	spin_unlock_irqrestore(&down_cpumask_lock, flags);
-
-	for_each_cpu(cpu, &tmp_mask) {
-		unsigned int j;
-		unsigned int max_freq = 0;
-
-		pcpu = &per_cpu(cpuinfo, cpu);
-		smp_rmb();
-
-		if (!pcpu->governor_enabled)
-			continue;
-
-		mutex_lock(&set_speed_lock);
-
-		for_each_cpu(j, pcpu->policy->cpus) {
-			struct cpufreq_interactive_cpuinfo *pjcpu =
-				&per_cpu(cpuinfo, j);
-
-			if (pjcpu->target_freq > max_freq)
-				max_freq = pjcpu->target_freq;
-		}
-
-		if (max_freq != pcpu->policy->cur)
-			__cpufreq_driver_target(pcpu->policy, max_freq,
-						CPUFREQ_RELATION_H);
-
-		mutex_unlock(&set_speed_lock);
-		trace_cpufreq_interactive_down(cpu, pcpu->target_freq,
-					       pcpu->policy->cur);
-	}
-}
-
 static void cpufreq_interactive_boost(void)
 {
 	int i;
@@ -502,14 +447,14 @@ static void cpufreq_interactive_boost(void)
 	unsigned long flags;
 	struct cpufreq_interactive_cpuinfo *pcpu;
 
-	spin_lock_irqsave(&up_cpumask_lock, flags);
+	spin_lock_irqsave(&speedchange_cpumask_lock, flags);
 
 	for_each_online_cpu(i) {
 		pcpu = &per_cpu(cpuinfo, i);
 
 		if (pcpu->target_freq < hispeed_freq) {
 			pcpu->target_freq = hispeed_freq;
-			cpumask_set_cpu(i, &up_cpumask);
+			cpumask_set_cpu(i, &speedchange_cpumask);
 			pcpu->target_set_time_in_idle =
 				get_cpu_idle_time_us(i, &pcpu->target_set_time);
 			pcpu->hispeed_validate_time = pcpu->target_set_time;
@@ -525,10 +470,10 @@ static void cpufreq_interactive_boost(void)
 		pcpu->floor_validate_time = ktime_to_us(ktime_get());
 	}
 
-	spin_unlock_irqrestore(&up_cpumask_lock, flags);
+	spin_unlock_irqrestore(&speedchange_cpumask_lock, flags);
 
 	if (anyboost)
-		wake_up_process(up_task);
+		wake_up_process(speedchange_task);
 }
 
 /*
@@ -580,7 +525,7 @@ static int cpufreq_interactive_input_connect(struct input_handler *handler,
 		goto err;
 
 	inputopen.handle = handle;
-	queue_work(down_wq, &inputopen.inputopen_work);
+	queue_work(inputopen_wq, &inputopen.inputopen_work);
 	return 0;
 err:
 	kfree(handle);
@@ -911,7 +856,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->idle_exit_time = 0;
 		}
 
-		flush_work(&freq_scale_down_work);
+		flush_work(&inputopen.inputopen_work);
 		if (atomic_dec_return(&active_count) > 0)
 			return 0;
 
@@ -953,35 +898,30 @@ static int __init cpufreq_interactive_init(void)
 		pcpu->cpu_timer.data = i;
 	}
 
-	spin_lock_init(&up_cpumask_lock);
-	spin_lock_init(&down_cpumask_lock);
-	mutex_init(&set_speed_lock);
-
-	up_task = kthread_create(cpufreq_interactive_up_task, NULL,
-				 "kinteractiveup");
-	if (IS_ERR(up_task))
-		return PTR_ERR(up_task);
+	spin_lock_init(&speedchange_cpumask_lock);
+	speedchange_task =
+		kthread_create(cpufreq_interactive_speedchange_task, NULL,
+			       "cfinteractive");
+	if (IS_ERR(speedchange_task))
+		return PTR_ERR(speedchange_task);
 
-	sched_setscheduler_nocheck(up_task, SCHED_FIFO, &param);
-	get_task_struct(up_task);
+	sched_setscheduler_nocheck(speedchange_task, SCHED_FIFO, &param);
+	get_task_struct(speedchange_task);
 
-	/* No rescuer thread, bind to CPU queuing the work for possibly
-	   warm cache (probably doesn't matter much). */
-	down_wq = alloc_workqueue("knteractive_down", 0, 1);
+	inputopen_wq = create_workqueue("cfinteractive");
 
-	if (!down_wq)
-		goto err_freeuptask;
+	if (!inputopen_wq)
+		goto err_freetask;
 
-	INIT_WORK(&freq_scale_down_work, cpufreq_interactive_freq_down);
 	INIT_WORK(&inputopen.inputopen_work, cpufreq_interactive_input_open);
 
 	/* NB: wake up so the thread does not look hung to the freezer */
-	wake_up_process(up_task);
+	wake_up_process(speedchange_task);
 
 	return cpufreq_register_governor(&cpufreq_gov_interactive);
 
-err_freeuptask:
-	put_task_struct(up_task);
+err_freetask:
+	put_task_struct(speedchange_task);
 	return -ENOMEM;
 }
 
@@ -994,9 +934,9 @@ module_init(cpufreq_interactive_init);
 static void __exit cpufreq_interactive_exit(void)
 {
 	cpufreq_unregister_governor(&cpufreq_gov_interactive);
-	kthread_stop(up_task);
-	put_task_struct(up_task);
-	destroy_workqueue(down_wq);
+	kthread_stop(speedchange_task);
+	put_task_struct(speedchange_task);
+	destroy_workqueue(inputopen_wq);
 }
 
 module_exit(cpufreq_interactive_exit);
diff --git a/include/trace/events/cpufreq_interactive.h b/include/trace/events/cpufreq_interactive.h
index 0791f27..ba0634e 100644
--- a/include/trace/events/cpufreq_interactive.h
+++ b/include/trace/events/cpufreq_interactive.h
@@ -28,13 +28,7 @@ DECLARE_EVENT_CLASS(set,
 	      __entry->actualfreq)
 );
 
-DEFINE_EVENT(set, cpufreq_interactive_up,
-	TP_PROTO(u32 cpu_id, unsigned long targfreq,
-	     unsigned long actualfreq),
-	TP_ARGS(cpu_id, targfreq, actualfreq)
-);
-
-DEFINE_EVENT(set, cpufreq_interactive_down,
+DEFINE_EVENT(set, cpufreq_interactive_setspeed,
 	TP_PROTO(u32 cpu_id, unsigned long targfreq,
 	     unsigned long actualfreq),
 	TP_ARGS(cpu_id, targfreq, actualfreq)
-- 
1.7.9.5

[PATCH 20/70] cpufreq: interactive: remove input_boost handling

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Now handled in userspace Power HAL instead.

Change-Id: I78a4a2fd471308bfcd785bbefcc65fede27314cf
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |  146 ---------------------------------
 1 file changed, 146 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index e9c8516..4b60b6e 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -30,7 +30,6 @@
 #include <linux/kthread.h>
 #include <linux/mutex.h>
 #include <linux/slab.h>
-#include <linux/input.h>
 #include <asm/cputime.h>
 
 #define CREATE_TRACE_POINTS
@@ -90,20 +89,6 @@ static unsigned long timer_rate;
 static unsigned long above_hispeed_delay_val;
 
 /*
- * Boost pulse to hispeed on touchscreen input.
- */
-
-static int input_boost_val;
-
-struct cpufreq_interactive_inputopen {
-	struct input_handle *handle;
-	struct work_struct inputopen_work;
-};
-
-static struct cpufreq_interactive_inputopen inputopen;
-static struct workqueue_struct *inputopen_wq;
-
-/*
  * Non-zero means longer-term speed boost active.
  */
 
@@ -476,96 +461,6 @@ static void cpufreq_interactive_boost(void)
 		wake_up_process(speedchange_task);
 }
 
-/*
- * Pulsed boost on input event raises CPUs to hispeed_freq and lets
- * usual algorithm of min_sample_time  decide when to allow speed
- * to drop.
- */
-
-static void cpufreq_interactive_input_event(struct input_handle *handle,
-					    unsigned int type,
-					    unsigned int code, int value)
-{
-	if (input_boost_val && type == EV_SYN && code == SYN_REPORT) {
-		trace_cpufreq_interactive_boost("input");
-		cpufreq_interactive_boost();
-	}
-}
-
-static void cpufreq_interactive_input_open(struct work_struct *w)
-{
-	struct cpufreq_interactive_inputopen *io =
-		container_of(w, struct cpufreq_interactive_inputopen,
-			     inputopen_work);
-	int error;
-
-	error = input_open_device(io->handle);
-	if (error)
-		input_unregister_handle(io->handle);
-}
-
-static int cpufreq_interactive_input_connect(struct input_handler *handler,
-					     struct input_dev *dev,
-					     const struct input_device_id *id)
-{
-	struct input_handle *handle;
-	int error;
-
-	pr_info("%s: connect to %s\n", __func__, dev->name);
-	handle = kzalloc(sizeof(struct input_handle), GFP_KERNEL);
-	if (!handle)
-		return -ENOMEM;
-
-	handle->dev = dev;
-	handle->handler = handler;
-	handle->name = "cpufreq_interactive";
-
-	error = input_register_handle(handle);
-	if (error)
-		goto err;
-
-	inputopen.handle = handle;
-	queue_work(inputopen_wq, &inputopen.inputopen_work);
-	return 0;
-err:
-	kfree(handle);
-	return error;
-}
-
-static void cpufreq_interactive_input_disconnect(struct input_handle *handle)
-{
-	input_close_device(handle);
-	input_unregister_handle(handle);
-	kfree(handle);
-}
-
-static const struct input_device_id cpufreq_interactive_ids[] = {
-	{
-		.flags = INPUT_DEVICE_ID_MATCH_EVBIT |
-			 INPUT_DEVICE_ID_MATCH_ABSBIT,
-		.evbit = { BIT_MASK(EV_ABS) },
-		.absbit = { [BIT_WORD(ABS_MT_POSITION_X)] =
-			    BIT_MASK(ABS_MT_POSITION_X) |
-			    BIT_MASK(ABS_MT_POSITION_Y) },
-	}, /* multi-touch touchscreen */
-	{
-		.flags = INPUT_DEVICE_ID_MATCH_KEYBIT |
-			 INPUT_DEVICE_ID_MATCH_ABSBIT,
-		.keybit = { [BIT_WORD(BTN_TOUCH)] = BIT_MASK(BTN_TOUCH) },
-		.absbit = { [BIT_WORD(ABS_X)] =
-			    BIT_MASK(ABS_X) | BIT_MASK(ABS_Y) },
-	}, /* touchpad */
-	{ },
-};
-
-static struct input_handler cpufreq_interactive_input_handler = {
-	.event          = cpufreq_interactive_input_event,
-	.connect        = cpufreq_interactive_input_connect,
-	.disconnect     = cpufreq_interactive_input_disconnect,
-	.name           = "cpufreq_interactive",
-	.id_table       = cpufreq_interactive_ids,
-};
-
 static ssize_t show_hispeed_freq(struct kobject *kobj,
 				 struct attribute *attr, char *buf)
 {
@@ -678,27 +573,6 @@ static ssize_t store_timer_rate(struct kobject *kobj,
 static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644,
 		show_timer_rate, store_timer_rate);
 
-static ssize_t show_input_boost(struct kobject *kobj, struct attribute *attr,
-				char *buf)
-{
-	return sprintf(buf, "%u\n", input_boost_val);
-}
-
-static ssize_t store_input_boost(struct kobject *kobj, struct attribute *attr,
-				 const char *buf, size_t count)
-{
-	int ret;
-	unsigned long val;
-
-	ret = strict_strtoul(buf, 0, &val);
-	if (ret < 0)
-		return ret;
-	input_boost_val = val;
-	return count;
-}
-
-define_one_global_rw(input_boost);
-
 static ssize_t show_boost(struct kobject *kobj, struct attribute *attr,
 			  char *buf)
 {
@@ -753,7 +627,6 @@ static struct attribute *interactive_attributes[] = {
 	&above_hispeed_delay.attr,
 	&min_sample_time_attr.attr,
 	&timer_rate_attr.attr,
-	&input_boost.attr,
 	&boost.attr,
 	&boostpulse.attr,
 	NULL,
@@ -832,11 +705,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		if (rc)
 			return rc;
 
-		rc = input_register_handler(&cpufreq_interactive_input_handler);
-		if (rc)
-			pr_warn("%s: failed to register input handler\n",
-				__func__);
-
 		idle_notifier_register(&cpufreq_interactive_idle_nb);
 		break;
 
@@ -856,12 +724,10 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->idle_exit_time = 0;
 		}
 
-		flush_work(&inputopen.inputopen_work);
 		if (atomic_dec_return(&active_count) > 0)
 			return 0;
 
 		idle_notifier_unregister(&cpufreq_interactive_idle_nb);
-		input_unregister_handler(&cpufreq_interactive_input_handler);
 		sysfs_remove_group(cpufreq_global_kobject,
 				&interactive_attr_group);
 
@@ -908,21 +774,10 @@ static int __init cpufreq_interactive_init(void)
 	sched_setscheduler_nocheck(speedchange_task, SCHED_FIFO, &param);
 	get_task_struct(speedchange_task);
 
-	inputopen_wq = create_workqueue("cfinteractive");
-
-	if (!inputopen_wq)
-		goto err_freetask;
-
-	INIT_WORK(&inputopen.inputopen_work, cpufreq_interactive_input_open);
-
 	/* NB: wake up so the thread does not look hung to the freezer */
 	wake_up_process(speedchange_task);
 
 	return cpufreq_register_governor(&cpufreq_gov_interactive);
-
-err_freetask:
-	put_task_struct(speedchange_task);
-	return -ENOMEM;
 }
 
 #ifdef CONFIG_CPU_FREQ_DEFAULT_GOV_INTERACTIVE
@@ -936,7 +791,6 @@ static void __exit cpufreq_interactive_exit(void)
 	cpufreq_unregister_governor(&cpufreq_gov_interactive);
 	kthread_stop(speedchange_task);
 	put_task_struct(speedchange_task);
-	destroy_workqueue(inputopen_wq);
 }
 
 module_exit(cpufreq_interactive_exit);
-- 
1.7.9.5

[PATCH 21/70] cpufreq: interactive: always limit initial speed bump to hispeed

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

First bump speed up to hispeed_freq whenever the current speed is below
hispeed_freq, instead of only when the current speed is the minimum speed.
The previous code made it too difficult to use hispeed_freq as a common
intermediate speed on systems that frequently run at speeds between
minimum and hispeed_freq.

Change-Id: I04ec30bafabf5741e267ff289209b8c2d846824b
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 4b60b6e..f85fbbd 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -178,7 +178,8 @@ static void cpufreq_interactive_timer(unsigned long data)
 		cpu_load = load_since_change;
 
 	if (cpu_load >= go_hispeed_load || boost_val) {
-		if (pcpu->target_freq <= pcpu->policy->min) {
+		if (pcpu->target_freq < hispeed_freq &&
+		    hispeed_freq < pcpu->policy->max) {
 			new_freq = hispeed_freq;
 		} else {
 			new_freq = pcpu->policy->max * cpu_load / 100;
-- 
1.7.9.5

[PATCH 22/70] cpufreq: interactive: run at fraction of hispeed_freq when load is low

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

When load is below go_hispeed_load, apply the percentage of CPU load to
a max frequency of hispeed_freq instead of the max speed.  This avoids
jumping too quickly to hispeed_freq when it is a relatively low
percentage of max speed.  This also allows go_hispeed_load to be set to
a high percentage relative to hispeed_freq (as a percentage of max speed,
again useful when hispeed_freq is a low fraction of max speed), to cap
larger loads at hispeed_freq.  For example, a load of 60% will typically
move to 60% of hispeed_freq, not 60% of max speed.  This causes the
governor to apply two different speed caps, depending on whether load is
below or above go_hispeed_load.

Also fix the type of hispeed_freq, which was u64, to match other
speed data types (and avoid overhead and allow division).

Change-Id: Ie2d0668be161c074aaad77db2037505431457b3a
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index f85fbbd..16bd23b 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -63,7 +63,7 @@ static cpumask_t speedchange_cpumask;
 static spinlock_t speedchange_cpumask_lock;
 
 /* Hi speed to bump to from lo speed when load burst (default max) */
-static u64 hispeed_freq;
+static unsigned int hispeed_freq;
 
 /* Go to hi speed when CPU load at or above this value. */
 #define DEFAULT_GO_HISPEED_LOAD 85
@@ -198,7 +198,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 			}
 		}
 	} else {
-		new_freq = pcpu->policy->max * cpu_load / 100;
+		new_freq = hispeed_freq * cpu_load / 100;
 	}
 
 	if (new_freq <= hispeed_freq)
@@ -465,7 +465,7 @@ static void cpufreq_interactive_boost(void)
 static ssize_t show_hispeed_freq(struct kobject *kobj,
 				 struct attribute *attr, char *buf)
 {
-	return sprintf(buf, "%llu\n", hispeed_freq);
+	return sprintf(buf, "%u\n", hispeed_freq);
 }
 
 static ssize_t store_hispeed_freq(struct kobject *kobj,
@@ -473,9 +473,9 @@ static ssize_t store_hispeed_freq(struct kobject *kobj,
 				  size_t count)
 {
 	int ret;
-	u64 val;
+	long unsigned int val;
 
-	ret = strict_strtoull(buf, 0, &val);
+	ret = strict_strtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
 	hispeed_freq = val;
-- 
1.7.9.5

[PATCH 23/70] cpufreq: interactive: pin timers to associated CPU

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Helps avoid waking up other CPUs to react to activity on the local CPU.

Change-Id: Ife272aaa7916894a437705d44521b1a1693fbe8e
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |  106 +++++++++------------------------
 1 file changed, 27 insertions(+), 79 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 16bd23b..e53eae2 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -42,8 +42,6 @@ struct cpufreq_interactive_cpuinfo {
 	int timer_idlecancel;
 	u64 time_in_idle;
 	u64 idle_exit_time;
-	u64 timer_run_time;
-	int idling;
 	u64 target_set_time;
 	u64 target_set_time_in_idle;
 	struct cpufreq_policy *policy;
@@ -109,6 +107,7 @@ struct cpufreq_governor cpufreq_gov_interactive = {
 
 static void cpufreq_interactive_timer(unsigned long data)
 {
+	u64 now;
 	unsigned int delta_idle;
 	unsigned int delta_time;
 	int cpu_load;
@@ -127,26 +126,11 @@ static void cpufreq_interactive_timer(unsigned long data)
 	if (!pcpu->governor_enabled)
 		goto exit;
 
-	/*
-	 * Once pcpu->timer_run_time is updated to >= pcpu->idle_exit_time,
-	 * this lets idle exit know the current idle time sample has
-	 * been processed, and idle exit can generate a new sample and
-	 * re-arm the timer.  This prevents a concurrent idle
-	 * exit on that CPU from writing a new set of info at the same time
-	 * the timer function runs (the timer function can't use that info
-	 * until more time passes).
-	 */
 	time_in_idle = pcpu->time_in_idle;
 	idle_exit_time = pcpu->idle_exit_time;
-	now_idle = get_cpu_idle_time_us(data, &pcpu->timer_run_time);
-	smp_wmb();
-
-	/* If we raced with cancelling a timer, skip. */
-	if (!idle_exit_time)
-		goto exit;
-
+	now_idle = get_cpu_idle_time_us(data, &now);
 	delta_idle = (unsigned int)(now_idle - time_in_idle);
-	delta_time = (unsigned int)(pcpu->timer_run_time - idle_exit_time);
+	delta_time = (unsigned int)(now - idle_exit_time);
 
 	/*
 	 * If timer ran less than 1ms after short-term sample started, retry.
@@ -160,8 +144,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 		cpu_load = 100 * (delta_time - delta_idle) / delta_time;
 
 	delta_idle = (unsigned int)(now_idle - pcpu->target_set_time_in_idle);
-	delta_time = (unsigned int)(pcpu->timer_run_time -
-				    pcpu->target_set_time);
+	delta_time = (unsigned int)(now - pcpu->target_set_time);
 
 	if ((delta_time == 0) || (delta_idle > delta_time))
 		load_since_change = 0;
@@ -189,7 +172,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 			if (pcpu->target_freq == hispeed_freq &&
 			    new_freq > hispeed_freq &&
-			    pcpu->timer_run_time - pcpu->hispeed_validate_time
+			    now - pcpu->hispeed_validate_time
 			    < above_hispeed_delay_val) {
 				trace_cpufreq_interactive_notyet(data, cpu_load,
 								 pcpu->target_freq,
@@ -202,7 +185,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	}
 
 	if (new_freq <= hispeed_freq)
-		pcpu->hispeed_validate_time = pcpu->timer_run_time;
+		pcpu->hispeed_validate_time = now;
 
 	if (cpufreq_frequency_table_target(pcpu->policy, pcpu->freq_table,
 					   new_freq, CPUFREQ_RELATION_H,
@@ -219,8 +202,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	 * floor frequency for the minimum sample time since last validated.
 	 */
 	if (new_freq < pcpu->floor_freq) {
-		if (pcpu->timer_run_time - pcpu->floor_validate_time
-		    < min_sample_time) {
+		if (now - pcpu->floor_validate_time < min_sample_time) {
 			trace_cpufreq_interactive_notyet(data, cpu_load,
 					 pcpu->target_freq, new_freq);
 			goto rearm;
@@ -228,7 +210,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	}
 
 	pcpu->floor_freq = new_freq;
-	pcpu->floor_validate_time = pcpu->timer_run_time;
+	pcpu->floor_validate_time = now;
 
 	if (pcpu->target_freq == new_freq) {
 		trace_cpufreq_interactive_already(data, cpu_load,
@@ -239,7 +221,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	trace_cpufreq_interactive_target(data, cpu_load, pcpu->target_freq,
 					 new_freq);
 	pcpu->target_set_time_in_idle = now_idle;
-	pcpu->target_set_time = pcpu->timer_run_time;
+	pcpu->target_set_time = now;
 
 	pcpu->target_freq = new_freq;
 	spin_lock_irqsave(&speedchange_cpumask_lock, flags);
@@ -258,23 +240,16 @@ rearm_if_notmax:
 rearm:
 	if (!timer_pending(&pcpu->cpu_timer)) {
 		/*
-		 * If already at min: if that CPU is idle, don't set timer.
-		 * Else cancel the timer if that CPU goes idle.  We don't
-		 * need to re-evaluate speed until the next idle exit.
+		 * If already at min, cancel the timer if that CPU goes idle.
+		 * We don't need to re-evaluate speed until the next idle exit.
 		 */
-		if (pcpu->target_freq == pcpu->policy->min) {
-			smp_rmb();
-
-			if (pcpu->idling)
-				goto exit;
-
+		if (pcpu->target_freq == pcpu->policy->min)
 			pcpu->timer_idlecancel = 1;
-		}
 
 		pcpu->time_in_idle = get_cpu_idle_time_us(
 			data, &pcpu->idle_exit_time);
-		mod_timer(&pcpu->cpu_timer,
-			  jiffies + usecs_to_jiffies(timer_rate));
+		mod_timer_pinned(&pcpu->cpu_timer,
+				 jiffies + usecs_to_jiffies(timer_rate));
 	}
 
 exit:
@@ -290,8 +265,6 @@ static void cpufreq_interactive_idle_start(void)
 	if (!pcpu->governor_enabled)
 		return;
 
-	pcpu->idling = 1;
-	smp_wmb();
 	pending = timer_pending(&pcpu->cpu_timer);
 
 	if (pcpu->target_freq != pcpu->policy->min) {
@@ -308,8 +281,9 @@ static void cpufreq_interactive_idle_start(void)
 			pcpu->time_in_idle = get_cpu_idle_time_us(
 				smp_processor_id(), &pcpu->idle_exit_time);
 			pcpu->timer_idlecancel = 0;
-			mod_timer(&pcpu->cpu_timer,
-				  jiffies + usecs_to_jiffies(timer_rate));
+			mod_timer_pinned(
+				&pcpu->cpu_timer,
+				jiffies + usecs_to_jiffies(timer_rate));
 		}
 #endif
 	} else {
@@ -321,12 +295,6 @@ static void cpufreq_interactive_idle_start(void)
 		 */
 		if (pending && pcpu->timer_idlecancel) {
 			del_timer(&pcpu->cpu_timer);
-			/*
-			 * Ensure last timer run time is after current idle
-			 * sample start time, so next idle exit will always
-			 * start a new idle sampling period.
-			 */
-			pcpu->idle_exit_time = 0;
 			pcpu->timer_idlecancel = 0;
 		}
 	}
@@ -341,29 +309,15 @@ static void cpufreq_interactive_idle_end(void)
 	if (!pcpu->governor_enabled)
 		return;
 
-	pcpu->idling = 0;
-	smp_wmb();
-
-	/*
-	 * Arm the timer for 1-2 ticks later if not already, and if the timer
-	 * function has already processed the previous load sampling
-	 * interval.  (If the timer is not pending but has not processed
-	 * the previous interval, it is probably racing with us on another
-	 * CPU.  Let it compute load based on the previous sample and then
-	 * re-arm the timer for another interval when it's done, rather
-	 * than updating the interval start time to be "now", which doesn't
-	 * give the timer function enough time to make a decision on this
-	 * run.)
-	 */
-	if (timer_pending(&pcpu->cpu_timer) == 0 &&
-	    pcpu->timer_run_time >= pcpu->idle_exit_time &&
-	    pcpu->governor_enabled) {
+	/* Arm the timer for 1-2 ticks later if not already. */
+	if (!timer_pending(&pcpu->cpu_timer)) {
 		pcpu->time_in_idle =
 			get_cpu_idle_time_us(smp_processor_id(),
 					     &pcpu->idle_exit_time);
 		pcpu->timer_idlecancel = 0;
-		mod_timer(&pcpu->cpu_timer,
-			  jiffies + usecs_to_jiffies(timer_rate));
+		mod_timer_pinned(
+			&pcpu->cpu_timer,
+			jiffies + usecs_to_jiffies(timer_rate));
 	}
 
 }
@@ -673,6 +627,8 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 
 		freq_table =
 			cpufreq_frequency_get_table(policy->cpu);
+		if (!hispeed_freq)
+			hispeed_freq = policy->max;
 
 		for_each_cpu(j, policy->cpus) {
 			pcpu = &per_cpu(cpuinfo, j);
@@ -689,11 +645,11 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 				pcpu->target_set_time;
 			pcpu->governor_enabled = 1;
 			smp_wmb();
+			pcpu->cpu_timer.expires =
+				jiffies + usecs_to_jiffies(timer_rate);
+			add_timer_on(&pcpu->cpu_timer, j);
 		}
 
-		if (!hispeed_freq)
-			hispeed_freq = policy->max;
-
 		/*
 		 * Do not register the idle hook and create sysfs
 		 * entries if we have already done so.
@@ -715,14 +671,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->governor_enabled = 0;
 			smp_wmb();
 			del_timer_sync(&pcpu->cpu_timer);
-
-			/*
-			 * Reset idle exit time since we may cancel the timer
-			 * before it can run after the last idle exit time,
-			 * to avoid tripping the check in idle exit for a timer
-			 * that is trying to run.
-			 */
-			pcpu->idle_exit_time = 0;
 		}
 
 		if (atomic_dec_return(&active_count) > 0)
-- 
1.7.9.5

[PATCH 24/70] cpufreq: interactive: use deferrable timer by default

[Bálint Czobor]

From: Lianwei Wang <a22439@motorola.com>

Avoid wakeups only to handle the governor timer when the system is otherwise
idle.

For platforms where the power cost of remaining in idle at higher CPU
speed may outweigh the cost of a governor wakeup from idle to lower the speed,
set parameter cpufreq_interactive.governidle=1.

Change-Id: Id6c43eb35caecf9b0574fcdd5b769711bc7e6de6
Signed-off-by: LianWei WANG <a22439@motorola.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index e53eae2..71887ca 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -20,6 +20,7 @@
 #include <linux/cpumask.h>
 #include <linux/cpufreq.h>
 #include <linux/module.h>
+#include <linux/moduleparam.h>
 #include <linux/mutex.h>
 #include <linux/sched.h>
 #include <linux/sched/rt.h>
@@ -92,6 +93,11 @@ static unsigned long above_hispeed_delay_val;
 
 static int boost_val;
 
+static bool governidle;
+module_param(governidle, bool, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(governidle,
+	"Set to 1 to wake up CPUs from idle to reduce speed (default 0)");
+
 static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		unsigned int event);
 
@@ -240,10 +246,11 @@ rearm_if_notmax:
 rearm:
 	if (!timer_pending(&pcpu->cpu_timer)) {
 		/*
-		 * If already at min, cancel the timer if that CPU goes idle.
-		 * We don't need to re-evaluate speed until the next idle exit.
+		 * If governing speed in idle and already at min, cancel the
+		 * timer if that CPU goes idle.  We don't need to re-evaluate
+		 * speed until the next idle exit.
 		 */
-		if (pcpu->target_freq == pcpu->policy->min)
+		if (governidle && pcpu->target_freq == pcpu->policy->min)
 			pcpu->timer_idlecancel = 1;
 
 		pcpu->time_in_idle = get_cpu_idle_time_us(
@@ -268,7 +275,6 @@ static void cpufreq_interactive_idle_start(void)
 	pending = timer_pending(&pcpu->cpu_timer);
 
 	if (pcpu->target_freq != pcpu->policy->min) {
-#ifdef CONFIG_SMP
 		/*
 		 * Entering idle while not at lowest speed.  On some
 		 * platforms this can hold the other CPU(s) at that speed
@@ -285,8 +291,7 @@ static void cpufreq_interactive_idle_start(void)
 				&pcpu->cpu_timer,
 				jiffies + usecs_to_jiffies(timer_rate));
 		}
-#endif
-	} else {
+	} else if (governidle) {
 		/*
 		 * If at min speed and entering idle after load has
 		 * already been evaluated, and a timer has been set just in
@@ -708,7 +713,10 @@ static int __init cpufreq_interactive_init(void)
 	/* Initalize per-cpu timers */
 	for_each_possible_cpu(i) {
 		pcpu = &per_cpu(cpuinfo, i);
-		init_timer(&pcpu->cpu_timer);
+		if (governidle)
+			init_timer(&pcpu->cpu_timer);
+		else
+			init_timer_deferrable(&pcpu->cpu_timer);
 		pcpu->cpu_timer.function = cpufreq_interactive_timer;
 		pcpu->cpu_timer.data = i;
 	}
-- 
1.7.9.5

[PATCH 25/70] cpufreq: interactive: kick timer on idle exit past expiry

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

The deferrable timer list isn't checked on all idle exits, such as when
hi-res timers expire or ISRs schedule workers.  If the idle loop is
exited and it's past time to run the governor load polling timer,
run it immediately.  This ensures we handle load spikes caused by actvity
that does not run the normal timer list.

Rename the field that timestamps the "time_in_idle" value to be more
accurate.

Change-Id: Ied590ecbefc83c9a9ec5eb9e31903557f6fa1614
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   43 +++++++++++++++------------------
 1 file changed, 20 insertions(+), 23 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 71887ca..8153769 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -42,7 +42,7 @@ struct cpufreq_interactive_cpuinfo {
 	struct timer_list cpu_timer;
 	int timer_idlecancel;
 	u64 time_in_idle;
-	u64 idle_exit_time;
+	u64 time_in_idle_timestamp;
 	u64 target_set_time;
 	u64 target_set_time_in_idle;
 	struct cpufreq_policy *policy;
@@ -111,6 +111,16 @@ struct cpufreq_governor cpufreq_gov_interactive = {
 	.owner = THIS_MODULE,
 };
 
+static void cpufreq_interactive_timer_resched(
+	struct cpufreq_interactive_cpuinfo *pcpu)
+{
+	mod_timer_pinned(&pcpu->cpu_timer,
+			 jiffies + usecs_to_jiffies(timer_rate));
+	pcpu->time_in_idle =
+		get_cpu_idle_time_us(smp_processor_id(),
+				     &pcpu->time_in_idle_timestamp);
+}
+
 static void cpufreq_interactive_timer(unsigned long data)
 {
 	u64 now;
@@ -118,8 +128,6 @@ static void cpufreq_interactive_timer(unsigned long data)
 	unsigned int delta_time;
 	int cpu_load;
 	int load_since_change;
-	u64 time_in_idle;
-	u64 idle_exit_time;
 	struct cpufreq_interactive_cpuinfo *pcpu =
 		&per_cpu(cpuinfo, data);
 	u64 now_idle;
@@ -132,11 +140,9 @@ static void cpufreq_interactive_timer(unsigned long data)
 	if (!pcpu->governor_enabled)
 		goto exit;
 
-	time_in_idle = pcpu->time_in_idle;
-	idle_exit_time = pcpu->idle_exit_time;
 	now_idle = get_cpu_idle_time_us(data, &now);
-	delta_idle = (unsigned int)(now_idle - time_in_idle);
-	delta_time = (unsigned int)(now - idle_exit_time);
+	delta_idle = (unsigned int)(now_idle - pcpu->time_in_idle);
+	delta_time = (unsigned int)(now - pcpu->time_in_idle_timestamp);
 
 	/*
 	 * If timer ran less than 1ms after short-term sample started, retry.
@@ -253,10 +259,7 @@ rearm:
 		if (governidle && pcpu->target_freq == pcpu->policy->min)
 			pcpu->timer_idlecancel = 1;
 
-		pcpu->time_in_idle = get_cpu_idle_time_us(
-			data, &pcpu->idle_exit_time);
-		mod_timer_pinned(&pcpu->cpu_timer,
-				 jiffies + usecs_to_jiffies(timer_rate));
+		cpufreq_interactive_timer_resched(pcpu);
 	}
 
 exit:
@@ -284,12 +287,8 @@ static void cpufreq_interactive_idle_start(void)
 		 * the CPUFreq driver.
 		 */
 		if (!pending) {
-			pcpu->time_in_idle = get_cpu_idle_time_us(
-				smp_processor_id(), &pcpu->idle_exit_time);
 			pcpu->timer_idlecancel = 0;
-			mod_timer_pinned(
-				&pcpu->cpu_timer,
-				jiffies + usecs_to_jiffies(timer_rate));
+			cpufreq_interactive_timer_resched(pcpu);
 		}
 	} else if (governidle) {
 		/*
@@ -316,15 +315,13 @@ static void cpufreq_interactive_idle_end(void)
 
 	/* Arm the timer for 1-2 ticks later if not already. */
 	if (!timer_pending(&pcpu->cpu_timer)) {
-		pcpu->time_in_idle =
-			get_cpu_idle_time_us(smp_processor_id(),
-					     &pcpu->idle_exit_time);
 		pcpu->timer_idlecancel = 0;
-		mod_timer_pinned(
-			&pcpu->cpu_timer,
-			jiffies + usecs_to_jiffies(timer_rate));
+		cpufreq_interactive_timer_resched(pcpu);
+	} else if (!governidle &&
+		   time_after_eq(jiffies, pcpu->cpu_timer.expires)) {
+		del_timer(&pcpu->cpu_timer);
+		cpufreq_interactive_timer(smp_processor_id());
 	}
-
 }
 
 static int cpufreq_interactive_speedchange_task(void *data)
-- 
1.7.9.5

[PATCH 26/70] cpufreq: interactive: trace actual speed in target speed decisions

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Tracing adds actual speed since this is expected to be key to the
choice of target speed.

Change-Id: Iec936102d0010c4e9dfa143c38a9fd0d551189c3
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c      |   18 +++++++-------
 include/trace/events/cpufreq_interactive.h |   36 ++++++++++++++++------------
 2 files changed, 31 insertions(+), 23 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 8153769..2b4aad9 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -186,9 +186,9 @@ static void cpufreq_interactive_timer(unsigned long data)
 			    new_freq > hispeed_freq &&
 			    now - pcpu->hispeed_validate_time
 			    < above_hispeed_delay_val) {
-				trace_cpufreq_interactive_notyet(data, cpu_load,
-								 pcpu->target_freq,
-								 new_freq);
+				trace_cpufreq_interactive_notyet(
+					data, cpu_load, pcpu->target_freq,
+					pcpu->policy->cur, new_freq);
 				goto rearm;
 			}
 		}
@@ -215,8 +215,9 @@ static void cpufreq_interactive_timer(unsigned long data)
 	 */
 	if (new_freq < pcpu->floor_freq) {
 		if (now - pcpu->floor_validate_time < min_sample_time) {
-			trace_cpufreq_interactive_notyet(data, cpu_load,
-					 pcpu->target_freq, new_freq);
+			trace_cpufreq_interactive_notyet(
+				data, cpu_load, pcpu->target_freq,
+				pcpu->policy->cur, new_freq);
 			goto rearm;
 		}
 	}
@@ -225,13 +226,14 @@ static void cpufreq_interactive_timer(unsigned long data)
 	pcpu->floor_validate_time = now;
 
 	if (pcpu->target_freq == new_freq) {
-		trace_cpufreq_interactive_already(data, cpu_load,
-						  pcpu->target_freq, new_freq);
+		trace_cpufreq_interactive_already(
+			data, cpu_load, pcpu->target_freq,
+			pcpu->policy->cur, new_freq);
 		goto rearm_if_notmax;
 	}
 
 	trace_cpufreq_interactive_target(data, cpu_load, pcpu->target_freq,
-					 new_freq);
+					 pcpu->policy->cur, new_freq);
 	pcpu->target_set_time_in_idle = now_idle;
 	pcpu->target_set_time = now;
 
diff --git a/include/trace/events/cpufreq_interactive.h b/include/trace/events/cpufreq_interactive.h
index ba0634e..b2ce6b0 100644
--- a/include/trace/events/cpufreq_interactive.h
+++ b/include/trace/events/cpufreq_interactive.h
@@ -36,44 +36,50 @@ DEFINE_EVENT(set, cpufreq_interactive_setspeed,
 
 DECLARE_EVENT_CLASS(loadeval,
 	    TP_PROTO(unsigned long cpu_id, unsigned long load,
-		     unsigned long curfreq, unsigned long targfreq),
-	    TP_ARGS(cpu_id, load, curfreq, targfreq),
+		     unsigned long curtarg, unsigned long curactual,
+		     unsigned long newtarg),
+		    TP_ARGS(cpu_id, load, curtarg, curactual, newtarg),
 
 	    TP_STRUCT__entry(
 		    __field(unsigned long, cpu_id    )
 		    __field(unsigned long, load      )
-		    __field(unsigned long, curfreq   )
-		    __field(unsigned long, targfreq  )
+		    __field(unsigned long, curtarg   )
+		    __field(unsigned long, curactual )
+		    __field(unsigned long, newtarg   )
 	    ),
 
 	    TP_fast_assign(
 		    __entry->cpu_id = cpu_id;
 		    __entry->load = load;
-		    __entry->curfreq = curfreq;
-		    __entry->targfreq = targfreq;
+		    __entry->curtarg = curtarg;
+		    __entry->curactual = curactual;
+		    __entry->newtarg = newtarg;
 	    ),
 
-	    TP_printk("cpu=%lu load=%lu cur=%lu targ=%lu",
-		      __entry->cpu_id, __entry->load, __entry->curfreq,
-		      __entry->targfreq)
+	    TP_printk("cpu=%lu load=%lu cur=%lu actual=%lu targ=%lu",
+		      __entry->cpu_id, __entry->load, __entry->curtarg,
+		      __entry->curactual, __entry->newtarg)
 );
 
 DEFINE_EVENT(loadeval, cpufreq_interactive_target,
 	    TP_PROTO(unsigned long cpu_id, unsigned long load,
-		     unsigned long curfreq, unsigned long targfreq),
-	    TP_ARGS(cpu_id, load, curfreq, targfreq)
+		     unsigned long curtarg, unsigned long curactual,
+		     unsigned long newtarg),
+	    TP_ARGS(cpu_id, load, curtarg, curactual, newtarg)
 );
 
 DEFINE_EVENT(loadeval, cpufreq_interactive_already,
 	    TP_PROTO(unsigned long cpu_id, unsigned long load,
-		     unsigned long curfreq, unsigned long targfreq),
-	    TP_ARGS(cpu_id, load, curfreq, targfreq)
+		     unsigned long curtarg, unsigned long curactual,
+		     unsigned long newtarg),
+	    TP_ARGS(cpu_id, load, curtarg, curactual, newtarg)
 );
 
 DEFINE_EVENT(loadeval, cpufreq_interactive_notyet,
 	    TP_PROTO(unsigned long cpu_id, unsigned long load,
-		     unsigned long curfreq, unsigned long targfreq),
-	    TP_ARGS(cpu_id, load, curfreq, targfreq)
+		     unsigned long curtarg, unsigned long curactual,
+		     unsigned long newtarg),
+	    TP_ARGS(cpu_id, load, curtarg, curactual, newtarg)
 );
 
 TRACE_EVENT(cpufreq_interactive_boost,
-- 
1.7.9.5

[PATCH 27/70] cpufreq: interactive: change speed according to current speed and target load

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Add a target_load attribute that specifies how aggressively the governor is
to adjust speed to meet the observed load.  New target speed is calculated
as the current actual speed (may be higher than target speed on SMP) times
the CPU load (as a fraction) divided by target load (fraction).

cpufreq_frequency_table_target() call use CPUFREQ_RELATION_L to set
the next higher speed rather than next lower speed.

Change-Id: If432451da82f5fed12e15c9421d7d27792376150
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   35 ++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 2b4aad9..c4293c5 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -68,6 +68,10 @@ static unsigned int hispeed_freq;
 #define DEFAULT_GO_HISPEED_LOAD 85
 static unsigned long go_hispeed_load;
 
+/* Target load.  Lower values result in higher CPU speeds. */
+#define DEFAULT_TARGET_LOAD 90
+static unsigned long target_load = DEFAULT_TARGET_LOAD;
+
 /*
  * The minimum amount of time to spend at a frequency before we can ramp down.
  */
@@ -177,7 +181,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 		    hispeed_freq < pcpu->policy->max) {
 			new_freq = hispeed_freq;
 		} else {
-			new_freq = pcpu->policy->max * cpu_load / 100;
+			new_freq = pcpu->policy->cur * cpu_load / target_load;
 
 			if (new_freq < hispeed_freq)
 				new_freq = hispeed_freq;
@@ -193,14 +197,14 @@ static void cpufreq_interactive_timer(unsigned long data)
 			}
 		}
 	} else {
-		new_freq = hispeed_freq * cpu_load / 100;
+		new_freq = pcpu->policy->cur * cpu_load / target_load;
 	}
 
 	if (new_freq <= hispeed_freq)
 		pcpu->hispeed_validate_time = now;
 
 	if (cpufreq_frequency_table_target(pcpu->policy, pcpu->freq_table,
-					   new_freq, CPUFREQ_RELATION_H,
+					   new_freq, CPUFREQ_RELATION_L,
 					   &index)) {
 		pr_warn_once("timer %d: cpufreq_frequency_table_target error\n",
 			     (int) data);
@@ -420,6 +424,30 @@ static void cpufreq_interactive_boost(void)
 		wake_up_process(speedchange_task);
 }
 
+static ssize_t show_target_load(
+	struct kobject *kobj, struct attribute *attr, char *buf)
+{
+	return sprintf(buf, "%lu\n", target_load);
+}
+
+static ssize_t store_target_load(
+	struct kobject *kobj, struct attribute *attr, const char *buf,
+	size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = strict_strtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+	target_load = val;
+	return count;
+}
+
+static struct global_attr target_load_attr =
+	__ATTR(target_load, S_IRUGO | S_IWUSR,
+		show_target_load, store_target_load);
+
 static ssize_t show_hispeed_freq(struct kobject *kobj,
 				 struct attribute *attr, char *buf)
 {
@@ -581,6 +609,7 @@ static struct global_attr boostpulse =
 	__ATTR(boostpulse, 0200, NULL, store_boostpulse);
 
 static struct attribute *interactive_attributes[] = {
+	&target_load_attr.attr,
 	&hispeed_freq_attr.attr,
 	&go_hispeed_load_attr.attr,
 	&above_hispeed_delay.attr,
-- 
1.7.9.5

[PATCH 28/70] cpufreq: interactive: apply above_hispeed_delay to each step above hispeed

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Apply above_hispeed_delay whenever increasing speed to a new speed above
hispeed (not just the first step above hispeed).

Change-Id: Ibb7add7db47f2a4306a9458c4e1ebabb60698636
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   36 ++++++++++++---------------------
 1 file changed, 13 insertions(+), 23 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index c4293c5..17c42cc 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -176,32 +176,22 @@ static void cpufreq_interactive_timer(unsigned long data)
 	if (load_since_change > cpu_load)
 		cpu_load = load_since_change;
 
-	if (cpu_load >= go_hispeed_load || boost_val) {
-		if (pcpu->target_freq < hispeed_freq &&
-		    hispeed_freq < pcpu->policy->max) {
-			new_freq = hispeed_freq;
-		} else {
-			new_freq = pcpu->policy->cur * cpu_load / target_load;
-
-			if (new_freq < hispeed_freq)
-				new_freq = hispeed_freq;
-
-			if (pcpu->target_freq == hispeed_freq &&
-			    new_freq > hispeed_freq &&
-			    now - pcpu->hispeed_validate_time
-			    < above_hispeed_delay_val) {
-				trace_cpufreq_interactive_notyet(
-					data, cpu_load, pcpu->target_freq,
-					pcpu->policy->cur, new_freq);
-				goto rearm;
-			}
-		}
-	} else {
+	if ((cpu_load >= go_hispeed_load || boost_val) &&
+	    pcpu->target_freq < hispeed_freq)
+		new_freq = hispeed_freq;
+	else
 		new_freq = pcpu->policy->cur * cpu_load / target_load;
+
+	if (pcpu->target_freq >= hispeed_freq &&
+	    new_freq > pcpu->target_freq &&
+	    now - pcpu->hispeed_validate_time < above_hispeed_delay_val) {
+		trace_cpufreq_interactive_notyet(
+			data, cpu_load, pcpu->target_freq,
+			pcpu->policy->cur, new_freq);
+		goto rearm;
 	}
 
-	if (new_freq <= hispeed_freq)
-		pcpu->hispeed_validate_time = now;
+	pcpu->hispeed_validate_time = now;
 
 	if (cpufreq_frequency_table_target(pcpu->policy, pcpu->freq_table,
 					   new_freq, CPUFREQ_RELATION_L,
-- 
1.7.9.5

[PATCH 29/70] cpufreq: interactive: allow arbitrary speed / target load mappings

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Accept a string of target loads and speeds at which to apply the
target loads, per the documentation update in this patch.  For example,
"85 1000000:90 1700000:99" targets CPU load 85% below speed 1GHz,  90%
at or above 1GHz, until 1.7GHz and above, at which load 99% is targeted.

Attempt to avoid oscillations by evaluating the current speed
weighted by current load against each new choice of speed, choosing a
higher speed if the current load requires a higher speed.

Change-Id: Ie3300206047c84eca5a26b0b63ea512e5207550e
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 Documentation/cpu-freq/governors.txt  |   17 +++
 drivers/cpufreq/cpufreq_interactive.c |  186 ++++++++++++++++++++++++++++++---
 2 files changed, 189 insertions(+), 14 deletions(-)

diff --git a/Documentation/cpu-freq/governors.txt b/Documentation/cpu-freq/governors.txt
index 509b179..b15f6d2 100644
--- a/Documentation/cpu-freq/governors.txt
+++ b/Documentation/cpu-freq/governors.txt
@@ -244,6 +244,23 @@ short-term load since idle exit to determine the cpu speed to ramp to.
 
 The tuneable values for this governor are:
 
+target_loads: CPU load values used to adjust speed to influence the
+current CPU load toward that value.  In general, the lower the target
+load, the more often the governor will raise CPU speeds to bring load
+below the target.  The format is a single target load, optionally
+followed by pairs of CPU speeds and CPU loads to target at or above
+those speeds.  Colons can be used between the speeds and associated
+target loads for readability.  For example:
+
+   85 1000000:90 1700000:99
+
+targets CPU load 85% below speed 1GHz, 90% at or above 1GHz, until
+1.7GHz and above, at which load 99% is targeted.  If speeds are
+specified these must appear in ascending order.  Higher target load
+values are typically specified for higher speeds, that is, target load
+values also usually appear in an ascending order. The default is
+target load 90% for all speeds.
+
 min_sample_time: The minimum amount of time to spend at the current
 frequency before ramping down. This is to ensure that the governor has
 seen enough historic cpu load data to determine the appropriate
diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 17c42cc..6ea77a0 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -70,7 +70,10 @@ static unsigned long go_hispeed_load;
 
 /* Target load.  Lower values result in higher CPU speeds. */
 #define DEFAULT_TARGET_LOAD 90
-static unsigned long target_load = DEFAULT_TARGET_LOAD;
+static unsigned int default_target_loads[] = {DEFAULT_TARGET_LOAD};
+static spinlock_t target_loads_lock;
+static unsigned int *target_loads = default_target_loads;
+static int ntarget_loads = ARRAY_SIZE(default_target_loads);
 
 /*
  * The minimum amount of time to spend at a frequency before we can ramp down.
@@ -125,6 +128,110 @@ static void cpufreq_interactive_timer_resched(
 				     &pcpu->time_in_idle_timestamp);
 }
 
+static unsigned int freq_to_targetload(unsigned int freq)
+{
+	int i;
+	unsigned int ret;
+
+	spin_lock(&target_loads_lock);
+
+	for (i = 0; i < ntarget_loads - 1 && freq >= target_loads[i+1]; i += 2)
+		;
+
+	ret = target_loads[i];
+	spin_unlock(&target_loads_lock);
+	return ret;
+}
+
+/*
+ * If increasing frequencies never map to a lower target load then
+ * choose_freq() will find the minimum frequency that does not exceed its
+ * target load given the current load.
+ */
+
+static unsigned int choose_freq(
+	struct cpufreq_interactive_cpuinfo *pcpu, unsigned int curload)
+{
+	unsigned int freq = pcpu->policy->cur;
+	unsigned int loadadjfreq = freq * curload;
+	unsigned int prevfreq, freqmin, freqmax;
+	unsigned int tl;
+	int index;
+
+	freqmin = 0;
+	freqmax = UINT_MAX;
+
+	do {
+		prevfreq = freq;
+		tl = freq_to_targetload(freq);
+
+		/*
+		 * Find the lowest frequency where the computed load is less
+		 * than or equal to the target load.
+		 */
+
+		cpufreq_frequency_table_target(
+			pcpu->policy, pcpu->freq_table, loadadjfreq / tl,
+			CPUFREQ_RELATION_L, &index);
+		freq = pcpu->freq_table[index].frequency;
+
+		if (freq > prevfreq) {
+			/* The previous frequency is too low. */
+			freqmin = prevfreq;
+
+			if (freq >= freqmax) {
+				/*
+				 * Find the highest frequency that is less
+				 * than freqmax.
+				 */
+				cpufreq_frequency_table_target(
+					pcpu->policy, pcpu->freq_table,
+					freqmax - 1, CPUFREQ_RELATION_H,
+					&index);
+				freq = pcpu->freq_table[index].frequency;
+
+				if (freq == freqmin) {
+					/*
+					 * The first frequency below freqmax
+					 * has already been found to be too
+					 * low.  freqmax is the lowest speed
+					 * we found that is fast enough.
+					 */
+					freq = freqmax;
+					break;
+				}
+			}
+		} else if (freq < prevfreq) {
+			/* The previous frequency is high enough. */
+			freqmax = prevfreq;
+
+			if (freq <= freqmin) {
+				/*
+				 * Find the lowest frequency that is higher
+				 * than freqmin.
+				 */
+				cpufreq_frequency_table_target(
+					pcpu->policy, pcpu->freq_table,
+					freqmin + 1, CPUFREQ_RELATION_L,
+					&index);
+				freq = pcpu->freq_table[index].frequency;
+
+				/*
+				 * If freqmax is the first frequency above
+				 * freqmin then we have already found that
+				 * this speed is fast enough.
+				 */
+				if (freq == freqmax)
+					break;
+			}
+		}
+
+		/* If same frequency chosen as previous then done. */
+	} while (freq != prevfreq);
+
+	return freq;
+}
+
 static void cpufreq_interactive_timer(unsigned long data)
 {
 	u64 now;
@@ -180,7 +287,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	    pcpu->target_freq < hispeed_freq)
 		new_freq = hispeed_freq;
 	else
-		new_freq = pcpu->policy->cur * cpu_load / target_load;
+		new_freq = choose_freq(pcpu, cpu_load);
 
 	if (pcpu->target_freq >= hispeed_freq &&
 	    new_freq > pcpu->target_freq &&
@@ -414,29 +521,79 @@ static void cpufreq_interactive_boost(void)
 		wake_up_process(speedchange_task);
 }
 
-static ssize_t show_target_load(
+static ssize_t show_target_loads(
 	struct kobject *kobj, struct attribute *attr, char *buf)
 {
-	return sprintf(buf, "%lu\n", target_load);
+	int i;
+	ssize_t ret = 0;
+
+	spin_lock(&target_loads_lock);
+
+	for (i = 0; i < ntarget_loads; i++)
+		ret += sprintf(buf + ret, "%u%s", target_loads[i],
+			       i & 0x1 ? ":" : " ");
+
+	ret += sprintf(buf + ret, "\n");
+	spin_unlock(&target_loads_lock);
+	return ret;
 }
 
-static ssize_t store_target_load(
+static ssize_t store_target_loads(
 	struct kobject *kobj, struct attribute *attr, const char *buf,
 	size_t count)
 {
 	int ret;
-	unsigned long val;
+	const char *cp;
+	unsigned int *new_target_loads = NULL;
+	int ntokens = 1;
+	int i;
 
-	ret = strict_strtoul(buf, 0, &val);
-	if (ret < 0)
-		return ret;
-	target_load = val;
+	cp = buf;
+	while ((cp = strpbrk(cp + 1, " :")))
+		ntokens++;
+
+	if (!(ntokens & 0x1))
+		goto err_inval;
+
+	new_target_loads = kmalloc(ntokens * sizeof(unsigned int), GFP_KERNEL);
+	if (!new_target_loads) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
+	cp = buf;
+	i = 0;
+	while (i < ntokens) {
+		if (sscanf(cp, "%u", &new_target_loads[i++]) != 1)
+			goto err_inval;
+
+		cp = strpbrk(cp, " :");
+		if (!cp)
+			break;
+		cp++;
+	}
+
+	if (i != ntokens)
+		goto err_inval;
+
+	spin_lock(&target_loads_lock);
+	if (target_loads != default_target_loads)
+		kfree(target_loads);
+	target_loads = new_target_loads;
+	ntarget_loads = ntokens;
+	spin_unlock(&target_loads_lock);
 	return count;
+
+err_inval:
+	ret = -EINVAL;
+err:
+	kfree(new_target_loads);
+	return ret;
 }
 
-static struct global_attr target_load_attr =
-	__ATTR(target_load, S_IRUGO | S_IWUSR,
-		show_target_load, store_target_load);
+static struct global_attr target_loads_attr =
+	__ATTR(target_loads, S_IRUGO | S_IWUSR,
+		show_target_loads, store_target_loads);
 
 static ssize_t show_hispeed_freq(struct kobject *kobj,
 				 struct attribute *attr, char *buf)
@@ -599,7 +756,7 @@ static struct global_attr boostpulse =
 	__ATTR(boostpulse, 0200, NULL, store_boostpulse);
 
 static struct attribute *interactive_attributes[] = {
-	&target_load_attr.attr,
+	&target_loads_attr.attr,
 	&hispeed_freq_attr.attr,
 	&go_hispeed_load_attr.attr,
 	&above_hispeed_delay.attr,
@@ -739,6 +896,7 @@ static int __init cpufreq_interactive_init(void)
 		pcpu->cpu_timer.data = i;
 	}
 
+	spin_lock_init(&target_loads_lock);
 	spin_lock_init(&speedchange_cpumask_lock);
 	speedchange_task =
 		kthread_create(cpufreq_interactive_speedchange_task, NULL,
-- 
1.7.9.5

[PATCH 30/70] cpufreq: interactive: remove load since last speed change

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

The longer-term load since last speed change isn't terribly useful,
may delay recognition of dropping load, and would need forthcoming
changes to adjust load for changing CPU speeds.  Drop it.

Change-Id: Ic3cbb0542cc3484617031787e03ed9bdd632dec1
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   34 ++++-----------------------------
 1 file changed, 4 insertions(+), 30 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 6ea77a0..3d8e7b4 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -43,8 +43,6 @@ struct cpufreq_interactive_cpuinfo {
 	int timer_idlecancel;
 	u64 time_in_idle;
 	u64 time_in_idle_timestamp;
-	u64 target_set_time;
-	u64 target_set_time_in_idle;
 	struct cpufreq_policy *policy;
 	struct cpufreq_frequency_table *freq_table;
 	unsigned int target_freq;
@@ -238,7 +236,6 @@ static void cpufreq_interactive_timer(unsigned long data)
 	unsigned int delta_idle;
 	unsigned int delta_time;
 	int cpu_load;
-	int load_since_change;
 	struct cpufreq_interactive_cpuinfo *pcpu =
 		&per_cpu(cpuinfo, data);
 	u64 now_idle;
@@ -266,23 +263,6 @@ static void cpufreq_interactive_timer(unsigned long data)
 	else
 		cpu_load = 100 * (delta_time - delta_idle) / delta_time;
 
-	delta_idle = (unsigned int)(now_idle - pcpu->target_set_time_in_idle);
-	delta_time = (unsigned int)(now - pcpu->target_set_time);
-
-	if ((delta_time == 0) || (delta_idle > delta_time))
-		load_since_change = 0;
-	else
-		load_since_change =
-			100 * (delta_time - delta_idle) / delta_time;
-
-	/*
-	 * Choose greater of short-term load (since last idle timer
-	 * started or timer function re-armed itself) or long-term load
-	 * (since last frequency change).
-	 */
-	if (load_since_change > cpu_load)
-		cpu_load = load_since_change;
-
 	if ((cpu_load >= go_hispeed_load || boost_val) &&
 	    pcpu->target_freq < hispeed_freq)
 		new_freq = hispeed_freq;
@@ -335,8 +315,6 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 	trace_cpufreq_interactive_target(data, cpu_load, pcpu->target_freq,
 					 pcpu->policy->cur, new_freq);
-	pcpu->target_set_time_in_idle = now_idle;
-	pcpu->target_set_time = now;
 
 	pcpu->target_freq = new_freq;
 	spin_lock_irqsave(&speedchange_cpumask_lock, flags);
@@ -500,9 +478,8 @@ static void cpufreq_interactive_boost(void)
 		if (pcpu->target_freq < hispeed_freq) {
 			pcpu->target_freq = hispeed_freq;
 			cpumask_set_cpu(i, &speedchange_cpumask);
-			pcpu->target_set_time_in_idle =
-				get_cpu_idle_time_us(i, &pcpu->target_set_time);
-			pcpu->hispeed_validate_time = pcpu->target_set_time;
+			pcpu->hispeed_validate_time =
+				ktime_to_us(ktime_get());
 			anyboost = 1;
 		}
 
@@ -815,14 +792,11 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->policy = policy;
 			pcpu->target_freq = policy->cur;
 			pcpu->freq_table = freq_table;
-			pcpu->target_set_time_in_idle =
-				get_cpu_idle_time_us(j,
-					     &pcpu->target_set_time);
 			pcpu->floor_freq = pcpu->target_freq;
 			pcpu->floor_validate_time =
-				pcpu->target_set_time;
+				ktime_to_us(ktime_get());
 			pcpu->hispeed_validate_time =
-				pcpu->target_set_time;
+				pcpu->floor_validate_time;
 			pcpu->governor_enabled = 1;
 			smp_wmb();
 			pcpu->cpu_timer.expires =
-- 
1.7.9.5

[PATCH 31/70] cpufreq: interactive: adjust load for changes in speed

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Add notifier for speed transitions.  Keep a count of CPU active
microseconds times current frequency, converted to a percentage relative
to the current frequency when load is evaluated.

Change-Id: I5c27adb11081c50490219784ca57cc46e97fc28c
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   87 +++++++++++++++++++++++++++------
 1 file changed, 71 insertions(+), 16 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 3d8e7b4..d0d51ee 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -41,8 +41,11 @@ static atomic_t active_count = ATOMIC_INIT(0);
 struct cpufreq_interactive_cpuinfo {
 	struct timer_list cpu_timer;
 	int timer_idlecancel;
+	spinlock_t load_lock; /* protects the next 4 fields */
 	u64 time_in_idle;
 	u64 time_in_idle_timestamp;
+	u64 cputime_speedadj;
+	u64 cputime_speedadj_timestamp;
 	struct cpufreq_policy *policy;
 	struct cpufreq_frequency_table *freq_table;
 	unsigned int target_freq;
@@ -121,9 +124,13 @@ static void cpufreq_interactive_timer_resched(
 {
 	mod_timer_pinned(&pcpu->cpu_timer,
 			 jiffies + usecs_to_jiffies(timer_rate));
+	spin_lock(&pcpu->load_lock);
 	pcpu->time_in_idle =
 		get_cpu_idle_time_us(smp_processor_id(),
 				     &pcpu->time_in_idle_timestamp);
+	pcpu->cputime_speedadj = 0;
+	pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp;
+	spin_unlock(&pcpu->load_lock);
 }
 
 static unsigned int freq_to_targetload(unsigned int freq)
@@ -148,10 +155,9 @@ static unsigned int freq_to_targetload(unsigned int freq)
  */
 
 static unsigned int choose_freq(
-	struct cpufreq_interactive_cpuinfo *pcpu, unsigned int curload)
+	struct cpufreq_interactive_cpuinfo *pcpu, unsigned int loadadjfreq)
 {
 	unsigned int freq = pcpu->policy->cur;
-	unsigned int loadadjfreq = freq * curload;
 	unsigned int prevfreq, freqmin, freqmax;
 	unsigned int tl;
 	int index;
@@ -230,16 +236,36 @@ static unsigned int choose_freq(
 	return freq;
 }
 
-static void cpufreq_interactive_timer(unsigned long data)
+static u64 update_load(int cpu)
 {
+	struct cpufreq_interactive_cpuinfo *pcpu = &per_cpu(cpuinfo, cpu);
 	u64 now;
+	u64 now_idle;
 	unsigned int delta_idle;
 	unsigned int delta_time;
+	u64 active_time;
+
+	now_idle = get_cpu_idle_time_us(cpu, &now);
+	delta_idle = (unsigned int)(now_idle - pcpu->time_in_idle);
+	delta_time = (unsigned int)(now - pcpu->time_in_idle_timestamp);
+	active_time = delta_time - delta_idle;
+	pcpu->cputime_speedadj += active_time * pcpu->policy->cur;
+
+	pcpu->time_in_idle = now_idle;
+	pcpu->time_in_idle_timestamp = now;
+	return now;
+}
+
+static void cpufreq_interactive_timer(unsigned long data)
+{
+	u64 now;
+	unsigned int delta_time;
+	u64 cputime_speedadj;
 	int cpu_load;
 	struct cpufreq_interactive_cpuinfo *pcpu =
 		&per_cpu(cpuinfo, data);
-	u64 now_idle;
 	unsigned int new_freq;
+	unsigned int loadadjfreq;
 	unsigned int index;
 	unsigned long flags;
 
@@ -248,26 +274,24 @@ static void cpufreq_interactive_timer(unsigned long data)
 	if (!pcpu->governor_enabled)
 		goto exit;
 
-	now_idle = get_cpu_idle_time_us(data, &now);
-	delta_idle = (unsigned int)(now_idle - pcpu->time_in_idle);
-	delta_time = (unsigned int)(now - pcpu->time_in_idle_timestamp);
+	spin_lock(&pcpu->load_lock);
+	now = update_load(data);
+	delta_time = (unsigned int)(now - pcpu->cputime_speedadj_timestamp);
+	cputime_speedadj = pcpu->cputime_speedadj;
+	spin_unlock(&pcpu->load_lock);
 
-	/*
-	 * If timer ran less than 1ms after short-term sample started, retry.
-	 */
-	if (delta_time < 1000)
+	if (WARN_ON_ONCE(!delta_time))
 		goto rearm;
 
-	if (delta_idle > delta_time)
-		cpu_load = 0;
-	else
-		cpu_load = 100 * (delta_time - delta_idle) / delta_time;
+	do_div(cputime_speedadj, delta_time);
+	loadadjfreq = (unsigned int)cputime_speedadj * 100;
+	cpu_load = loadadjfreq / pcpu->target_freq;
 
 	if ((cpu_load >= go_hispeed_load || boost_val) &&
 	    pcpu->target_freq < hispeed_freq)
 		new_freq = hispeed_freq;
 	else
-		new_freq = choose_freq(pcpu, cpu_load);
+		new_freq = choose_freq(pcpu, loadadjfreq);
 
 	if (pcpu->target_freq >= hispeed_freq &&
 	    new_freq > pcpu->target_freq &&
@@ -498,6 +522,32 @@ static void cpufreq_interactive_boost(void)
 		wake_up_process(speedchange_task);
 }
 
+static int cpufreq_interactive_notifier(
+	struct notifier_block *nb, unsigned long val, void *data)
+{
+	struct cpufreq_freqs *freq = data;
+	struct cpufreq_interactive_cpuinfo *pcpu;
+	int cpu;
+
+	if (val == CPUFREQ_POSTCHANGE) {
+		pcpu = &per_cpu(cpuinfo, freq->cpu);
+
+		for_each_cpu(cpu, pcpu->policy->cpus) {
+			struct cpufreq_interactive_cpuinfo *pjcpu =
+				&per_cpu(cpuinfo, cpu);
+			spin_lock(&pjcpu->load_lock);
+			update_load(cpu);
+			spin_unlock(&pjcpu->load_lock);
+		}
+	}
+
+	return 0;
+}
+
+static struct notifier_block cpufreq_notifier_block = {
+	.notifier_call = cpufreq_interactive_notifier,
+};
+
 static ssize_t show_target_loads(
 	struct kobject *kobj, struct attribute *attr, char *buf)
 {
@@ -817,6 +867,8 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			return rc;
 
 		idle_notifier_register(&cpufreq_interactive_idle_nb);
+		cpufreq_register_notifier(
+			&cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER);
 		break;
 
 	case CPUFREQ_GOV_STOP:
@@ -830,6 +882,8 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		if (atomic_dec_return(&active_count) > 0)
 			return 0;
 
+		cpufreq_unregister_notifier(
+			&cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER);
 		idle_notifier_unregister(&cpufreq_interactive_idle_nb);
 		sysfs_remove_group(cpufreq_global_kobject,
 				&interactive_attr_group);
@@ -868,6 +922,7 @@ static int __init cpufreq_interactive_init(void)
 			init_timer_deferrable(&pcpu->cpu_timer);
 		pcpu->cpu_timer.function = cpufreq_interactive_timer;
 		pcpu->cpu_timer.data = i;
+		spin_lock_init(&pcpu->load_lock);
 	}
 
 	spin_lock_init(&target_loads_lock);
-- 
1.7.9.5

[PATCH 32/70] cpufreq: interactive: specify duration of CPU speed boost pulse

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Sysfs attribute boostpulse_duration specifies the duration of boosting CPU
speed in response to bootpulse events.  Duration is specified in usecs,
default 80ms.

Change-Id: Ifd41625574891a44f1787a4e85d1e7b4f2afb52b
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   52 ++++++++++++++++++++++++++++-----
 1 file changed, 45 insertions(+), 7 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index d0d51ee..8b44a82 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -95,11 +95,12 @@ static unsigned long timer_rate;
 #define DEFAULT_ABOVE_HISPEED_DELAY DEFAULT_TIMER_RATE
 static unsigned long above_hispeed_delay_val;
 
-/*
- * Non-zero means longer-term speed boost active.
- */
-
+/* Non-zero means indefinite speed boost active */
 static int boost_val;
+/* Duration of a boot pulse in usecs */
+static int boostpulse_duration_val = DEFAULT_MIN_SAMPLE_TIME;
+/* End time of boost pulse in ktime converted to usecs */
+static u64 boostpulse_endtime;
 
 static bool governidle;
 module_param(governidle, bool, S_IWUSR | S_IRUGO);
@@ -268,6 +269,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	unsigned int loadadjfreq;
 	unsigned int index;
 	unsigned long flags;
+	bool boosted;
 
 	smp_rmb();
 
@@ -286,8 +288,9 @@ static void cpufreq_interactive_timer(unsigned long data)
 	do_div(cputime_speedadj, delta_time);
 	loadadjfreq = (unsigned int)cputime_speedadj * 100;
 	cpu_load = loadadjfreq / pcpu->target_freq;
+	boosted = boost_val || now < boostpulse_endtime;
 
-	if ((cpu_load >= go_hispeed_load || boost_val) &&
+	if ((cpu_load >= go_hispeed_load || boosted) &&
 	    pcpu->target_freq < hispeed_freq)
 		new_freq = hispeed_freq;
 	else
@@ -327,8 +330,18 @@ static void cpufreq_interactive_timer(unsigned long data)
 		}
 	}
 
-	pcpu->floor_freq = new_freq;
-	pcpu->floor_validate_time = now;
+	/*
+	 * Update the timestamp for checking whether speed has been held at
+	 * or above the selected frequency for a minimum of min_sample_time,
+	 * if not boosted to hispeed_freq.  If boosted to hispeed_freq then we
+	 * allow the speed to drop as soon as the boostpulse duration expires
+	 * (or the indefinite boost is turned off).
+	 */
+
+	if (!boosted || new_freq > hispeed_freq) {
+		pcpu->floor_freq = new_freq;
+		pcpu->floor_validate_time = now;
+	}
 
 	if (pcpu->target_freq == new_freq) {
 		trace_cpufreq_interactive_already(
@@ -774,6 +787,7 @@ static ssize_t store_boostpulse(struct kobject *kobj, struct attribute *attr,
 	if (ret < 0)
 		return ret;
 
+	boostpulse_endtime = ktime_to_us(ktime_get()) + boostpulse_duration_val;
 	trace_cpufreq_interactive_boost("pulse");
 	cpufreq_interactive_boost();
 	return count;
@@ -782,6 +796,29 @@ static ssize_t store_boostpulse(struct kobject *kobj, struct attribute *attr,
 static struct global_attr boostpulse =
 	__ATTR(boostpulse, 0200, NULL, store_boostpulse);
 
+static ssize_t show_boostpulse_duration(
+	struct kobject *kobj, struct attribute *attr, char *buf)
+{
+	return sprintf(buf, "%d\n", boostpulse_duration_val);
+}
+
+static ssize_t store_boostpulse_duration(
+	struct kobject *kobj, struct attribute *attr, const char *buf,
+	size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = kstrtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+
+	boostpulse_duration_val = val;
+	return count;
+}
+
+define_one_global_rw(boostpulse_duration);
+
 static struct attribute *interactive_attributes[] = {
 	&target_loads_attr.attr,
 	&hispeed_freq_attr.attr,
@@ -791,6 +828,7 @@ static struct attribute *interactive_attributes[] = {
 	&timer_rate_attr.attr,
 	&boost.attr,
 	&boostpulse.attr,
+	&boostpulse_duration.attr,
 	NULL,
 };
 
-- 
1.7.9.5

[PATCH 33/70] cpufreq: interactive: add timer slack to limit idle at speed > min

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Always use deferrable timer for load sampling.

Set a non-deferrable timer to an additional slack time to allow prior to
waking up from idle to drop speed when not at minimum speed.  Slack value
-1 avoids wakeups to drop speed.  Default is 80ms.

Remove the governidle module param and its timer management in idle.  For
platforms on which holding speed above mimum in idle costs power, use the
new timer slack to select how long to wait before waking up to drop speed.

Change-Id: I270b3980667e2c70a68e5bff534124b4411dbad5
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |  101 ++++++++++++++++++++-------------
 1 file changed, 61 insertions(+), 40 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 8b44a82..690be16 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -40,7 +40,7 @@ static atomic_t active_count = ATOMIC_INIT(0);
 
 struct cpufreq_interactive_cpuinfo {
 	struct timer_list cpu_timer;
-	int timer_idlecancel;
+	struct timer_list cpu_slack_timer;
 	spinlock_t load_lock; /* protects the next 4 fields */
 	u64 time_in_idle;
 	u64 time_in_idle_timestamp;
@@ -102,10 +102,12 @@ static int boostpulse_duration_val = DEFAULT_MIN_SAMPLE_TIME;
 /* End time of boost pulse in ktime converted to usecs */
 static u64 boostpulse_endtime;
 
-static bool governidle;
-module_param(governidle, bool, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(governidle,
-	"Set to 1 to wake up CPUs from idle to reduce speed (default 0)");
+/*
+ * Max additional time to wait in idle, beyond timer_rate, at speeds above
+ * minimum before wakeup to reduce speed, or -1 if unnecessary.
+ */
+#define DEFAULT_TIMER_SLACK (4 * DEFAULT_TIMER_RATE)
+static int timer_slack_val = DEFAULT_TIMER_SLACK;
 
 static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		unsigned int event);
@@ -123,8 +125,14 @@ struct cpufreq_governor cpufreq_gov_interactive = {
 static void cpufreq_interactive_timer_resched(
 	struct cpufreq_interactive_cpuinfo *pcpu)
 {
-	mod_timer_pinned(&pcpu->cpu_timer,
-			 jiffies + usecs_to_jiffies(timer_rate));
+	unsigned long expires = jiffies + usecs_to_jiffies(timer_rate);
+
+	mod_timer_pinned(&pcpu->cpu_timer, expires);
+	if (timer_slack_val >= 0 && pcpu->target_freq > pcpu->policy->min) {
+		expires += usecs_to_jiffies(timer_slack_val);
+		mod_timer_pinned(&pcpu->cpu_slack_timer, expires);
+	}
+
 	spin_lock(&pcpu->load_lock);
 	pcpu->time_in_idle =
 		get_cpu_idle_time_us(smp_processor_id(),
@@ -368,17 +376,8 @@ rearm_if_notmax:
 		goto exit;
 
 rearm:
-	if (!timer_pending(&pcpu->cpu_timer)) {
-		/*
-		 * If governing speed in idle and already at min, cancel the
-		 * timer if that CPU goes idle.  We don't need to re-evaluate
-		 * speed until the next idle exit.
-		 */
-		if (governidle && pcpu->target_freq == pcpu->policy->min)
-			pcpu->timer_idlecancel = 1;
-
+	if (!timer_pending(&pcpu->cpu_timer))
 		cpufreq_interactive_timer_resched(pcpu);
-	}
 
 exit:
 	return;
@@ -404,21 +403,8 @@ static void cpufreq_interactive_idle_start(void)
 		 * min indefinitely.  This should probably be a quirk of
 		 * the CPUFreq driver.
 		 */
-		if (!pending) {
-			pcpu->timer_idlecancel = 0;
+		if (!pending)
 			cpufreq_interactive_timer_resched(pcpu);
-		}
-	} else if (governidle) {
-		/*
-		 * If at min speed and entering idle after load has
-		 * already been evaluated, and a timer has been set just in
-		 * case the CPU suddenly goes busy, cancel that timer.  The
-		 * CPU didn't go busy; we'll recheck things upon idle exit.
-		 */
-		if (pending && pcpu->timer_idlecancel) {
-			del_timer(&pcpu->cpu_timer);
-			pcpu->timer_idlecancel = 0;
-		}
 	}
 
 }
@@ -433,11 +419,10 @@ static void cpufreq_interactive_idle_end(void)
 
 	/* Arm the timer for 1-2 ticks later if not already. */
 	if (!timer_pending(&pcpu->cpu_timer)) {
-		pcpu->timer_idlecancel = 0;
 		cpufreq_interactive_timer_resched(pcpu);
-	} else if (!governidle &&
-		   time_after_eq(jiffies, pcpu->cpu_timer.expires)) {
+	} else if (time_after_eq(jiffies, pcpu->cpu_timer.expires)) {
 		del_timer(&pcpu->cpu_timer);
+		del_timer(&pcpu->cpu_slack_timer);
 		cpufreq_interactive_timer(smp_processor_id());
 	}
 }
@@ -747,6 +732,29 @@ static ssize_t store_timer_rate(struct kobject *kobj,
 static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644,
 		show_timer_rate, store_timer_rate);
 
+static ssize_t show_timer_slack(
+	struct kobject *kobj, struct attribute *attr, char *buf)
+{
+	return sprintf(buf, "%d\n", timer_slack_val);
+}
+
+static ssize_t store_timer_slack(
+	struct kobject *kobj, struct attribute *attr, const char *buf,
+	size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = kstrtol(buf, 10, &val);
+	if (ret < 0)
+		return ret;
+
+	timer_slack_val = val;
+	return count;
+}
+
+define_one_global_rw(timer_slack);
+
 static ssize_t show_boost(struct kobject *kobj, struct attribute *attr,
 			  char *buf)
 {
@@ -826,6 +834,7 @@ static struct attribute *interactive_attributes[] = {
 	&above_hispeed_delay.attr,
 	&min_sample_time_attr.attr,
 	&timer_rate_attr.attr,
+	&timer_slack.attr,
 	&boost.attr,
 	&boostpulse.attr,
 	&boostpulse_duration.attr,
@@ -876,6 +885,8 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			hispeed_freq = policy->max;
 
 		for_each_cpu(j, policy->cpus) {
+			unsigned long expires;
+
 			pcpu = &per_cpu(cpuinfo, j);
 			pcpu->policy = policy;
 			pcpu->target_freq = policy->cur;
@@ -887,9 +898,15 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 				pcpu->floor_validate_time;
 			pcpu->governor_enabled = 1;
 			smp_wmb();
-			pcpu->cpu_timer.expires =
-				jiffies + usecs_to_jiffies(timer_rate);
+			expires = jiffies + usecs_to_jiffies(timer_rate);
+			pcpu->cpu_timer.expires = expires;
 			add_timer_on(&pcpu->cpu_timer, j);
+
+			if (timer_slack_val >= 0) {
+				expires += usecs_to_jiffies(timer_slack_val);
+				pcpu->cpu_slack_timer.expires = expires;
+				add_timer_on(&pcpu->cpu_slack_timer, j);
+			}
 		}
 
 		/*
@@ -915,6 +932,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->governor_enabled = 0;
 			smp_wmb();
 			del_timer_sync(&pcpu->cpu_timer);
+			del_timer_sync(&pcpu->cpu_slack_timer);
 		}
 
 		if (atomic_dec_return(&active_count) > 0)
@@ -940,6 +958,10 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 	return 0;
 }
 
+static void cpufreq_interactive_nop_timer(unsigned long data)
+{
+}
+
 static int __init cpufreq_interactive_init(void)
 {
 	unsigned int i;
@@ -954,12 +976,11 @@ static int __init cpufreq_interactive_init(void)
 	/* Initalize per-cpu timers */
 	for_each_possible_cpu(i) {
 		pcpu = &per_cpu(cpuinfo, i);
-		if (governidle)
-			init_timer(&pcpu->cpu_timer);
-		else
-			init_timer_deferrable(&pcpu->cpu_timer);
+		init_timer_deferrable(&pcpu->cpu_timer);
 		pcpu->cpu_timer.function = cpufreq_interactive_timer;
 		pcpu->cpu_timer.data = i;
+		init_timer(&pcpu->cpu_slack_timer);
+		pcpu->cpu_slack_timer.function = cpufreq_interactive_nop_timer;
 		spin_lock_init(&pcpu->load_lock);
 	}
 
-- 
1.7.9.5

[PATCH 34/70] cpufreq: interactive: fix boosting logic

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

35a84de cpufreq: interactive: apply above_hispeed_delay to each step above hispeed

caused the speed choice logic to osciallate between boosting and not boosting.
Add back code to ensure speed does not drop below boost frequency while
boosting.

Change-Id: Id420068480fcc7f5c4989ff523e2a8d22e2f4db2
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 690be16..f8e9ee9 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -298,11 +298,18 @@ static void cpufreq_interactive_timer(unsigned long data)
 	cpu_load = loadadjfreq / pcpu->target_freq;
 	boosted = boost_val || now < boostpulse_endtime;
 
-	if ((cpu_load >= go_hispeed_load || boosted) &&
-	    pcpu->target_freq < hispeed_freq)
-		new_freq = hispeed_freq;
-	else
+	if (cpu_load >= go_hispeed_load || boosted) {
+		if (pcpu->target_freq < hispeed_freq) {
+			new_freq = hispeed_freq;
+		} else {
+			new_freq = choose_freq(pcpu, loadadjfreq);
+
+			if (new_freq < hispeed_freq)
+				new_freq = hispeed_freq;
+		}
+	} else {
 		new_freq = choose_freq(pcpu, loadadjfreq);
+	}
 
 	if (pcpu->target_freq >= hispeed_freq &&
 	    new_freq > pcpu->target_freq &&
-- 
1.7.9.5

[PATCH 35/70] cpufreq: interactive: fix racy timer stopping

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

When stopping the governor, del_timer_sync() can race against an
invocation of the idle notifier callback, which has the potential
to reactivate the timer.

To fix this issue, a read-write semaphore is used. Multiple readers are
allowed as long as pcpu->governor_enabled is true.  However it can be
moved to false only after taking a write semaphore which would wait for
any on-going asynchronous activities to complete and prevent any more of
those activities to be initiated.

[toddpoynor@google.com: cosmetic and commit text changes]
Change-Id: Ib51165a735d73dcf964a06754c48bdc1913e13d0
Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   38 ++++++++++++++++++++++++---------
 1 file changed, 28 insertions(+), 10 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index f8e9ee9..74f5609 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -21,7 +21,7 @@
 #include <linux/cpufreq.h>
 #include <linux/module.h>
 #include <linux/moduleparam.h>
-#include <linux/mutex.h>
+#include <linux/rwsem.h>
 #include <linux/sched.h>
 #include <linux/sched/rt.h>
 #include <linux/tick.h>
@@ -29,7 +29,6 @@
 #include <linux/timer.h>
 #include <linux/workqueue.h>
 #include <linux/kthread.h>
-#include <linux/mutex.h>
 #include <linux/slab.h>
 #include <asm/cputime.h>
 
@@ -52,6 +51,7 @@ struct cpufreq_interactive_cpuinfo {
 	unsigned int floor_freq;
 	u64 floor_validate_time;
 	u64 hispeed_validate_time;
+	struct rw_semaphore enable_sem;
 	int governor_enabled;
 };
 
@@ -279,8 +279,8 @@ static void cpufreq_interactive_timer(unsigned long data)
 	unsigned long flags;
 	bool boosted;
 
-	smp_rmb();
-
+	if (!down_read_trylock(&pcpu->enable_sem))
+		return;
 	if (!pcpu->governor_enabled)
 		goto exit;
 
@@ -387,6 +387,7 @@ rearm:
 		cpufreq_interactive_timer_resched(pcpu);
 
 exit:
+	up_read(&pcpu->enable_sem);
 	return;
 }
 
@@ -396,8 +397,12 @@ static void cpufreq_interactive_idle_start(void)
 		&per_cpu(cpuinfo, smp_processor_id());
 	int pending;
 
-	if (!pcpu->governor_enabled)
+	if (!down_read_trylock(&pcpu->enable_sem))
+		return;
+	if (!pcpu->governor_enabled) {
+		up_read(&pcpu->enable_sem);
 		return;
+	}
 
 	pending = timer_pending(&pcpu->cpu_timer);
 
@@ -414,6 +419,7 @@ static void cpufreq_interactive_idle_start(void)
 			cpufreq_interactive_timer_resched(pcpu);
 	}
 
+	up_read(&pcpu->enable_sem);
 }
 
 static void cpufreq_interactive_idle_end(void)
@@ -421,8 +427,12 @@ static void cpufreq_interactive_idle_end(void)
 	struct cpufreq_interactive_cpuinfo *pcpu =
 		&per_cpu(cpuinfo, smp_processor_id());
 
-	if (!pcpu->governor_enabled)
+	if (!down_read_trylock(&pcpu->enable_sem))
+		return;
+	if (!pcpu->governor_enabled) {
+		up_read(&pcpu->enable_sem);
 		return;
+	}
 
 	/* Arm the timer for 1-2 ticks later if not already. */
 	if (!timer_pending(&pcpu->cpu_timer)) {
@@ -432,6 +442,8 @@ static void cpufreq_interactive_idle_end(void)
 		del_timer(&pcpu->cpu_slack_timer);
 		cpufreq_interactive_timer(smp_processor_id());
 	}
+
+	up_read(&pcpu->enable_sem);
 }
 
 static int cpufreq_interactive_speedchange_task(void *data)
@@ -466,10 +478,12 @@ static int cpufreq_interactive_speedchange_task(void *data)
 			unsigned int max_freq = 0;
 
 			pcpu = &per_cpu(cpuinfo, cpu);
-			smp_rmb();
-
-			if (!pcpu->governor_enabled)
+			if (!down_read_trylock(&pcpu->enable_sem))
+				continue;
+			if (!pcpu->governor_enabled) {
+				up_read(&pcpu->enable_sem);
 				continue;
+			}
 
 			for_each_cpu(j, pcpu->policy->cpus) {
 				struct cpufreq_interactive_cpuinfo *pjcpu =
@@ -486,6 +500,8 @@ static int cpufreq_interactive_speedchange_task(void *data)
 			trace_cpufreq_interactive_setspeed(cpu,
 						     pcpu->target_freq,
 						     pcpu->policy->cur);
+
+			up_read(&pcpu->enable_sem);
 		}
 	}
 
@@ -936,10 +952,11 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 	case CPUFREQ_GOV_STOP:
 		for_each_cpu(j, policy->cpus) {
 			pcpu = &per_cpu(cpuinfo, j);
+			down_write(&pcpu->enable_sem);
 			pcpu->governor_enabled = 0;
-			smp_wmb();
 			del_timer_sync(&pcpu->cpu_timer);
 			del_timer_sync(&pcpu->cpu_slack_timer);
+			up_write(&pcpu->enable_sem);
 		}
 
 		if (atomic_dec_return(&active_count) > 0)
@@ -989,6 +1006,7 @@ static int __init cpufreq_interactive_init(void)
 		init_timer(&pcpu->cpu_slack_timer);
 		pcpu->cpu_slack_timer.function = cpufreq_interactive_nop_timer;
 		spin_lock_init(&pcpu->load_lock);
+		init_rwsem(&pcpu->enable_sem);
 	}
 
 	spin_lock_init(&target_loads_lock);
-- 
1.7.9.5

[PATCH 36/70] cpufreq: interactive: fix race on timer restart on governor start

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Starting the governor, or restarting on a hotplugged-in CPU, can race
with the timer start in idle, triggering a BUG on timer already pending.
Start the timer before setting the enable flag, and use enable_sem to
protect the sequence (and ensure correct order of the update to the
enable flag).  Delete any existing timer for safety.

Change-Id: Ife77cf9fe099e8fd8543224cbf148c6722c2ffb0
Reported-by: Francisco Franco <francisco.franco@cloudcar.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 74f5609..d22d162 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -919,17 +919,17 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 				ktime_to_us(ktime_get());
 			pcpu->hispeed_validate_time =
 				pcpu->floor_validate_time;
-			pcpu->governor_enabled = 1;
-			smp_wmb();
+			down_write(&pcpu->enable_sem);
 			expires = jiffies + usecs_to_jiffies(timer_rate);
 			pcpu->cpu_timer.expires = expires;
 			add_timer_on(&pcpu->cpu_timer, j);
-
 			if (timer_slack_val >= 0) {
 				expires += usecs_to_jiffies(timer_slack_val);
 				pcpu->cpu_slack_timer.expires = expires;
 				add_timer_on(&pcpu->cpu_slack_timer, j);
 			}
+			pcpu->governor_enabled = 1;
+			up_write(&pcpu->enable_sem);
 		}
 
 		/*
-- 
1.7.9.5

[PATCH 37/70] cpufreq: interactive: default go_hispeed_load 99%, doc updates

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Update default go_hispeed_load from 85% to 99%.  Recent changes to the
governor now use a default target_load of 90%.  go_hispeed_load should
not be lower than the target load for hispeed_freq, which could lead
to oscillating speed decisions.  Other recent changes reduce the need
to dampen speed jumps on load spikes, while input event boosts from
userspace are the preferred method for anticipating load spikes with
UI impacts.

General update to the documentation to reflect recent changes.

Change-Id: I1b92f3091f42c04b10503cd1169a943b5dfd6faf
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 Documentation/cpu-freq/governors.txt  |   64 ++++++++++++++++-----------------
 drivers/cpufreq/cpufreq_interactive.c |    2 +-
 2 files changed, 33 insertions(+), 33 deletions(-)

diff --git a/Documentation/cpu-freq/governors.txt b/Documentation/cpu-freq/governors.txt
index b15f6d2..4d8ae34 100644
--- a/Documentation/cpu-freq/governors.txt
+++ b/Documentation/cpu-freq/governors.txt
@@ -224,23 +224,8 @@ sampling rate.
 
 The CPUfreq governor "interactive" is designed for latency-sensitive,
 interactive workloads. This governor sets the CPU speed depending on
-usage, similar to "ondemand" and "conservative" governors.  However,
-the governor is more aggressive about scaling the CPU speed up in
-response to CPU-intensive activity.
-
-Sampling the CPU load every X ms can lead to under-powering the CPU
-for X ms, leading to dropped frames, stuttering UI, etc.  Instead of
-sampling the cpu at a specified rate, the interactive governor will
-check whether to scale the cpu frequency up soon after coming out of
-idle.  When the cpu comes out of idle, a timer is configured to fire
-within 1-2 ticks.  If the cpu is very busy between exiting idle and
-when the timer fires then we assume the cpu is underpowered and ramp
-to MAX speed.
-
-If the cpu was not sufficiently busy to immediately ramp to MAX speed,
-then governor evaluates the cpu load since the last speed adjustment,
-choosing the highest value between that longer-term load or the
-short-term load since idle exit to determine the cpu speed to ramp to.
+usage, similar to "ondemand" and "conservative" governors, but with a
+different set of configurable behaviors.
 
 The tuneable values for this governor are:
 
@@ -262,36 +247,51 @@ values also usually appear in an ascending order. The default is
 target load 90% for all speeds.
 
 min_sample_time: The minimum amount of time to spend at the current
-frequency before ramping down. This is to ensure that the governor has
-seen enough historic cpu load data to determine the appropriate
-workload.  Default is 80000 uS.
+frequency before ramping down. Default is 80000 uS.
 
 hispeed_freq: An intermediate "hi speed" at which to initially ramp
 when CPU load hits the value specified in go_hispeed_load.  If load
 stays high for the amount of time specified in above_hispeed_delay,
-then speed may be bumped higher.  Default is maximum speed.
+then speed may be bumped higher.  Default is the maximum speed
+allowed by the policy at governor initialization time.
 
-go_hispeed_load: The CPU load at which to ramp to the intermediate "hi
-speed".  Default is 85%.
+go_hispeed_load: The CPU load at which to ramp to hispeed_freq.
+Default is 99%.
 
-above_hispeed_delay: Once speed is set to hispeed_freq, wait for this
-long before bumping speed higher in response to continued high load.
+above_hispeed_delay: When speed is at or above hispeed_freq, wait for
+this long before raising speed in response to continued high load.
 Default is 20000 uS.
 
-timer_rate: Sample rate for reevaluating cpu load when the system is
-not idle.  Default is 20000 uS.
-
-input_boost: If non-zero, boost speed of all CPUs to hispeed_freq on
-touchscreen activity.  Default is 0.
+timer_rate: Sample rate for reevaluating CPU load when the CPU is not
+idle.  A deferrable timer is used, such that the CPU will not be woken
+from idle to service this timer until something else needs to run.
+(The maximum time to allow deferring this timer when not running at
+minimum speed is configurable via timer_slack.)  Default is 20000 uS.
+
+timer_slack: Maximum additional time to defer handling the governor
+sampling timer beyond timer_rate when running at speeds above the
+minimum.  For platforms that consume additional power at idle when
+CPUs are running at speeds greater than minimum, this places an upper
+bound on how long the timer will be deferred prior to re-evaluating
+load and dropping speed.  For example, if timer_rate is 20000uS and
+timer_slack is 10000uS then timers will be deferred for up to 30msec
+when not at lowest speed.  A value of -1 means defer timers
+indefinitely at all speeds.  Default is 80000 uS.
 
 boost: If non-zero, immediately boost speed of all CPUs to at least
 hispeed_freq until zero is written to this attribute.  If zero, allow
 CPU speeds to drop below hispeed_freq according to load as usual.
+Default is zero.
 
-boostpulse: Immediately boost speed of all CPUs to hispeed_freq for
-min_sample_time, after which speeds are allowed to drop below
+boostpulse: On each write, immediately boost speed of all CPUs to
+hispeed_freq for at least the period of time specified by
+boostpulse_duration, after which speeds are allowed to drop below
 hispeed_freq according to load as usual.
 
+boostpulse_duration: Length of time to hold CPU speed at hispeed_freq
+on a write to boostpulse, before allowing speed to drop according to
+load as usual.  Default is 80000 uS.
+
 
 3. The Governor Interface in the CPUfreq Core
 =============================================
diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index d22d162..fdc2a47 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -66,7 +66,7 @@ static spinlock_t speedchange_cpumask_lock;
 static unsigned int hispeed_freq;
 
 /* Go to hi speed when CPU load at or above this value. */
-#define DEFAULT_GO_HISPEED_LOAD 85
+#define DEFAULT_GO_HISPEED_LOAD 99
 static unsigned long go_hispeed_load;
 
 /* Target load.  Lower values result in higher CPU speeds. */
-- 
1.7.9.5

[PATCH 38/70] cpufreq: interactive: init default values at compile time

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Change-Id: Ia4966e949a6c24c34fdbd4a6e522cd7c37e4108e
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index fdc2a47..c8358a3 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -67,7 +67,7 @@ static unsigned int hispeed_freq;
 
 /* Go to hi speed when CPU load at or above this value. */
 #define DEFAULT_GO_HISPEED_LOAD 99
-static unsigned long go_hispeed_load;
+static unsigned long go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;
 
 /* Target load.  Lower values result in higher CPU speeds. */
 #define DEFAULT_TARGET_LOAD 90
@@ -80,20 +80,20 @@ static int ntarget_loads = ARRAY_SIZE(default_target_loads);
  * The minimum amount of time to spend at a frequency before we can ramp down.
  */
 #define DEFAULT_MIN_SAMPLE_TIME (80 * USEC_PER_MSEC)
-static unsigned long min_sample_time;
+static unsigned long min_sample_time = DEFAULT_MIN_SAMPLE_TIME;
 
 /*
  * The sample rate of the timer used to increase frequency
  */
 #define DEFAULT_TIMER_RATE (20 * USEC_PER_MSEC)
-static unsigned long timer_rate;
+static unsigned long timer_rate = DEFAULT_TIMER_RATE;
 
 /*
  * Wait this long before raising speed above hispeed, by default a single
  * timer interval.
  */
 #define DEFAULT_ABOVE_HISPEED_DELAY DEFAULT_TIMER_RATE
-static unsigned long above_hispeed_delay_val;
+static unsigned long above_hispeed_delay_val = DEFAULT_ABOVE_HISPEED_DELAY;
 
 /* Non-zero means indefinite speed boost active */
 static int boost_val;
@@ -992,11 +992,6 @@ static int __init cpufreq_interactive_init(void)
 	struct cpufreq_interactive_cpuinfo *pcpu;
 	struct sched_param param = { .sched_priority = MAX_RT_PRIO-1 };
 
-	go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;
-	min_sample_time = DEFAULT_MIN_SAMPLE_TIME;
-	above_hispeed_delay_val = DEFAULT_ABOVE_HISPEED_DELAY;
-	timer_rate = DEFAULT_TIMER_RATE;
-
 	/* Initalize per-cpu timers */
 	for_each_possible_cpu(i) {
 		pcpu = &per_cpu(cpuinfo, i);
-- 
1.7.9.5

[PATCH 39/70] cpufreq: interactive: don't handle transition notification if not enabled

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

If multiple governors are in use then avoid processing frequency transition
notifications for CPUs on which the interactive governor is not enabled.

Change-Id: Ibd75255b921d887501a64774a8c4f62302f2d4e4
Reported-by: Francisco Franco <francisco.franco@cloudcar.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index c8358a3..51c34bf 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -552,6 +552,12 @@ static int cpufreq_interactive_notifier(
 
 	if (val == CPUFREQ_POSTCHANGE) {
 		pcpu = &per_cpu(cpuinfo, freq->cpu);
+		if (!down_read_trylock(&pcpu->enable_sem))
+			return 0;
+		if (!pcpu->governor_enabled) {
+			up_read(&pcpu->enable_sem);
+			return 0;
+		}
 
 		for_each_cpu(cpu, pcpu->policy->cpus) {
 			struct cpufreq_interactive_cpuinfo *pjcpu =
@@ -560,8 +566,9 @@ static int cpufreq_interactive_notifier(
 			update_load(cpu);
 			spin_unlock(&pjcpu->load_lock);
 		}
-	}
 
+		up_read(&pcpu->enable_sem);
+	}
 	return 0;
 }
 
-- 
1.7.9.5

[PATCH 40/70] cpufreq: interactive: fix deadlock on spinlock in timer

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Need to use irqsave/restore spinlock calls to avoid a deadlock in calls
from the timer.

Change-Id: I15b6b590045ba1447e34ca7b5ff342723e53a605
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 51c34bf..e7f26aa 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -126,6 +126,7 @@ static void cpufreq_interactive_timer_resched(
 	struct cpufreq_interactive_cpuinfo *pcpu)
 {
 	unsigned long expires = jiffies + usecs_to_jiffies(timer_rate);
+	unsigned long flags;
 
 	mod_timer_pinned(&pcpu->cpu_timer, expires);
 	if (timer_slack_val >= 0 && pcpu->target_freq > pcpu->policy->min) {
@@ -133,27 +134,28 @@ static void cpufreq_interactive_timer_resched(
 		mod_timer_pinned(&pcpu->cpu_slack_timer, expires);
 	}
 
-	spin_lock(&pcpu->load_lock);
+	spin_lock_irqsave(&pcpu->load_lock, flags);
 	pcpu->time_in_idle =
 		get_cpu_idle_time_us(smp_processor_id(),
 				     &pcpu->time_in_idle_timestamp);
 	pcpu->cputime_speedadj = 0;
 	pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp;
-	spin_unlock(&pcpu->load_lock);
+	spin_unlock_irqrestore(&pcpu->load_lock, flags);
 }
 
 static unsigned int freq_to_targetload(unsigned int freq)
 {
 	int i;
 	unsigned int ret;
+	unsigned long flags;
 
-	spin_lock(&target_loads_lock);
+	spin_lock_irqsave(&target_loads_lock, flags);
 
 	for (i = 0; i < ntarget_loads - 1 && freq >= target_loads[i+1]; i += 2)
 		;
 
 	ret = target_loads[i];
-	spin_unlock(&target_loads_lock);
+	spin_unlock_irqrestore(&target_loads_lock, flags);
 	return ret;
 }
 
@@ -284,11 +286,11 @@ static void cpufreq_interactive_timer(unsigned long data)
 	if (!pcpu->governor_enabled)
 		goto exit;
 
-	spin_lock(&pcpu->load_lock);
+	spin_lock_irqsave(&pcpu->load_lock, flags);
 	now = update_load(data);
 	delta_time = (unsigned int)(now - pcpu->cputime_speedadj_timestamp);
 	cputime_speedadj = pcpu->cputime_speedadj;
-	spin_unlock(&pcpu->load_lock);
+	spin_unlock_irqrestore(&pcpu->load_lock, flags);
 
 	if (WARN_ON_ONCE(!delta_time))
 		goto rearm;
@@ -549,6 +551,7 @@ static int cpufreq_interactive_notifier(
 	struct cpufreq_freqs *freq = data;
 	struct cpufreq_interactive_cpuinfo *pcpu;
 	int cpu;
+	unsigned long flags;
 
 	if (val == CPUFREQ_POSTCHANGE) {
 		pcpu = &per_cpu(cpuinfo, freq->cpu);
@@ -562,9 +565,9 @@ static int cpufreq_interactive_notifier(
 		for_each_cpu(cpu, pcpu->policy->cpus) {
 			struct cpufreq_interactive_cpuinfo *pjcpu =
 				&per_cpu(cpuinfo, cpu);
-			spin_lock(&pjcpu->load_lock);
+			spin_lock_irqsave(&pjcpu->load_lock, flags);
 			update_load(cpu);
-			spin_unlock(&pjcpu->load_lock);
+			spin_unlock_irqrestore(&pjcpu->load_lock, flags);
 		}
 
 		up_read(&pcpu->enable_sem);
@@ -581,15 +584,16 @@ static ssize_t show_target_loads(
 {
 	int i;
 	ssize_t ret = 0;
+	unsigned long flags;
 
-	spin_lock(&target_loads_lock);
+	spin_lock_irqsave(&target_loads_lock, flags);
 
 	for (i = 0; i < ntarget_loads; i++)
 		ret += sprintf(buf + ret, "%u%s", target_loads[i],
 			       i & 0x1 ? ":" : " ");
 
 	ret += sprintf(buf + ret, "\n");
-	spin_unlock(&target_loads_lock);
+	spin_unlock_irqrestore(&target_loads_lock, flags);
 	return ret;
 }
 
@@ -602,6 +606,7 @@ static ssize_t store_target_loads(
 	unsigned int *new_target_loads = NULL;
 	int ntokens = 1;
 	int i;
+	unsigned long flags;
 
 	cp = buf;
 	while ((cp = strpbrk(cp + 1, " :")))
@@ -631,12 +636,12 @@ static ssize_t store_target_loads(
 	if (i != ntokens)
 		goto err_inval;
 
-	spin_lock(&target_loads_lock);
+	spin_lock_irqsave(&target_loads_lock, flags);
 	if (target_loads != default_target_loads)
 		kfree(target_loads);
 	target_loads = new_target_loads;
 	ntarget_loads = ntokens;
-	spin_unlock(&target_loads_lock);
+	spin_unlock_irqrestore(&target_loads_lock, flags);
 	return count;
 
 err_inval:
-- 
1.7.9.5

[PATCH 41/70] cpufreq: interactive: fix race on governor start/stop

[Bálint Czobor]

From: Lianwei Wang <a22439@motorola.com>

There is race condition when both two cpu do CPUFREQ_GOV_STOP and one cpu
do CPUFREQ_GOV_START soon. The sysfs_remove_group is not done yet on one
cpu, but sysfs_create_group is called on another cpu, which cause governor
start failed and then kernel panic in timer callback because the policy and
cpu mask are all kfree in cpufreq driver.

Replace atomic with mutex to lock the whole START/STOP sequence.

Change-Id: I3762b3d44315ae021b8275aca84f5ea9147cc540
Signed-off-by: Lianwei Wang <a22439@motorola.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index e7f26aa..3447e58 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -35,7 +35,7 @@
 #define CREATE_TRACE_POINTS
 #include <trace/events/cpufreq_interactive.h>
 
-static atomic_t active_count = ATOMIC_INIT(0);
+static int active_count;
 
 struct cpufreq_interactive_cpuinfo {
 	struct timer_list cpu_timer;
@@ -61,6 +61,7 @@ static DEFINE_PER_CPU(struct cpufreq_interactive_cpuinfo, cpuinfo);
 static struct task_struct *speedchange_task;
 static cpumask_t speedchange_cpumask;
 static spinlock_t speedchange_cpumask_lock;
+static struct mutex gov_lock;
 
 /* Hi speed to bump to from lo speed when load burst (default max) */
 static unsigned int hispeed_freq;
@@ -914,6 +915,8 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		if (!cpu_online(policy->cpu))
 			return -EINVAL;
 
+		mutex_lock(&gov_lock);
+
 		freq_table =
 			cpufreq_frequency_get_table(policy->cpu);
 		if (!hispeed_freq)
@@ -948,20 +951,26 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		 * Do not register the idle hook and create sysfs
 		 * entries if we have already done so.
 		 */
-		if (atomic_inc_return(&active_count) > 1)
+		if (++active_count > 1) {
+			mutex_unlock(&gov_lock);
 			return 0;
+		}
 
 		rc = sysfs_create_group(cpufreq_global_kobject,
 				&interactive_attr_group);
-		if (rc)
+		if (rc) {
+			mutex_unlock(&gov_lock);
 			return rc;
+		}
 
 		idle_notifier_register(&cpufreq_interactive_idle_nb);
 		cpufreq_register_notifier(
 			&cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER);
+		mutex_unlock(&gov_lock);
 		break;
 
 	case CPUFREQ_GOV_STOP:
+		mutex_lock(&gov_lock);
 		for_each_cpu(j, policy->cpus) {
 			pcpu = &per_cpu(cpuinfo, j);
 			down_write(&pcpu->enable_sem);
@@ -971,14 +980,17 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			up_write(&pcpu->enable_sem);
 		}
 
-		if (atomic_dec_return(&active_count) > 0)
+		if (--active_count > 0) {
+			mutex_unlock(&gov_lock);
 			return 0;
+		}
 
 		cpufreq_unregister_notifier(
 			&cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER);
 		idle_notifier_unregister(&cpufreq_interactive_idle_nb);
 		sysfs_remove_group(cpufreq_global_kobject,
 				&interactive_attr_group);
+		mutex_unlock(&gov_lock);
 
 		break;
 
@@ -1018,6 +1030,7 @@ static int __init cpufreq_interactive_init(void)
 
 	spin_lock_init(&target_loads_lock);
 	spin_lock_init(&speedchange_cpumask_lock);
+	mutex_init(&gov_lock);
 	speedchange_task =
 		kthread_create(cpufreq_interactive_speedchange_task, NULL,
 			       "cfinteractive");
-- 
1.7.9.5

[PATCH 42/70] cpufreq: interactive: allow arbitrary speed / delay mappings

[Bálint Czobor]

From: Minsung Kim <ms925.kim@samsung.com>

Accept a string of delays and speeds at which to apply the delay before
raising each step above hispeed. For example, "80000 1300000:200000
1500000:40000" means that the delay at or above 1GHz, until 1.3GHz is 80 msecs,
the delay until 1.5GHz is 200 msecs and the delay at or above 1.5GHz is 40
msecs when hispeed_freq is 1GHz.

[toddpoynor@google.com: add documentation]
Change-Id: Ifeebede8b1acbdd0a53e5c6916bccbf764dc854f
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 Documentation/cpu-freq/governors.txt  |   12 ++-
 drivers/cpufreq/cpufreq_interactive.c |  185 ++++++++++++++++++++++-----------
 2 files changed, 134 insertions(+), 63 deletions(-)

diff --git a/Documentation/cpu-freq/governors.txt b/Documentation/cpu-freq/governors.txt
index 4d8ae34..ac8a37e 100644
--- a/Documentation/cpu-freq/governors.txt
+++ b/Documentation/cpu-freq/governors.txt
@@ -260,7 +260,17 @@ Default is 99%.
 
 above_hispeed_delay: When speed is at or above hispeed_freq, wait for
 this long before raising speed in response to continued high load.
-Default is 20000 uS.
+The format is a single delay value, optionally followed by pairs of
+CPU speeds and the delay to use at or above those speeds.  Colons can
+be used between the speeds and associated delays for readability.  For
+example:
+
+   80000 1300000:200000 1500000:40000
+
+uses delay 80000 uS until CPU speed 1.3 GHz, at which speed delay
+200000 uS is used until speed 1.5 GHz, at which speed (and above)
+delay 40000 uS is used.  If speeds are specified these must appear in
+ascending order.  Default is 20000 uS.
 
 timer_rate: Sample rate for reevaluating CPU load when the CPU is not
 idle.  A deferrable timer is used, such that the CPU will not be woken
diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 3447e58..620b46c 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -94,7 +94,11 @@ static unsigned long timer_rate = DEFAULT_TIMER_RATE;
  * timer interval.
  */
 #define DEFAULT_ABOVE_HISPEED_DELAY DEFAULT_TIMER_RATE
-static unsigned long above_hispeed_delay_val = DEFAULT_ABOVE_HISPEED_DELAY;
+static unsigned int default_above_hispeed_delay[] = {
+	DEFAULT_ABOVE_HISPEED_DELAY };
+static spinlock_t above_hispeed_delay_lock;
+static unsigned int *above_hispeed_delay = default_above_hispeed_delay;
+static int nabove_hispeed_delay = ARRAY_SIZE(default_above_hispeed_delay);
 
 /* Non-zero means indefinite speed boost active */
 static int boost_val;
@@ -144,6 +148,23 @@ static void cpufreq_interactive_timer_resched(
 	spin_unlock_irqrestore(&pcpu->load_lock, flags);
 }
 
+static unsigned int freq_to_above_hispeed_delay(unsigned int freq)
+{
+	int i;
+	unsigned int ret;
+	unsigned long flags;
+
+	spin_lock_irqsave(&above_hispeed_delay_lock, flags);
+
+	for (i = 0; i < nabove_hispeed_delay - 1 &&
+			freq >= above_hispeed_delay[i+1]; i += 2)
+		;
+
+	ret = above_hispeed_delay[i];
+	spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
+	return ret;
+}
+
 static unsigned int freq_to_targetload(unsigned int freq)
 {
 	int i;
@@ -316,7 +337,8 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 	if (pcpu->target_freq >= hispeed_freq &&
 	    new_freq > pcpu->target_freq &&
-	    now - pcpu->hispeed_validate_time < above_hispeed_delay_val) {
+	    now - pcpu->hispeed_validate_time <
+	    freq_to_above_hispeed_delay(pcpu->policy->cur)) {
 		trace_cpufreq_interactive_notyet(
 			data, cpu_load, pcpu->target_freq,
 			pcpu->policy->cur, new_freq);
@@ -580,6 +602,56 @@ static struct notifier_block cpufreq_notifier_block = {
 	.notifier_call = cpufreq_interactive_notifier,
 };
 
+static unsigned int *get_tokenized_data(const char *buf, int *num_tokens)
+{
+	const char *cp;
+	int i;
+	int ntokens = 1;
+	unsigned int *tokenized_data;
+
+	cp = buf;
+	while ((cp = strpbrk(cp + 1, " :")))
+		ntokens++;
+
+	if (!(ntokens & 0x1)) {
+		tokenized_data = ERR_PTR(-EINVAL);
+		goto err;
+	}
+
+	tokenized_data = kmalloc(ntokens * sizeof(unsigned int), GFP_KERNEL);
+	if (!tokenized_data) {
+		tokenized_data = ERR_PTR(-ENOMEM);
+		goto err;
+	}
+
+	cp = buf;
+	i = 0;
+	while (i < ntokens) {
+		if (sscanf(cp, "%u", &tokenized_data[i++]) != 1) {
+			tokenized_data = ERR_PTR(-EINVAL);
+			goto err_kfree;
+		}
+
+		cp = strpbrk(cp, " :");
+		if (!cp)
+			break;
+		cp++;
+	}
+
+	if (i != ntokens) {
+		tokenized_data = ERR_PTR(-EINVAL);
+		goto err_kfree;
+	}
+
+	*num_tokens = ntokens;
+	return tokenized_data;
+
+err_kfree:
+	kfree(tokenized_data);
+err:
+	return tokenized_data;
+}
+
 static ssize_t show_target_loads(
 	struct kobject *kobj, struct attribute *attr, char *buf)
 {
@@ -602,40 +674,13 @@ static ssize_t store_target_loads(
 	struct kobject *kobj, struct attribute *attr, const char *buf,
 	size_t count)
 {
-	int ret;
-	const char *cp;
+	int ntokens;
 	unsigned int *new_target_loads = NULL;
-	int ntokens = 1;
-	int i;
 	unsigned long flags;
 
-	cp = buf;
-	while ((cp = strpbrk(cp + 1, " :")))
-		ntokens++;
-
-	if (!(ntokens & 0x1))
-		goto err_inval;
-
-	new_target_loads = kmalloc(ntokens * sizeof(unsigned int), GFP_KERNEL);
-	if (!new_target_loads) {
-		ret = -ENOMEM;
-		goto err;
-	}
-
-	cp = buf;
-	i = 0;
-	while (i < ntokens) {
-		if (sscanf(cp, "%u", &new_target_loads[i++]) != 1)
-			goto err_inval;
-
-		cp = strpbrk(cp, " :");
-		if (!cp)
-			break;
-		cp++;
-	}
-
-	if (i != ntokens)
-		goto err_inval;
+	new_target_loads = get_tokenized_data(buf, &ntokens);
+	if (IS_ERR(new_target_loads))
+		return PTR_RET(new_target_loads);
 
 	spin_lock_irqsave(&target_loads_lock, flags);
 	if (target_loads != default_target_loads)
@@ -644,18 +689,56 @@ static ssize_t store_target_loads(
 	ntarget_loads = ntokens;
 	spin_unlock_irqrestore(&target_loads_lock, flags);
 	return count;
-
-err_inval:
-	ret = -EINVAL;
-err:
-	kfree(new_target_loads);
-	return ret;
 }
 
 static struct global_attr target_loads_attr =
 	__ATTR(target_loads, S_IRUGO | S_IWUSR,
 		show_target_loads, store_target_loads);
 
+static ssize_t show_above_hispeed_delay(
+	struct kobject *kobj, struct attribute *attr, char *buf)
+{
+	int i;
+	ssize_t ret = 0;
+	unsigned long flags;
+
+	spin_lock_irqsave(&above_hispeed_delay_lock, flags);
+
+	for (i = 0; i < nabove_hispeed_delay; i++)
+		ret += sprintf(buf + ret, "%u%s", above_hispeed_delay[i],
+			       i & 0x1 ? ":" : " ");
+
+	ret += sprintf(buf + ret, "\n");
+	spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
+	return ret;
+}
+
+static ssize_t store_above_hispeed_delay(
+	struct kobject *kobj, struct attribute *attr, const char *buf,
+	size_t count)
+{
+	int ntokens;
+	unsigned int *new_above_hispeed_delay = NULL;
+	unsigned long flags;
+
+	new_above_hispeed_delay = get_tokenized_data(buf, &ntokens);
+	if (IS_ERR(new_above_hispeed_delay))
+		return PTR_RET(new_above_hispeed_delay);
+
+	spin_lock_irqsave(&above_hispeed_delay_lock, flags);
+	if (above_hispeed_delay != default_above_hispeed_delay)
+		kfree(above_hispeed_delay);
+	above_hispeed_delay = new_above_hispeed_delay;
+	nabove_hispeed_delay = ntokens;
+	spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
+	return count;
+
+}
+
+static struct global_attr above_hispeed_delay_attr =
+	__ATTR(above_hispeed_delay, S_IRUGO | S_IWUSR,
+		show_above_hispeed_delay, store_above_hispeed_delay);
+
 static ssize_t show_hispeed_freq(struct kobject *kobj,
 				 struct attribute *attr, char *buf)
 {
@@ -724,28 +807,6 @@ static ssize_t store_min_sample_time(struct kobject *kobj,
 static struct global_attr min_sample_time_attr = __ATTR(min_sample_time, 0644,
 		show_min_sample_time, store_min_sample_time);
 
-static ssize_t show_above_hispeed_delay(struct kobject *kobj,
-					struct attribute *attr, char *buf)
-{
-	return sprintf(buf, "%lu\n", above_hispeed_delay_val);
-}
-
-static ssize_t store_above_hispeed_delay(struct kobject *kobj,
-					 struct attribute *attr,
-					 const char *buf, size_t count)
-{
-	int ret;
-	unsigned long val;
-
-	ret = strict_strtoul(buf, 0, &val);
-	if (ret < 0)
-		return ret;
-	above_hispeed_delay_val = val;
-	return count;
-}
-
-define_one_global_rw(above_hispeed_delay);
-
 static ssize_t show_timer_rate(struct kobject *kobj,
 			struct attribute *attr, char *buf)
 {
@@ -865,9 +926,9 @@ define_one_global_rw(boostpulse_duration);
 
 static struct attribute *interactive_attributes[] = {
 	&target_loads_attr.attr,
+	&above_hispeed_delay_attr.attr,
 	&hispeed_freq_attr.attr,
 	&go_hispeed_load_attr.attr,
-	&above_hispeed_delay.attr,
 	&min_sample_time_attr.attr,
 	&timer_rate_attr.attr,
 	&timer_slack.attr,
-- 
1.7.9.5

[PATCH 43/70] cpufreq: interactive: add io_is_busy interface

[Bálint Czobor]

From: Lianwei Wang <lian-wei.wang@motorola.com>

Previously the idle time returned from get_cpu_idle_time_us included the
iowait time. So the iowait time was always calculated as idle time.

But now the idle time returned from get_cpu_idle_time_us does not include
the iowait time anymore because of below commit which cause the iowait time
always calculated as busy time:
    6beea0c nohz: Fix update_ts_time_stat idle accounting

Add the io_is_busy interface, as does the ondemand governor, and let the user
configure the iowait time as busy or idle through the io_is_busy sysfs
interface.

By default, io_is_busy is disabled.

[toddpoynor@google.com: minor updates]
Change-Id: If7d70ff864c43bc9c8d7fd7cfc66f930d339f9b4
Signed-off-by: Lianwei Wang <lian-wei.wang@motorola.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   66 ++++++++++++++++++++++++++++++++-
 1 file changed, 64 insertions(+), 2 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 620b46c..7c734fa 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -30,6 +30,7 @@
 #include <linux/workqueue.h>
 #include <linux/kthread.h>
 #include <linux/slab.h>
+#include <linux/kernel_stat.h>
 #include <asm/cputime.h>
 
 #define CREATE_TRACE_POINTS
@@ -114,6 +115,8 @@ static u64 boostpulse_endtime;
 #define DEFAULT_TIMER_SLACK (4 * DEFAULT_TIMER_RATE)
 static int timer_slack_val = DEFAULT_TIMER_SLACK;
 
+static bool io_is_busy;
+
 static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		unsigned int event);
 
@@ -127,6 +130,42 @@ struct cpufreq_governor cpufreq_gov_interactive = {
 	.owner = THIS_MODULE,
 };
 
+static inline cputime64_t get_cpu_idle_time_jiffy(unsigned int cpu,
+						  cputime64_t *wall)
+{
+	u64 idle_time;
+	u64 cur_wall_time;
+	u64 busy_time;
+
+	cur_wall_time = jiffies64_to_cputime64(get_jiffies_64());
+
+	busy_time  = kcpustat_cpu(cpu).cpustat[CPUTIME_USER];
+	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SYSTEM];
+	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_IRQ];
+	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SOFTIRQ];
+	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL];
+	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_NICE];
+
+	idle_time = cur_wall_time - busy_time;
+	if (wall)
+		*wall = jiffies_to_usecs(cur_wall_time);
+
+	return jiffies_to_usecs(idle_time);
+}
+
+static inline cputime64_t get_cpu_idle_time(unsigned int cpu,
+					    cputime64_t *wall)
+{
+	u64 idle_time = get_cpu_idle_time_us(cpu, wall);
+
+	if (idle_time == -1ULL)
+		idle_time = get_cpu_idle_time_jiffy(cpu, wall);
+	else if (!io_is_busy)
+		idle_time += get_cpu_iowait_time_us(cpu, wall);
+
+	return idle_time;
+}
+
 static void cpufreq_interactive_timer_resched(
 	struct cpufreq_interactive_cpuinfo *pcpu)
 {
@@ -141,7 +180,7 @@ static void cpufreq_interactive_timer_resched(
 
 	spin_lock_irqsave(&pcpu->load_lock, flags);
 	pcpu->time_in_idle =
-		get_cpu_idle_time_us(smp_processor_id(),
+		get_cpu_idle_time(smp_processor_id(),
 				     &pcpu->time_in_idle_timestamp);
 	pcpu->cputime_speedadj = 0;
 	pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp;
@@ -278,7 +317,7 @@ static u64 update_load(int cpu)
 	unsigned int delta_time;
 	u64 active_time;
 
-	now_idle = get_cpu_idle_time_us(cpu, &now);
+	now_idle = get_cpu_idle_time(cpu, &now);
 	delta_idle = (unsigned int)(now_idle - pcpu->time_in_idle);
 	delta_time = (unsigned int)(now - pcpu->time_in_idle_timestamp);
 	active_time = delta_time - delta_idle;
@@ -924,6 +963,28 @@ static ssize_t store_boostpulse_duration(
 
 define_one_global_rw(boostpulse_duration);
 
+static ssize_t show_io_is_busy(struct kobject *kobj,
+			struct attribute *attr, char *buf)
+{
+	return sprintf(buf, "%u\n", io_is_busy);
+}
+
+static ssize_t store_io_is_busy(struct kobject *kobj,
+			struct attribute *attr, const char *buf, size_t count)
+{
+	int ret;
+	unsigned long val;
+
+	ret = kstrtoul(buf, 0, &val);
+	if (ret < 0)
+		return ret;
+	io_is_busy = val;
+	return count;
+}
+
+static struct global_attr io_is_busy_attr = __ATTR(io_is_busy, 0644,
+		show_io_is_busy, store_io_is_busy);
+
 static struct attribute *interactive_attributes[] = {
 	&target_loads_attr.attr,
 	&above_hispeed_delay_attr.attr,
@@ -935,6 +996,7 @@ static struct attribute *interactive_attributes[] = {
 	&boost.attr,
 	&boostpulse.attr,
 	&boostpulse_duration.attr,
+	&io_is_busy_attr.attr,
 	NULL,
 };
 
-- 
1.7.9.5

[PATCH 44/70] cpufreq: interactive: fix crash on error paths in get_tokenized_data

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Use separate variable for error code, free proper pointer.

Change-Id: Ia83cccb195997789ac6afbf5b8761f7b278196d6
Reported-by: Arve Hjønnevåg <arve@android.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 7c734fa..3dc067e 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -647,29 +647,26 @@ static unsigned int *get_tokenized_data(const char *buf, int *num_tokens)
 	int i;
 	int ntokens = 1;
 	unsigned int *tokenized_data;
+	int err = -EINVAL;
 
 	cp = buf;
 	while ((cp = strpbrk(cp + 1, " :")))
 		ntokens++;
 
-	if (!(ntokens & 0x1)) {
-		tokenized_data = ERR_PTR(-EINVAL);
+	if (!(ntokens & 0x1))
 		goto err;
-	}
 
 	tokenized_data = kmalloc(ntokens * sizeof(unsigned int), GFP_KERNEL);
 	if (!tokenized_data) {
-		tokenized_data = ERR_PTR(-ENOMEM);
+		err = -ENOMEM;
 		goto err;
 	}
 
 	cp = buf;
 	i = 0;
 	while (i < ntokens) {
-		if (sscanf(cp, "%u", &tokenized_data[i++]) != 1) {
-			tokenized_data = ERR_PTR(-EINVAL);
+		if (sscanf(cp, "%u", &tokenized_data[i++]) != 1)
 			goto err_kfree;
-		}
 
 		cp = strpbrk(cp, " :");
 		if (!cp)
@@ -677,10 +674,8 @@ static unsigned int *get_tokenized_data(const char *buf, int *num_tokens)
 		cp++;
 	}
 
-	if (i != ntokens) {
-		tokenized_data = ERR_PTR(-EINVAL);
+	if (i != ntokens)
 		goto err_kfree;
-	}
 
 	*num_tokens = ntokens;
 	return tokenized_data;
@@ -688,7 +683,7 @@ static unsigned int *get_tokenized_data(const char *buf, int *num_tokens)
 err_kfree:
 	kfree(tokenized_data);
 err:
-	return tokenized_data;
+	return ERR_PTR(err);
 }
 
 static ssize_t show_target_loads(
-- 
1.7.9.5

[PATCH 45/70] cpufreq: interactive: base above_hispeed_delay on target freq, not current

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Time to wait should be based on the intended target speed, not the
actual speed (which may be held high by another CPU).

Change-Id: Ifc5bb55d06adddb9a02af90af05398a78f282272
Reported-by: Arve Hjønnevåg <arve@android.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 3dc067e..9dd4d38 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -377,7 +377,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	if (pcpu->target_freq >= hispeed_freq &&
 	    new_freq > pcpu->target_freq &&
 	    now - pcpu->hispeed_validate_time <
-	    freq_to_above_hispeed_delay(pcpu->policy->cur)) {
+	    freq_to_above_hispeed_delay(pcpu->target_freq)) {
 		trace_cpufreq_interactive_notyet(
 			data, cpu_load, pcpu->target_freq,
 			pcpu->policy->cur, new_freq);
-- 
1.7.9.5

[PATCH 46/70] cpufreq: interactive: fix uninitialized spinlock

[Bálint Czobor]

From: Minsung Kim <ms925.kim@samsung.com>

Add missing spinlock init

Backtrace:
[<c0011ce4>] (dump_backtrace+0x0/0x10c) from [<c0662a68>] (dump_stack+0x18/0x1c)
 r6:00000032 r5:c0bd09ec r4:e6848000 r3:00000000
[<c0662a50>] (dump_stack+0x0/0x1c) from [<c06670b0>] (spin_dump+0x80/0x94)
[<c0667030>] (spin_dump+0x0/0x94) from [<c06670f0>] (spin_bug+0x2c/0x30)
 r5:c08f91fc r4:c0bd09ec
[<c06670c4>] (spin_bug+0x0/0x30) from [<c0245f74>] (do_raw_spin_unlock+0x88/0xcc)
 r5:e547bac0 r4:c0bd09ec
[<c0245eec>] (do_raw_spin_unlock+0x0/0xcc) from [<c066c9cc>] (_raw_spin_unlock_irqrestore+0x14/0x40)
 r5:e547bac0 r4:60000013
[<c066c9b8>] (_raw_spin_unlock_irqrestore+0x0/0x40) from [<c044b884>] (store_above_hispeed_delay+0x6c/0x80)
 r4:c0b4cf78 r3:00000007
[<c044b818>] (store_above_hispeed_delay+0x0/0x80) from [<c0235d24>] (kobj_attr_store+0x1c/0x28)
 r7:e68ff000 r6:00000032 r5:e58137c0 r4:e61cde80
[<c0235d08>] (kobj_attr_store+0x0/0x28) from [<c0156b78>] (sysfs_write_file+0x104/0x184)
[<c0156a74>] (sysfs_write_file+0x0/0x184) from [<c0100680>] (vfs_write+0xb0/0x140)
[<c01005d0>] (vfs_write+0x0/0x140) from [<c0100900>] (sys_write+0x44/0x70)
 r8:00000000 r7:00000004 r6:00000032 r5:bee43c90 r4:e5600300
[<c01008bc>] (sys_write+0x0/0x70) from [<c000e400>] (ret_fast_syscall+0x0/0x30)
 r9:e6842000 r8:c000e584 r6:00000032 r5:bee43c90 r4:00000009

Change-Id: I80a1e0b3fecb24adba501ff44f568479deeff7fa
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 9dd4d38..baf8740 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -1148,6 +1148,7 @@ static int __init cpufreq_interactive_init(void)
 
 	spin_lock_init(&target_loads_lock);
 	spin_lock_init(&speedchange_cpumask_lock);
+	spin_lock_init(&above_hispeed_delay_lock);
 	mutex_init(&gov_lock);
 	speedchange_task =
 		kthread_create(cpufreq_interactive_speedchange_task, NULL,
-- 
1.7.9.5

[PATCH 47/70] cpufreq: interactive: handle errors from cpufreq_frequency_table_target

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Add checks for error return from cpufreq_frequency_table_target, and be
less noisy on the existing call with an error check.  CPU hotplug and
system shutdown may cause this call to return -EINVAL.

Bug: 8613560
Change-Id: Id78d8829920462c0db1c7e14e717d91740d6cb44
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index baf8740..1b5d930 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -246,9 +246,10 @@ static unsigned int choose_freq(
 		 * than or equal to the target load.
 		 */
 
-		cpufreq_frequency_table_target(
-			pcpu->policy, pcpu->freq_table, loadadjfreq / tl,
-			CPUFREQ_RELATION_L, &index);
+		if (cpufreq_frequency_table_target(
+			    pcpu->policy, pcpu->freq_table, loadadjfreq / tl,
+			    CPUFREQ_RELATION_L, &index))
+			break;
 		freq = pcpu->freq_table[index].frequency;
 
 		if (freq > prevfreq) {
@@ -260,10 +261,11 @@ static unsigned int choose_freq(
 				 * Find the highest frequency that is less
 				 * than freqmax.
 				 */
-				cpufreq_frequency_table_target(
-					pcpu->policy, pcpu->freq_table,
-					freqmax - 1, CPUFREQ_RELATION_H,
-					&index);
+				if (cpufreq_frequency_table_target(
+					    pcpu->policy, pcpu->freq_table,
+					    freqmax - 1, CPUFREQ_RELATION_H,
+					    &index))
+					break;
 				freq = pcpu->freq_table[index].frequency;
 
 				if (freq == freqmin) {
@@ -286,10 +288,11 @@ static unsigned int choose_freq(
 				 * Find the lowest frequency that is higher
 				 * than freqmin.
 				 */
-				cpufreq_frequency_table_target(
-					pcpu->policy, pcpu->freq_table,
-					freqmin + 1, CPUFREQ_RELATION_L,
-					&index);
+				if (cpufreq_frequency_table_target(
+					    pcpu->policy, pcpu->freq_table,
+					    freqmin + 1, CPUFREQ_RELATION_L,
+					    &index))
+					break;
 				freq = pcpu->freq_table[index].frequency;
 
 				/*
@@ -388,11 +391,8 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 	if (cpufreq_frequency_table_target(pcpu->policy, pcpu->freq_table,
 					   new_freq, CPUFREQ_RELATION_L,
-					   &index)) {
-		pr_warn_once("timer %d: cpufreq_frequency_table_target error\n",
-			     (int) data);
+					   &index))
 		goto rearm;
-	}
 
 	new_freq = pcpu->freq_table[index].frequency;
 
-- 
1.7.9.5

[PATCH 48/70] cpufreq: interactive: reduce chance of zero time delta on load eval

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

Reschedule load sampling timer after timestamp of sample start taken,
hold spinlock across entire sequence to avoid preemption.  Avoid the
WARN for zero time delta in the load sampling timer function.

Change-Id: Idc10a756f09141decb6df92669521a1ebf0dbc10
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 1b5d930..b4d12b2 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -169,21 +169,23 @@ static inline cputime64_t get_cpu_idle_time(unsigned int cpu,
 static void cpufreq_interactive_timer_resched(
 	struct cpufreq_interactive_cpuinfo *pcpu)
 {
-	unsigned long expires = jiffies + usecs_to_jiffies(timer_rate);
+	unsigned long expires;
 	unsigned long flags;
 
-	mod_timer_pinned(&pcpu->cpu_timer, expires);
-	if (timer_slack_val >= 0 && pcpu->target_freq > pcpu->policy->min) {
-		expires += usecs_to_jiffies(timer_slack_val);
-		mod_timer_pinned(&pcpu->cpu_slack_timer, expires);
-	}
-
 	spin_lock_irqsave(&pcpu->load_lock, flags);
 	pcpu->time_in_idle =
 		get_cpu_idle_time(smp_processor_id(),
 				     &pcpu->time_in_idle_timestamp);
 	pcpu->cputime_speedadj = 0;
 	pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp;
+	expires = jiffies + usecs_to_jiffies(timer_rate);
+	mod_timer_pinned(&pcpu->cpu_timer, expires);
+
+	if (timer_slack_val >= 0 && pcpu->target_freq > pcpu->policy->min) {
+		expires += usecs_to_jiffies(timer_slack_val);
+		mod_timer_pinned(&pcpu->cpu_slack_timer, expires);
+	}
+
 	spin_unlock_irqrestore(&pcpu->load_lock, flags);
 }
 
-- 
1.7.9.5

[PATCH 49/70] cpufreq: interactive: avoid underflow on active time calculation

[Bálint Czobor]

From: Minsung Kim <ms925.kim@samsung.com>

Check for idle time delta less than elapsed time delta, avoid
underflow computing active time.

Change-Id: I3e4c6ef1ad794eec49ed379c0c50fa727fd6ad28
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index b4d12b2..7303f50 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -325,7 +325,12 @@ static u64 update_load(int cpu)
 	now_idle = get_cpu_idle_time(cpu, &now);
 	delta_idle = (unsigned int)(now_idle - pcpu->time_in_idle);
 	delta_time = (unsigned int)(now - pcpu->time_in_idle_timestamp);
-	active_time = delta_time - delta_idle;
+
+	if (delta_time <= delta_idle)
+		active_time = 0;
+	else
+		active_time = delta_time - delta_idle;
+
 	pcpu->cputime_speedadj += active_time * pcpu->policy->cur;
 
 	pcpu->time_in_idle = now_idle;
-- 
1.7.9.5

[PATCH 50/70] cpufreq: interactive: fix race on cpufreq TRANSITION notifier

[Bálint Czobor]

From: Lianwei Wang <a22439@motorola.com>

The cpufreq TRANSTION notifier callback does not check the
governor_enabled state on affected CPUS, which will case
kernel panic in update_load because the policy object maybe
NULL or invalid when governor_enabled is false.

Change-Id: Ie0f1718124f61e2f9b5da57abc6981ada5b83908
Signed-off-by: Lianwei Wang <a22439@motorola.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 7303f50..691923d 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -634,9 +634,19 @@ static int cpufreq_interactive_notifier(
 		for_each_cpu(cpu, pcpu->policy->cpus) {
 			struct cpufreq_interactive_cpuinfo *pjcpu =
 				&per_cpu(cpuinfo, cpu);
+			if (cpu != freq->cpu) {
+				if (!down_read_trylock(&pjcpu->enable_sem))
+					continue;
+				if (!pjcpu->governor_enabled) {
+					up_read(&pjcpu->enable_sem);
+					continue;
+				}
+			}
 			spin_lock_irqsave(&pjcpu->load_lock, flags);
 			update_load(cpu);
 			spin_unlock_irqrestore(&pjcpu->load_lock, flags);
+			if (cpu != freq->cpu)
+				up_read(&pjcpu->enable_sem);
 		}
 
 		up_read(&pcpu->enable_sem);
-- 
1.7.9.5

[PATCH 51/70] cpufreq: interactive: resched timer if max freq raised

[Bálint Czobor]

From: Lianwei Wang <a22439@motorola.com>

When the policy max freq is raised, and before the timer is
rescheduled in idle callback, the cpu freq may stuck at a
lower freq.

The target_freq shall be updated too, else on a high load
situation, the new_freq is always equal to target_freq and
which will cause freq stuck at a lower freq too.

Reschedule the timer on gov limits callback.

Change-Id: I6c187001ab43e859731429b64f75a74eebc37a24
Signed-off-by: Lianwei Wang <a22439@motorola.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   64 +++++++++++++++++++++++++++------
 1 file changed, 54 insertions(+), 10 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 691923d..90958fd 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -189,6 +189,32 @@ static void cpufreq_interactive_timer_resched(
 	spin_unlock_irqrestore(&pcpu->load_lock, flags);
 }
 
+/* The caller shall take enable_sem write semaphore to avoid any timer race.
+ * The cpu_timer and cpu_slack_timer must be deactivated when calling this
+ * function.
+ */
+static void cpufreq_interactive_timer_start(int cpu)
+{
+	struct cpufreq_interactive_cpuinfo *pcpu = &per_cpu(cpuinfo, cpu);
+	unsigned long expires = jiffies + usecs_to_jiffies(timer_rate);
+	unsigned long flags;
+
+	pcpu->cpu_timer.expires = expires;
+	add_timer_on(&pcpu->cpu_timer, cpu);
+	if (timer_slack_val >= 0 && pcpu->target_freq > pcpu->policy->min) {
+		expires += usecs_to_jiffies(timer_slack_val);
+		pcpu->cpu_slack_timer.expires = expires;
+		add_timer_on(&pcpu->cpu_slack_timer, cpu);
+	}
+
+	spin_lock_irqsave(&pcpu->load_lock, flags);
+	pcpu->time_in_idle =
+		get_cpu_idle_time(cpu, &pcpu->time_in_idle_timestamp);
+	pcpu->cputime_speedadj = 0;
+	pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp;
+	spin_unlock_irqrestore(&pcpu->load_lock, flags);
+}
+
 static unsigned int freq_to_above_hispeed_delay(unsigned int freq)
 {
 	int i;
@@ -1058,8 +1084,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			hispeed_freq = policy->max;
 
 		for_each_cpu(j, policy->cpus) {
-			unsigned long expires;
-
 			pcpu = &per_cpu(cpuinfo, j);
 			pcpu->policy = policy;
 			pcpu->target_freq = policy->cur;
@@ -1070,14 +1094,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->hispeed_validate_time =
 				pcpu->floor_validate_time;
 			down_write(&pcpu->enable_sem);
-			expires = jiffies + usecs_to_jiffies(timer_rate);
-			pcpu->cpu_timer.expires = expires;
-			add_timer_on(&pcpu->cpu_timer, j);
-			if (timer_slack_val >= 0) {
-				expires += usecs_to_jiffies(timer_slack_val);
-				pcpu->cpu_slack_timer.expires = expires;
-				add_timer_on(&pcpu->cpu_slack_timer, j);
-			}
+			cpufreq_interactive_timer_start(j);
 			pcpu->governor_enabled = 1;
 			up_write(&pcpu->enable_sem);
 		}
@@ -1136,6 +1153,33 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		else if (policy->min > policy->cur)
 			__cpufreq_driver_target(policy,
 					policy->min, CPUFREQ_RELATION_L);
+		for_each_cpu(j, policy->cpus) {
+			pcpu = &per_cpu(cpuinfo, j);
+
+			/* hold write semaphore to avoid race */
+			down_write(&pcpu->enable_sem);
+			if (pcpu->governor_enabled == 0) {
+				up_write(&pcpu->enable_sem);
+				continue;
+			}
+
+			/* update target_freq firstly */
+			if (policy->max < pcpu->target_freq)
+				pcpu->target_freq = policy->max;
+			else if (policy->min > pcpu->target_freq)
+				pcpu->target_freq = policy->min;
+
+			/* Reschedule timer.
+			 * Delete the timers, else the timer callback may
+			 * return without re-arm the timer when failed
+			 * acquire the semaphore. This race may cause timer
+			 * stopped unexpectedly.
+			 */
+			del_timer_sync(&pcpu->cpu_timer);
+			del_timer_sync(&pcpu->cpu_slack_timer);
+			cpufreq_interactive_timer_start(j);
+			up_write(&pcpu->enable_sem);
+		}
 		break;
 	}
 	return 0;
-- 
1.7.9.5

[PATCH 52/70] cpufreq: interactive: fix show_target_loads and show_above_hispeed_delay

[Bálint Czobor]

From: Minsung Kim <ms925.kim@samsung.com>

Remove a trailing whitespace from target_loads and above_hispeed_delay. Problem
happens when user-space program tried to restore parameters that saved before
changing parameters. In this case was returned error(EINVAL).

Change-Id: I5a74e3824602cd6f2b74651adda5ec1b627e61e9
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 90958fd..a66748e 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -742,7 +742,7 @@ static ssize_t show_target_loads(
 		ret += sprintf(buf + ret, "%u%s", target_loads[i],
 			       i & 0x1 ? ":" : " ");
 
-	ret += sprintf(buf + ret, "\n");
+	ret += sprintf(buf + --ret, "\n");
 	spin_unlock_irqrestore(&target_loads_lock, flags);
 	return ret;
 }
@@ -785,7 +785,7 @@ static ssize_t show_above_hispeed_delay(
 		ret += sprintf(buf + ret, "%u%s", above_hispeed_delay[i],
 			       i & 0x1 ? ":" : " ");
 
-	ret += sprintf(buf + ret, "\n");
+	ret += sprintf(buf + --ret, "\n");
 	spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
 	return ret;
 }
-- 
1.7.9.5

[PATCH 53/70] cpufreq: interactive: Remove unnecessary cpu_online() check

[Bálint Czobor]

From: Viresh Kumar <viresh.kumar@linaro.org>

Cpufreq no longer calls governor callback for offlined cpus. i.e. All
policy->cpus are guaranteed to be online. Hence we don't need explicit check to
see if cpu is online or not.

Change-Id: I9ad85ea4addd5b4a40952e59ed730dd15e328690
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    3 ---
 1 file changed, 3 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index a66748e..378835b 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -1073,9 +1073,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 
 	switch (event) {
 	case CPUFREQ_GOV_START:
-		if (!cpu_online(policy->cpu))
-			return -EINVAL;
-
 		mutex_lock(&gov_lock);
 
 		freq_table =
-- 
1.7.9.5

[PATCH 54/70] cpufreq: interactive: Move definition of cpufreq_gov_interactive downwards

[Bálint Czobor]

From: Viresh Kumar <viresh.kumar@linaro.org>

This moves definition of cpufreq_gov_interactive towards the bottom of file, so
that we don't have to add prototype of cpufreq_governor_interactive() in the
beginning of file.

Change-Id: I04bd1004954eb36502c5cd7e35d3d7274cddaf95
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 378835b..b0685c7 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -117,19 +117,6 @@ static int timer_slack_val = DEFAULT_TIMER_SLACK;
 
 static bool io_is_busy;
 
-static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
-		unsigned int event);
-
-#ifndef CONFIG_CPU_FREQ_DEFAULT_GOV_INTERACTIVE
-static
-#endif
-struct cpufreq_governor cpufreq_gov_interactive = {
-	.name = "interactive",
-	.governor = cpufreq_governor_interactive,
-	.max_transition_latency = 10000000,
-	.owner = THIS_MODULE,
-};
-
 static inline cputime64_t get_cpu_idle_time_jiffy(unsigned int cpu,
 						  cputime64_t *wall)
 {
@@ -1182,6 +1169,16 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 	return 0;
 }
 
+#ifndef CONFIG_CPU_FREQ_DEFAULT_GOV_INTERACTIVE
+static
+#endif
+struct cpufreq_governor cpufreq_gov_interactive = {
+	.name = "interactive",
+	.governor = cpufreq_governor_interactive,
+	.max_transition_latency = 10000000,
+	.owner = THIS_MODULE,
+};
+
 static void cpufreq_interactive_nop_timer(unsigned long data)
 {
 }
-- 
1.7.9.5

[PATCH 55/70] cpufreq: Interactive: Implement per policy instances of governor

[Bálint Czobor]

From: Viresh Kumar <viresh.kumar@linaro.org>

If we have a multi-package system, where we have multiple instances of struct
policy (per package), currently we can't have multiple instances of same
governor. i.e. We can't have multiple instances of Interactive governor for
multiple packages.

This is a bottleneck for multicluster system, where we want different packages
to use Interactive governor, but with different tunables.

This patch uses the infrastructure provided by earlier patches pushed in
Mainline in v3.10-rc1/rc2 and implements per policy instances of Interactive
governor.

Change-Id: I70436d4a5a45c6cb6edf37f3e46d0b9fbc930982
[toddpoynor@google.com: merge with later code, minor changes]
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |  629 ++++++++++++++++++++-------------
 1 file changed, 383 insertions(+), 246 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index b0685c7..66c096d 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -36,8 +36,6 @@
 #define CREATE_TRACE_POINTS
 #include <trace/events/cpufreq_interactive.h>
 
-static int active_count;
-
 struct cpufreq_interactive_cpuinfo {
 	struct timer_list cpu_timer;
 	struct timer_list cpu_slack_timer;
@@ -64,58 +62,62 @@ static cpumask_t speedchange_cpumask;
 static spinlock_t speedchange_cpumask_lock;
 static struct mutex gov_lock;
 
-/* Hi speed to bump to from lo speed when load burst (default max) */
-static unsigned int hispeed_freq;
-
-/* Go to hi speed when CPU load at or above this value. */
-#define DEFAULT_GO_HISPEED_LOAD 99
-static unsigned long go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;
-
 /* Target load.  Lower values result in higher CPU speeds. */
 #define DEFAULT_TARGET_LOAD 90
 static unsigned int default_target_loads[] = {DEFAULT_TARGET_LOAD};
-static spinlock_t target_loads_lock;
-static unsigned int *target_loads = default_target_loads;
-static int ntarget_loads = ARRAY_SIZE(default_target_loads);
-
-/*
- * The minimum amount of time to spend at a frequency before we can ramp down.
- */
-#define DEFAULT_MIN_SAMPLE_TIME (80 * USEC_PER_MSEC)
-static unsigned long min_sample_time = DEFAULT_MIN_SAMPLE_TIME;
 
-/*
- * The sample rate of the timer used to increase frequency
- */
 #define DEFAULT_TIMER_RATE (20 * USEC_PER_MSEC)
-static unsigned long timer_rate = DEFAULT_TIMER_RATE;
-
-/*
- * Wait this long before raising speed above hispeed, by default a single
- * timer interval.
- */
 #define DEFAULT_ABOVE_HISPEED_DELAY DEFAULT_TIMER_RATE
 static unsigned int default_above_hispeed_delay[] = {
 	DEFAULT_ABOVE_HISPEED_DELAY };
-static spinlock_t above_hispeed_delay_lock;
-static unsigned int *above_hispeed_delay = default_above_hispeed_delay;
-static int nabove_hispeed_delay = ARRAY_SIZE(default_above_hispeed_delay);
-
-/* Non-zero means indefinite speed boost active */
-static int boost_val;
-/* Duration of a boot pulse in usecs */
-static int boostpulse_duration_val = DEFAULT_MIN_SAMPLE_TIME;
-/* End time of boost pulse in ktime converted to usecs */
-static u64 boostpulse_endtime;
 
-/*
- * Max additional time to wait in idle, beyond timer_rate, at speeds above
- * minimum before wakeup to reduce speed, or -1 if unnecessary.
- */
+struct cpufreq_interactive_tunables {
+	int usage_count;
+	/* Hi speed to bump to from lo speed when load burst (default max) */
+	unsigned int hispeed_freq;
+	/* Go to hi speed when CPU load at or above this value. */
+#define DEFAULT_GO_HISPEED_LOAD 99
+	unsigned long go_hispeed_load;
+	/* Target load. Lower values result in higher CPU speeds. */
+	spinlock_t target_loads_lock;
+	unsigned int *target_loads;
+	int ntarget_loads;
+	/*
+	 * The minimum amount of time to spend at a frequency before we can ramp
+	 * down.
+	 */
+#define DEFAULT_MIN_SAMPLE_TIME (80 * USEC_PER_MSEC)
+	unsigned long min_sample_time;
+	/*
+	 * The sample rate of the timer used to increase frequency
+	 */
+	unsigned long timer_rate;
+	/*
+	 * Wait this long before raising speed above hispeed, by default a
+	 * single timer interval.
+	 */
+	spinlock_t above_hispeed_delay_lock;
+	unsigned int *above_hispeed_delay;
+	int nabove_hispeed_delay;
+	/* Non-zero means indefinite speed boost active */
+	int boost_val;
+	/* Duration of a boot pulse in usecs */
+	int boostpulse_duration_val;
+	/* End time of boost pulse in ktime converted to usecs */
+	u64 boostpulse_endtime;
+	/*
+	 * Max additional time to wait in idle, beyond timer_rate, at speeds
+	 * above minimum before wakeup to reduce speed, or -1 if unnecessary.
+	 */
 #define DEFAULT_TIMER_SLACK (4 * DEFAULT_TIMER_RATE)
-static int timer_slack_val = DEFAULT_TIMER_SLACK;
+	int timer_slack_val;
+	bool io_is_busy;
+};
+
+/* For cases where we have single governor instance for system */
+struct cpufreq_interactive_tunables *common_tunables;
 
-static bool io_is_busy;
+static struct attribute_group *get_sysfs_attr(void);
 
 static inline cputime64_t get_cpu_idle_time_jiffy(unsigned int cpu,
 						  cputime64_t *wall)
@@ -140,8 +142,10 @@ static inline cputime64_t get_cpu_idle_time_jiffy(unsigned int cpu,
 	return jiffies_to_usecs(idle_time);
 }
 
-static inline cputime64_t get_cpu_idle_time(unsigned int cpu,
-					    cputime64_t *wall)
+static inline cputime64_t get_cpu_idle_time(
+	unsigned int cpu,
+	cputime64_t *wall,
+	bool io_is_busy)
 {
 	u64 idle_time = get_cpu_idle_time_us(cpu, wall);
 
@@ -156,20 +160,24 @@ static inline cputime64_t get_cpu_idle_time(unsigned int cpu,
 static void cpufreq_interactive_timer_resched(
 	struct cpufreq_interactive_cpuinfo *pcpu)
 {
+	struct cpufreq_interactive_tunables *tunables =
+		pcpu->policy->governor_data;
 	unsigned long expires;
 	unsigned long flags;
 
 	spin_lock_irqsave(&pcpu->load_lock, flags);
 	pcpu->time_in_idle =
 		get_cpu_idle_time(smp_processor_id(),
-				     &pcpu->time_in_idle_timestamp);
+				  &pcpu->time_in_idle_timestamp,
+				  tunables->io_is_busy);
 	pcpu->cputime_speedadj = 0;
 	pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp;
-	expires = jiffies + usecs_to_jiffies(timer_rate);
+	expires = jiffies + usecs_to_jiffies(tunables->timer_rate);
 	mod_timer_pinned(&pcpu->cpu_timer, expires);
 
-	if (timer_slack_val >= 0 && pcpu->target_freq > pcpu->policy->min) {
-		expires += usecs_to_jiffies(timer_slack_val);
+	if (tunables->timer_slack_val >= 0 &&
+	    pcpu->target_freq > pcpu->policy->min) {
+		expires += usecs_to_jiffies(tunables->timer_slack_val);
 		mod_timer_pinned(&pcpu->cpu_slack_timer, expires);
 	}
 
@@ -180,58 +188,66 @@ static void cpufreq_interactive_timer_resched(
  * The cpu_timer and cpu_slack_timer must be deactivated when calling this
  * function.
  */
-static void cpufreq_interactive_timer_start(int cpu)
+static void cpufreq_interactive_timer_start(
+	struct cpufreq_interactive_tunables *tunables, int cpu)
 {
 	struct cpufreq_interactive_cpuinfo *pcpu = &per_cpu(cpuinfo, cpu);
-	unsigned long expires = jiffies + usecs_to_jiffies(timer_rate);
+	unsigned long expires = jiffies +
+		usecs_to_jiffies(tunables->timer_rate);
 	unsigned long flags;
 
 	pcpu->cpu_timer.expires = expires;
 	add_timer_on(&pcpu->cpu_timer, cpu);
-	if (timer_slack_val >= 0 && pcpu->target_freq > pcpu->policy->min) {
-		expires += usecs_to_jiffies(timer_slack_val);
+	if (tunables->timer_slack_val >= 0 &&
+	    pcpu->target_freq > pcpu->policy->min) {
+		expires += usecs_to_jiffies(tunables->timer_slack_val);
 		pcpu->cpu_slack_timer.expires = expires;
 		add_timer_on(&pcpu->cpu_slack_timer, cpu);
 	}
 
 	spin_lock_irqsave(&pcpu->load_lock, flags);
 	pcpu->time_in_idle =
-		get_cpu_idle_time(cpu, &pcpu->time_in_idle_timestamp);
+		get_cpu_idle_time(cpu, &pcpu->time_in_idle_timestamp,
+				  tunables->io_is_busy);
 	pcpu->cputime_speedadj = 0;
 	pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp;
 	spin_unlock_irqrestore(&pcpu->load_lock, flags);
 }
 
-static unsigned int freq_to_above_hispeed_delay(unsigned int freq)
+static unsigned int freq_to_above_hispeed_delay(
+	struct cpufreq_interactive_tunables *tunables,
+	unsigned int freq)
 {
 	int i;
 	unsigned int ret;
 	unsigned long flags;
 
-	spin_lock_irqsave(&above_hispeed_delay_lock, flags);
+	spin_lock_irqsave(&tunables->above_hispeed_delay_lock, flags);
 
-	for (i = 0; i < nabove_hispeed_delay - 1 &&
-			freq >= above_hispeed_delay[i+1]; i += 2)
+	for (i = 0; i < tunables->nabove_hispeed_delay - 1 &&
+			freq >= tunables->above_hispeed_delay[i+1]; i += 2)
 		;
 
-	ret = above_hispeed_delay[i];
-	spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
+	ret = tunables->above_hispeed_delay[i];
+	spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags);
 	return ret;
 }
 
-static unsigned int freq_to_targetload(unsigned int freq)
+static unsigned int freq_to_targetload(
+	struct cpufreq_interactive_tunables *tunables, unsigned int freq)
 {
 	int i;
 	unsigned int ret;
 	unsigned long flags;
 
-	spin_lock_irqsave(&target_loads_lock, flags);
+	spin_lock_irqsave(&tunables->target_loads_lock, flags);
 
-	for (i = 0; i < ntarget_loads - 1 && freq >= target_loads[i+1]; i += 2)
+	for (i = 0; i < tunables->ntarget_loads - 1 &&
+		    freq >= tunables->target_loads[i+1]; i += 2)
 		;
 
-	ret = target_loads[i];
-	spin_unlock_irqrestore(&target_loads_lock, flags);
+	ret = tunables->target_loads[i];
+	spin_unlock_irqrestore(&tunables->target_loads_lock, flags);
 	return ret;
 }
 
@@ -240,9 +256,8 @@ static unsigned int freq_to_targetload(unsigned int freq)
  * choose_freq() will find the minimum frequency that does not exceed its
  * target load given the current load.
  */
-
-static unsigned int choose_freq(
-	struct cpufreq_interactive_cpuinfo *pcpu, unsigned int loadadjfreq)
+static unsigned int choose_freq(struct cpufreq_interactive_cpuinfo *pcpu,
+		unsigned int loadadjfreq)
 {
 	unsigned int freq = pcpu->policy->cur;
 	unsigned int prevfreq, freqmin, freqmax;
@@ -254,7 +269,7 @@ static unsigned int choose_freq(
 
 	do {
 		prevfreq = freq;
-		tl = freq_to_targetload(freq);
+		tl = freq_to_targetload(pcpu->policy->governor_data, freq);
 
 		/*
 		 * Find the lowest frequency where the computed load is less
@@ -329,13 +344,15 @@ static unsigned int choose_freq(
 static u64 update_load(int cpu)
 {
 	struct cpufreq_interactive_cpuinfo *pcpu = &per_cpu(cpuinfo, cpu);
+	struct cpufreq_interactive_tunables *tunables =
+		pcpu->policy->governor_data;
 	u64 now;
 	u64 now_idle;
 	unsigned int delta_idle;
 	unsigned int delta_time;
 	u64 active_time;
 
-	now_idle = get_cpu_idle_time(cpu, &now);
+	now_idle = get_cpu_idle_time(cpu, &now, tunables->io_is_busy);
 	delta_idle = (unsigned int)(now_idle - pcpu->time_in_idle);
 	delta_time = (unsigned int)(now - pcpu->time_in_idle_timestamp);
 
@@ -359,6 +376,8 @@ static void cpufreq_interactive_timer(unsigned long data)
 	int cpu_load;
 	struct cpufreq_interactive_cpuinfo *pcpu =
 		&per_cpu(cpuinfo, data);
+	struct cpufreq_interactive_tunables *tunables =
+		pcpu->policy->governor_data;
 	unsigned int new_freq;
 	unsigned int loadadjfreq;
 	unsigned int index;
@@ -382,25 +401,25 @@ static void cpufreq_interactive_timer(unsigned long data)
 	do_div(cputime_speedadj, delta_time);
 	loadadjfreq = (unsigned int)cputime_speedadj * 100;
 	cpu_load = loadadjfreq / pcpu->target_freq;
-	boosted = boost_val || now < boostpulse_endtime;
+	boosted = tunables->boost_val || now < tunables->boostpulse_endtime;
 
-	if (cpu_load >= go_hispeed_load || boosted) {
-		if (pcpu->target_freq < hispeed_freq) {
-			new_freq = hispeed_freq;
+	if (cpu_load >= tunables->go_hispeed_load || boosted) {
+		if (pcpu->target_freq < tunables->hispeed_freq) {
+			new_freq = tunables->hispeed_freq;
 		} else {
 			new_freq = choose_freq(pcpu, loadadjfreq);
 
-			if (new_freq < hispeed_freq)
-				new_freq = hispeed_freq;
+			if (new_freq < tunables->hispeed_freq)
+				new_freq = tunables->hispeed_freq;
 		}
 	} else {
 		new_freq = choose_freq(pcpu, loadadjfreq);
 	}
 
-	if (pcpu->target_freq >= hispeed_freq &&
+	if (pcpu->target_freq >= tunables->hispeed_freq &&
 	    new_freq > pcpu->target_freq &&
 	    now - pcpu->hispeed_validate_time <
-	    freq_to_above_hispeed_delay(pcpu->target_freq)) {
+	    freq_to_above_hispeed_delay(tunables, pcpu->target_freq)) {
 		trace_cpufreq_interactive_notyet(
 			data, cpu_load, pcpu->target_freq,
 			pcpu->policy->cur, new_freq);
@@ -421,7 +440,8 @@ static void cpufreq_interactive_timer(unsigned long data)
 	 * floor frequency for the minimum sample time since last validated.
 	 */
 	if (new_freq < pcpu->floor_freq) {
-		if (now - pcpu->floor_validate_time < min_sample_time) {
+		if (now - pcpu->floor_validate_time <
+				tunables->min_sample_time) {
 			trace_cpufreq_interactive_notyet(
 				data, cpu_load, pcpu->target_freq,
 				pcpu->policy->cur, new_freq);
@@ -437,7 +457,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	 * (or the indefinite boost is turned off).
 	 */
 
-	if (!boosted || new_freq > hispeed_freq) {
+	if (!boosted || new_freq > tunables->hispeed_freq) {
 		pcpu->floor_freq = new_freq;
 		pcpu->floor_validate_time = now;
 	}
@@ -598,14 +618,16 @@ static void cpufreq_interactive_boost(void)
 	int anyboost = 0;
 	unsigned long flags;
 	struct cpufreq_interactive_cpuinfo *pcpu;
+	struct cpufreq_interactive_tunables *tunables;
 
 	spin_lock_irqsave(&speedchange_cpumask_lock, flags);
 
 	for_each_online_cpu(i) {
 		pcpu = &per_cpu(cpuinfo, i);
+		tunables = pcpu->policy->governor_data;
 
-		if (pcpu->target_freq < hispeed_freq) {
-			pcpu->target_freq = hispeed_freq;
+		if (pcpu->target_freq < tunables->hispeed_freq) {
+			pcpu->target_freq = tunables->hispeed_freq;
 			cpumask_set_cpu(i, &speedchange_cpumask);
 			pcpu->hispeed_validate_time =
 				ktime_to_us(ktime_get());
@@ -617,7 +639,7 @@ static void cpufreq_interactive_boost(void)
 		 * validated.
 		 */
 
-		pcpu->floor_freq = hispeed_freq;
+		pcpu->floor_freq = tunables->hispeed_freq;
 		pcpu->floor_validate_time = ktime_to_us(ktime_get());
 	}
 
@@ -717,26 +739,27 @@ err:
 }
 
 static ssize_t show_target_loads(
-	struct kobject *kobj, struct attribute *attr, char *buf)
+	struct cpufreq_interactive_tunables *tunables,
+	char *buf)
 {
 	int i;
 	ssize_t ret = 0;
 	unsigned long flags;
 
-	spin_lock_irqsave(&target_loads_lock, flags);
+	spin_lock_irqsave(&tunables->target_loads_lock, flags);
 
-	for (i = 0; i < ntarget_loads; i++)
-		ret += sprintf(buf + ret, "%u%s", target_loads[i],
+	for (i = 0; i < tunables->ntarget_loads; i++)
+		ret += sprintf(buf + ret, "%u%s", tunables->target_loads[i],
 			       i & 0x1 ? ":" : " ");
 
 	ret += sprintf(buf + --ret, "\n");
-	spin_unlock_irqrestore(&target_loads_lock, flags);
+	spin_unlock_irqrestore(&tunables->target_loads_lock, flags);
 	return ret;
 }
 
 static ssize_t store_target_loads(
-	struct kobject *kobj, struct attribute *attr, const char *buf,
-	size_t count)
+	struct cpufreq_interactive_tunables *tunables,
+	const char *buf, size_t count)
 {
 	int ntokens;
 	unsigned int *new_target_loads = NULL;
@@ -746,40 +769,37 @@ static ssize_t store_target_loads(
 	if (IS_ERR(new_target_loads))
 		return PTR_RET(new_target_loads);
 
-	spin_lock_irqsave(&target_loads_lock, flags);
-	if (target_loads != default_target_loads)
-		kfree(target_loads);
-	target_loads = new_target_loads;
-	ntarget_loads = ntokens;
-	spin_unlock_irqrestore(&target_loads_lock, flags);
+	spin_lock_irqsave(&tunables->target_loads_lock, flags);
+	if (tunables->target_loads != default_target_loads)
+		kfree(tunables->target_loads);
+	tunables->target_loads = new_target_loads;
+	tunables->ntarget_loads = ntokens;
+	spin_unlock_irqrestore(&tunables->target_loads_lock, flags);
 	return count;
 }
 
-static struct global_attr target_loads_attr =
-	__ATTR(target_loads, S_IRUGO | S_IWUSR,
-		show_target_loads, store_target_loads);
-
 static ssize_t show_above_hispeed_delay(
-	struct kobject *kobj, struct attribute *attr, char *buf)
+	struct cpufreq_interactive_tunables *tunables, char *buf)
 {
 	int i;
 	ssize_t ret = 0;
 	unsigned long flags;
 
-	spin_lock_irqsave(&above_hispeed_delay_lock, flags);
+	spin_lock_irqsave(&tunables->above_hispeed_delay_lock, flags);
 
-	for (i = 0; i < nabove_hispeed_delay; i++)
-		ret += sprintf(buf + ret, "%u%s", above_hispeed_delay[i],
+	for (i = 0; i < tunables->nabove_hispeed_delay; i++)
+		ret += sprintf(buf + ret, "%u%s",
+			       tunables->above_hispeed_delay[i],
 			       i & 0x1 ? ":" : " ");
 
 	ret += sprintf(buf + --ret, "\n");
-	spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
+	spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags);
 	return ret;
 }
 
 static ssize_t store_above_hispeed_delay(
-	struct kobject *kobj, struct attribute *attr, const char *buf,
-	size_t count)
+	struct cpufreq_interactive_tunables *tunables,
+	const char *buf, size_t count)
 {
 	int ntokens;
 	unsigned int *new_above_hispeed_delay = NULL;
@@ -789,29 +809,24 @@ static ssize_t store_above_hispeed_delay(
 	if (IS_ERR(new_above_hispeed_delay))
 		return PTR_RET(new_above_hispeed_delay);
 
-	spin_lock_irqsave(&above_hispeed_delay_lock, flags);
-	if (above_hispeed_delay != default_above_hispeed_delay)
-		kfree(above_hispeed_delay);
-	above_hispeed_delay = new_above_hispeed_delay;
-	nabove_hispeed_delay = ntokens;
-	spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
+	spin_lock_irqsave(&tunables->above_hispeed_delay_lock, flags);
+	if (tunables->above_hispeed_delay != default_above_hispeed_delay)
+		kfree(tunables->above_hispeed_delay);
+	tunables->above_hispeed_delay = new_above_hispeed_delay;
+	tunables->nabove_hispeed_delay = ntokens;
+	spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags);
 	return count;
 
 }
 
-static struct global_attr above_hispeed_delay_attr =
-	__ATTR(above_hispeed_delay, S_IRUGO | S_IWUSR,
-		show_above_hispeed_delay, store_above_hispeed_delay);
-
-static ssize_t show_hispeed_freq(struct kobject *kobj,
-				 struct attribute *attr, char *buf)
+static ssize_t show_hispeed_freq(struct cpufreq_interactive_tunables *tunables,
+		char *buf)
 {
-	return sprintf(buf, "%u\n", hispeed_freq);
+	return sprintf(buf, "%u\n", tunables->hispeed_freq);
 }
 
-static ssize_t store_hispeed_freq(struct kobject *kobj,
-				  struct attribute *attr, const char *buf,
-				  size_t count)
+static ssize_t store_hispeed_freq(struct cpufreq_interactive_tunables *tunables,
+		const char *buf, size_t count)
 {
 	int ret;
 	long unsigned int val;
@@ -819,22 +834,18 @@ static ssize_t store_hispeed_freq(struct kobject *kobj,
 	ret = strict_strtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
-	hispeed_freq = val;
+	tunables->hispeed_freq = val;
 	return count;
 }
 
-static struct global_attr hispeed_freq_attr = __ATTR(hispeed_freq, 0644,
-		show_hispeed_freq, store_hispeed_freq);
-
-
-static ssize_t show_go_hispeed_load(struct kobject *kobj,
-				     struct attribute *attr, char *buf)
+static ssize_t show_go_hispeed_load(struct cpufreq_interactive_tunables
+		*tunables, char *buf)
 {
-	return sprintf(buf, "%lu\n", go_hispeed_load);
+	return sprintf(buf, "%lu\n", tunables->go_hispeed_load);
 }
 
-static ssize_t store_go_hispeed_load(struct kobject *kobj,
-			struct attribute *attr, const char *buf, size_t count)
+static ssize_t store_go_hispeed_load(struct cpufreq_interactive_tunables
+		*tunables, const char *buf, size_t count)
 {
 	int ret;
 	unsigned long val;
@@ -842,21 +853,18 @@ static ssize_t store_go_hispeed_load(struct kobject *kobj,
 	ret = strict_strtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
-	go_hispeed_load = val;
+	tunables->go_hispeed_load = val;
 	return count;
 }
 
-static struct global_attr go_hispeed_load_attr = __ATTR(go_hispeed_load, 0644,
-		show_go_hispeed_load, store_go_hispeed_load);
-
-static ssize_t show_min_sample_time(struct kobject *kobj,
-				struct attribute *attr, char *buf)
+static ssize_t show_min_sample_time(struct cpufreq_interactive_tunables
+		*tunables, char *buf)
 {
-	return sprintf(buf, "%lu\n", min_sample_time);
+	return sprintf(buf, "%lu\n", tunables->min_sample_time);
 }
 
-static ssize_t store_min_sample_time(struct kobject *kobj,
-			struct attribute *attr, const char *buf, size_t count)
+static ssize_t store_min_sample_time(struct cpufreq_interactive_tunables
+		*tunables, const char *buf, size_t count)
 {
 	int ret;
 	unsigned long val;
@@ -864,21 +872,18 @@ static ssize_t store_min_sample_time(struct kobject *kobj,
 	ret = strict_strtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
-	min_sample_time = val;
+	tunables->min_sample_time = val;
 	return count;
 }
 
-static struct global_attr min_sample_time_attr = __ATTR(min_sample_time, 0644,
-		show_min_sample_time, store_min_sample_time);
-
-static ssize_t show_timer_rate(struct kobject *kobj,
-			struct attribute *attr, char *buf)
+static ssize_t show_timer_rate(struct cpufreq_interactive_tunables *tunables,
+		char *buf)
 {
-	return sprintf(buf, "%lu\n", timer_rate);
+	return sprintf(buf, "%lu\n", tunables->timer_rate);
 }
 
-static ssize_t store_timer_rate(struct kobject *kobj,
-			struct attribute *attr, const char *buf, size_t count)
+static ssize_t store_timer_rate(struct cpufreq_interactive_tunables *tunables,
+		const char *buf, size_t count)
 {
 	int ret;
 	unsigned long val;
@@ -886,22 +891,18 @@ static ssize_t store_timer_rate(struct kobject *kobj,
 	ret = strict_strtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
-	timer_rate = val;
+	tunables->timer_rate = val;
 	return count;
 }
 
-static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644,
-		show_timer_rate, store_timer_rate);
-
-static ssize_t show_timer_slack(
-	struct kobject *kobj, struct attribute *attr, char *buf)
+static ssize_t show_timer_slack(struct cpufreq_interactive_tunables *tunables,
+		char *buf)
 {
-	return sprintf(buf, "%d\n", timer_slack_val);
+	return sprintf(buf, "%d\n", tunables->timer_slack_val);
 }
 
-static ssize_t store_timer_slack(
-	struct kobject *kobj, struct attribute *attr, const char *buf,
-	size_t count)
+static ssize_t store_timer_slack(struct cpufreq_interactive_tunables *tunables,
+		const char *buf, size_t count)
 {
 	int ret;
 	unsigned long val;
@@ -910,19 +911,17 @@ static ssize_t store_timer_slack(
 	if (ret < 0)
 		return ret;
 
-	timer_slack_val = val;
+	tunables->timer_slack_val = val;
 	return count;
 }
 
-define_one_global_rw(timer_slack);
-
-static ssize_t show_boost(struct kobject *kobj, struct attribute *attr,
+static ssize_t show_boost(struct cpufreq_interactive_tunables *tunables,
 			  char *buf)
 {
-	return sprintf(buf, "%d\n", boost_val);
+	return sprintf(buf, "%d\n", tunables->boost_val);
 }
 
-static ssize_t store_boost(struct kobject *kobj, struct attribute *attr,
+static ssize_t store_boost(struct cpufreq_interactive_tunables *tunables,
 			   const char *buf, size_t count)
 {
 	int ret;
@@ -932,9 +931,9 @@ static ssize_t store_boost(struct kobject *kobj, struct attribute *attr,
 	if (ret < 0)
 		return ret;
 
-	boost_val = val;
+	tunables->boost_val = val;
 
-	if (boost_val) {
+	if (tunables->boost_val) {
 		trace_cpufreq_interactive_boost("on");
 		cpufreq_interactive_boost();
 	} else {
@@ -944,9 +943,7 @@ static ssize_t store_boost(struct kobject *kobj, struct attribute *attr,
 	return count;
 }
 
-define_one_global_rw(boost);
-
-static ssize_t store_boostpulse(struct kobject *kobj, struct attribute *attr,
+static ssize_t store_boostpulse(struct cpufreq_interactive_tunables *tunables,
 				const char *buf, size_t count)
 {
 	int ret;
@@ -956,24 +953,21 @@ static ssize_t store_boostpulse(struct kobject *kobj, struct attribute *attr,
 	if (ret < 0)
 		return ret;
 
-	boostpulse_endtime = ktime_to_us(ktime_get()) + boostpulse_duration_val;
+	tunables->boostpulse_endtime = ktime_to_us(ktime_get()) +
+		tunables->boostpulse_duration_val;
 	trace_cpufreq_interactive_boost("pulse");
 	cpufreq_interactive_boost();
 	return count;
 }
 
-static struct global_attr boostpulse =
-	__ATTR(boostpulse, 0200, NULL, store_boostpulse);
-
-static ssize_t show_boostpulse_duration(
-	struct kobject *kobj, struct attribute *attr, char *buf)
+static ssize_t show_boostpulse_duration(struct cpufreq_interactive_tunables
+		*tunables, char *buf)
 {
-	return sprintf(buf, "%d\n", boostpulse_duration_val);
+	return sprintf(buf, "%d\n", tunables->boostpulse_duration_val);
 }
 
-static ssize_t store_boostpulse_duration(
-	struct kobject *kobj, struct attribute *attr, const char *buf,
-	size_t count)
+static ssize_t store_boostpulse_duration(struct cpufreq_interactive_tunables
+		*tunables, const char *buf, size_t count)
 {
 	int ret;
 	unsigned long val;
@@ -982,20 +976,18 @@ static ssize_t store_boostpulse_duration(
 	if (ret < 0)
 		return ret;
 
-	boostpulse_duration_val = val;
+	tunables->boostpulse_duration_val = val;
 	return count;
 }
 
-define_one_global_rw(boostpulse_duration);
-
-static ssize_t show_io_is_busy(struct kobject *kobj,
-			struct attribute *attr, char *buf)
+static ssize_t show_io_is_busy(struct cpufreq_interactive_tunables *tunables,
+		char *buf)
 {
-	return sprintf(buf, "%u\n", io_is_busy);
+	return sprintf(buf, "%u\n", tunables->io_is_busy);
 }
 
-static ssize_t store_io_is_busy(struct kobject *kobj,
-			struct attribute *attr, const char *buf, size_t count)
+static ssize_t store_io_is_busy(struct cpufreq_interactive_tunables *tunables,
+		const char *buf, size_t count)
 {
 	int ret;
 	unsigned long val;
@@ -1003,33 +995,137 @@ static ssize_t store_io_is_busy(struct kobject *kobj,
 	ret = kstrtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
-	io_is_busy = val;
+	tunables->io_is_busy = val;
 	return count;
 }
 
-static struct global_attr io_is_busy_attr = __ATTR(io_is_busy, 0644,
-		show_io_is_busy, store_io_is_busy);
-
-static struct attribute *interactive_attributes[] = {
-	&target_loads_attr.attr,
-	&above_hispeed_delay_attr.attr,
-	&hispeed_freq_attr.attr,
-	&go_hispeed_load_attr.attr,
-	&min_sample_time_attr.attr,
-	&timer_rate_attr.attr,
-	&timer_slack.attr,
-	&boost.attr,
-	&boostpulse.attr,
-	&boostpulse_duration.attr,
-	&io_is_busy_attr.attr,
+/*
+ * Create show/store routines
+ * - sys: One governor instance for complete SYSTEM
+ * - pol: One governor instance per struct cpufreq_policy
+ */
+#define show_gov_pol_sys(file_name)					\
+static ssize_t show_##file_name##_gov_sys				\
+(struct kobject *kobj, struct attribute *attr, char *buf)		\
+{									\
+	return show_##file_name(common_tunables, buf);			\
+}									\
+									\
+static ssize_t show_##file_name##_gov_pol				\
+(struct cpufreq_policy *policy, char *buf)				\
+{									\
+	return show_##file_name(policy->governor_data, buf);		\
+}
+
+#define store_gov_pol_sys(file_name)					\
+static ssize_t store_##file_name##_gov_sys				\
+(struct kobject *kobj, struct attribute *attr, const char *buf,		\
+	size_t count)							\
+{									\
+	return store_##file_name(common_tunables, buf, count);		\
+}									\
+									\
+static ssize_t store_##file_name##_gov_pol				\
+(struct cpufreq_policy *policy, const char *buf, size_t count)		\
+{									\
+	return store_##file_name(policy->governor_data, buf, count);	\
+}
+
+#define show_store_gov_pol_sys(file_name)				\
+show_gov_pol_sys(file_name);						\
+store_gov_pol_sys(file_name)
+
+show_store_gov_pol_sys(target_loads);
+show_store_gov_pol_sys(above_hispeed_delay);
+show_store_gov_pol_sys(hispeed_freq);
+show_store_gov_pol_sys(go_hispeed_load);
+show_store_gov_pol_sys(min_sample_time);
+show_store_gov_pol_sys(timer_rate);
+show_store_gov_pol_sys(timer_slack);
+show_store_gov_pol_sys(boost);
+store_gov_pol_sys(boostpulse);
+show_store_gov_pol_sys(boostpulse_duration);
+show_store_gov_pol_sys(io_is_busy);
+
+#define gov_sys_attr_rw(_name)						\
+static struct global_attr _name##_gov_sys =				\
+__ATTR(_name, 0644, show_##_name##_gov_sys, store_##_name##_gov_sys)
+
+#define gov_pol_attr_rw(_name)						\
+static struct freq_attr _name##_gov_pol =				\
+__ATTR(_name, 0644, show_##_name##_gov_pol, store_##_name##_gov_pol)
+
+#define gov_sys_pol_attr_rw(_name)					\
+	gov_sys_attr_rw(_name);						\
+	gov_pol_attr_rw(_name)
+
+gov_sys_pol_attr_rw(target_loads);
+gov_sys_pol_attr_rw(above_hispeed_delay);
+gov_sys_pol_attr_rw(hispeed_freq);
+gov_sys_pol_attr_rw(go_hispeed_load);
+gov_sys_pol_attr_rw(min_sample_time);
+gov_sys_pol_attr_rw(timer_rate);
+gov_sys_pol_attr_rw(timer_slack);
+gov_sys_pol_attr_rw(boost);
+gov_sys_pol_attr_rw(boostpulse_duration);
+gov_sys_pol_attr_rw(io_is_busy);
+
+static struct global_attr boostpulse_gov_sys =
+	__ATTR(boostpulse, 0200, NULL, store_boostpulse_gov_sys);
+
+static struct freq_attr boostpulse_gov_pol =
+	__ATTR(boostpulse, 0200, NULL, store_boostpulse_gov_pol);
+
+/* One Governor instance for entire system */
+static struct attribute *interactive_attributes_gov_sys[] = {
+	&target_loads_gov_sys.attr,
+	&above_hispeed_delay_gov_sys.attr,
+	&hispeed_freq_gov_sys.attr,
+	&go_hispeed_load_gov_sys.attr,
+	&min_sample_time_gov_sys.attr,
+	&timer_rate_gov_sys.attr,
+	&timer_slack_gov_sys.attr,
+	&boost_gov_sys.attr,
+	&boostpulse_gov_sys.attr,
+	&boostpulse_duration_gov_sys.attr,
+	&io_is_busy_gov_sys.attr,
 	NULL,
 };
 
-static struct attribute_group interactive_attr_group = {
-	.attrs = interactive_attributes,
+static struct attribute_group interactive_attr_group_gov_sys = {
+	.attrs = interactive_attributes_gov_sys,
 	.name = "interactive",
 };
 
+/* Per policy governor instance */
+static struct attribute *interactive_attributes_gov_pol[] = {
+	&target_loads_gov_pol.attr,
+	&above_hispeed_delay_gov_pol.attr,
+	&hispeed_freq_gov_pol.attr,
+	&go_hispeed_load_gov_pol.attr,
+	&min_sample_time_gov_pol.attr,
+	&timer_rate_gov_pol.attr,
+	&timer_slack_gov_pol.attr,
+	&boost_gov_pol.attr,
+	&boostpulse_gov_pol.attr,
+	&boostpulse_duration_gov_pol.attr,
+	&io_is_busy_gov_pol.attr,
+	NULL,
+};
+
+static struct attribute_group interactive_attr_group_gov_pol = {
+	.attrs = interactive_attributes_gov_pol,
+	.name = "interactive",
+};
+
+static struct attribute_group *get_sysfs_attr(void)
+{
+	if (have_governor_per_policy())
+		return &interactive_attr_group_gov_pol;
+	else
+		return &interactive_attr_group_gov_sys;
+}
+
 static int cpufreq_interactive_idle_notifier(struct notifier_block *nb,
 					     unsigned long val,
 					     void *data)
@@ -1057,15 +1153,88 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 	unsigned int j;
 	struct cpufreq_interactive_cpuinfo *pcpu;
 	struct cpufreq_frequency_table *freq_table;
+	struct cpufreq_interactive_tunables *tunables;
+
+	if (have_governor_per_policy())
+		tunables = policy->governor_data;
+	else
+		tunables = common_tunables;
+
+	WARN_ON(!tunables && (event != CPUFREQ_GOV_POLICY_INIT));
 
 	switch (event) {
+	case CPUFREQ_GOV_POLICY_INIT:
+		if (have_governor_per_policy()) {
+			WARN_ON(tunables);
+		} else if (tunables) {
+			tunables->usage_count++;
+			policy->governor_data = tunables;
+			return 0;
+		}
+
+		tunables = kzalloc(sizeof(*tunables), GFP_KERNEL);
+		if (!tunables) {
+			pr_err("%s: POLICY_INIT: kzalloc failed\n", __func__);
+			return -ENOMEM;
+		}
+
+		rc = sysfs_create_group(get_governor_parent_kobj(policy),
+				get_sysfs_attr());
+		if (rc) {
+			kfree(tunables);
+			return rc;
+		}
+
+		tunables->usage_count = 1;
+		tunables->above_hispeed_delay = default_above_hispeed_delay;
+		tunables->nabove_hispeed_delay =
+			ARRAY_SIZE(default_above_hispeed_delay);
+		tunables->go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;
+		tunables->target_loads = default_target_loads;
+		tunables->ntarget_loads = ARRAY_SIZE(default_target_loads);
+		tunables->min_sample_time = DEFAULT_MIN_SAMPLE_TIME;
+		tunables->timer_rate = DEFAULT_TIMER_RATE;
+		tunables->boostpulse_duration_val = DEFAULT_MIN_SAMPLE_TIME;
+		tunables->timer_slack_val = DEFAULT_TIMER_SLACK;
+
+		spin_lock_init(&tunables->target_loads_lock);
+		spin_lock_init(&tunables->above_hispeed_delay_lock);
+
+		if (!policy->governor->initialized) {
+			idle_notifier_register(&cpufreq_interactive_idle_nb);
+			cpufreq_register_notifier(&cpufreq_notifier_block,
+					CPUFREQ_TRANSITION_NOTIFIER);
+		}
+
+		policy->governor_data = tunables;
+		if (!have_governor_per_policy())
+			common_tunables = tunables;
+
+		break;
+
+	case CPUFREQ_GOV_POLICY_EXIT:
+		if (!--tunables->usage_count) {
+			if (policy->governor->initialized == 1) {
+				cpufreq_unregister_notifier(&cpufreq_notifier_block,
+						CPUFREQ_TRANSITION_NOTIFIER);
+				idle_notifier_unregister(&cpufreq_interactive_idle_nb);
+			}
+
+			sysfs_remove_group(get_governor_parent_kobj(policy),
+					get_sysfs_attr());
+			kfree(tunables);
+			common_tunables = NULL;
+		}
+
+		policy->governor_data = NULL;
+		break;
+
 	case CPUFREQ_GOV_START:
 		mutex_lock(&gov_lock);
 
-		freq_table =
-			cpufreq_frequency_get_table(policy->cpu);
-		if (!hispeed_freq)
-			hispeed_freq = policy->max;
+		freq_table = cpufreq_frequency_get_table(policy->cpu);
+		if (!tunables->hispeed_freq)
+			tunables->hispeed_freq = policy->max;
 
 		for_each_cpu(j, policy->cpus) {
 			pcpu = &per_cpu(cpuinfo, j);
@@ -1078,30 +1247,11 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->hispeed_validate_time =
 				pcpu->floor_validate_time;
 			down_write(&pcpu->enable_sem);
-			cpufreq_interactive_timer_start(j);
+			cpufreq_interactive_timer_start(tunables, j);
 			pcpu->governor_enabled = 1;
 			up_write(&pcpu->enable_sem);
 		}
 
-		/*
-		 * Do not register the idle hook and create sysfs
-		 * entries if we have already done so.
-		 */
-		if (++active_count > 1) {
-			mutex_unlock(&gov_lock);
-			return 0;
-		}
-
-		rc = sysfs_create_group(cpufreq_global_kobject,
-				&interactive_attr_group);
-		if (rc) {
-			mutex_unlock(&gov_lock);
-			return rc;
-		}
-
-		idle_notifier_register(&cpufreq_interactive_idle_nb);
-		cpufreq_register_notifier(
-			&cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER);
 		mutex_unlock(&gov_lock);
 		break;
 
@@ -1116,18 +1266,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			up_write(&pcpu->enable_sem);
 		}
 
-		if (--active_count > 0) {
-			mutex_unlock(&gov_lock);
-			return 0;
-		}
-
-		cpufreq_unregister_notifier(
-			&cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER);
-		idle_notifier_unregister(&cpufreq_interactive_idle_nb);
-		sysfs_remove_group(cpufreq_global_kobject,
-				&interactive_attr_group);
 		mutex_unlock(&gov_lock);
-
 		break;
 
 	case CPUFREQ_GOV_LIMITS:
@@ -1161,7 +1300,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			 */
 			del_timer_sync(&pcpu->cpu_timer);
 			del_timer_sync(&pcpu->cpu_slack_timer);
-			cpufreq_interactive_timer_start(j);
+			cpufreq_interactive_timer_start(tunables, j);
 			up_write(&pcpu->enable_sem);
 		}
 		break;
@@ -1201,9 +1340,7 @@ static int __init cpufreq_interactive_init(void)
 		init_rwsem(&pcpu->enable_sem);
 	}
 
-	spin_lock_init(&target_loads_lock);
 	spin_lock_init(&speedchange_cpumask_lock);
-	spin_lock_init(&above_hispeed_delay_lock);
 	mutex_init(&gov_lock);
 	speedchange_task =
 		kthread_create(cpufreq_interactive_speedchange_task, NULL,
-- 
1.7.9.5

[PATCH 56/70] cpufreq: interactive: delete timers for GOV_START

[Bálint Czobor]

From: Shridhar Rasal <srasal@nvidia.com>

Make sure that timers cpu_timer and cpu_slack_timer
deactivated before addition of new.

Change-Id: If31c4049606871df6f00efdc24b1d713c86a6f69
Signed-off-by: Shridhar Rasal <srasal@nvidia.com>
Signed-off-by: Bharat Nihalani <bnihalani@nvidia.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 66c096d..d72e8c4 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -1247,6 +1247,8 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			pcpu->hispeed_validate_time =
 				pcpu->floor_validate_time;
 			down_write(&pcpu->enable_sem);
+			del_timer_sync(&pcpu->cpu_timer);
+			del_timer_sync(&pcpu->cpu_slack_timer);
 			cpufreq_interactive_timer_start(tunables, j);
 			pcpu->governor_enabled = 1;
 			up_write(&pcpu->enable_sem);
-- 
1.7.9.5

[PATCH 57/70] cpufreq: interactive: fix compiling warnings

[Bálint Czobor]

From: Chih-Wei Huang <cwhuang@linux.org.tw>

The gcc warns like:

  cpufreq_interactive.c:745:6: warning: operation on 'ret' may be undefined [-Wsequence-point]

It was introduced by commit cf0fad49d17cb8273ce555dd5b7afab67d7923bf.

Since sprintf(...) just return 1 (one character) in this case, ret should not changed.
Just discarding the result of sprintf(...) leads to the result that
the committer of cf0fad49d17cb8273ce555dd5b7afab67d7923bf wants.

Change-Id: Ifed1cef6d6a31c3ed23dad03a567b3b9eddf3a57
Signed-off-by: Chih-Wei Huang <cwhuang@android-x86.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index d72e8c4..9f63bd1 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -752,7 +752,7 @@ static ssize_t show_target_loads(
 		ret += sprintf(buf + ret, "%u%s", tunables->target_loads[i],
 			       i & 0x1 ? ":" : " ");
 
-	ret += sprintf(buf + --ret, "\n");
+	sprintf(buf + ret - 1, "\n");
 	spin_unlock_irqrestore(&tunables->target_loads_lock, flags);
 	return ret;
 }
@@ -792,7 +792,7 @@ static ssize_t show_above_hispeed_delay(
 			       tunables->above_hispeed_delay[i],
 			       i & 0x1 ? ":" : " ");
 
-	ret += sprintf(buf + --ret, "\n");
+	sprintf(buf + ret - 1, "\n");
 	spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags);
 	return ret;
 }
-- 
1.7.9.5

[PATCH 58/70] cpufreq: interactive: fix NULL pointer dereference at sysfs ops

[Bálint Czobor]

From: Minsung Kim <ms925.kim@samsung.com>

sysfs ops for target_loads and above_hispeed_delay can be called before
initializing tunables at CPUFREQ_GOV_POLICY_INIT. Create sysfs entries after
initialization.

Change-Id: I50356198d7629731c0d32a3066d61fe8354e0001
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 9f63bd1..8297ac6 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -1178,13 +1178,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 			return -ENOMEM;
 		}
 
-		rc = sysfs_create_group(get_governor_parent_kobj(policy),
-				get_sysfs_attr());
-		if (rc) {
-			kfree(tunables);
-			return rc;
-		}
-
 		tunables->usage_count = 1;
 		tunables->above_hispeed_delay = default_above_hispeed_delay;
 		tunables->nabove_hispeed_delay =
@@ -1200,16 +1193,26 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		spin_lock_init(&tunables->target_loads_lock);
 		spin_lock_init(&tunables->above_hispeed_delay_lock);
 
+		policy->governor_data = tunables;
+		if (!have_governor_per_policy())
+			common_tunables = tunables;
+
+		rc = sysfs_create_group(get_governor_parent_kobj(policy),
+				get_sysfs_attr());
+		if (rc) {
+			kfree(tunables);
+			policy->governor_data = NULL;
+			if (!have_governor_per_policy())
+				common_tunables = NULL;
+			return rc;
+		}
+
 		if (!policy->governor->initialized) {
 			idle_notifier_register(&cpufreq_interactive_idle_nb);
 			cpufreq_register_notifier(&cpufreq_notifier_block,
 					CPUFREQ_TRANSITION_NOTIFIER);
 		}
 
-		policy->governor_data = tunables;
-		if (!have_governor_per_policy())
-			common_tunables = tunables;
-
 		break;
 
 	case CPUFREQ_GOV_POLICY_EXIT:
-- 
1.7.9.5

[PATCH 59/70] cpufreq: interactive: Use generic get_cpu_idle_time() from cpufreq.c

[Bálint Czobor]

From: Viresh Kumar <viresh.kumar@linaro.org>

Now that a generic version of get_cpu_idle_time() is available, use that for the interactive governor.

[toddpoynor@google.com: commit text changes]
Change-Id: Ia38b57085aac99ec3d415fe44471d5dfde519c2c
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Jon Medhurst <tixy@linaro.org>
Signed-off-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   40 ---------------------------------
 1 file changed, 40 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 8297ac6..ff77b30 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -30,8 +30,6 @@
 #include <linux/workqueue.h>
 #include <linux/kthread.h>
 #include <linux/slab.h>
-#include <linux/kernel_stat.h>
-#include <asm/cputime.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/cpufreq_interactive.h>
@@ -119,44 +117,6 @@ struct cpufreq_interactive_tunables *common_tunables;
 
 static struct attribute_group *get_sysfs_attr(void);
 
-static inline cputime64_t get_cpu_idle_time_jiffy(unsigned int cpu,
-						  cputime64_t *wall)
-{
-	u64 idle_time;
-	u64 cur_wall_time;
-	u64 busy_time;
-
-	cur_wall_time = jiffies64_to_cputime64(get_jiffies_64());
-
-	busy_time  = kcpustat_cpu(cpu).cpustat[CPUTIME_USER];
-	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SYSTEM];
-	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_IRQ];
-	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SOFTIRQ];
-	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL];
-	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_NICE];
-
-	idle_time = cur_wall_time - busy_time;
-	if (wall)
-		*wall = jiffies_to_usecs(cur_wall_time);
-
-	return jiffies_to_usecs(idle_time);
-}
-
-static inline cputime64_t get_cpu_idle_time(
-	unsigned int cpu,
-	cputime64_t *wall,
-	bool io_is_busy)
-{
-	u64 idle_time = get_cpu_idle_time_us(cpu, wall);
-
-	if (idle_time == -1ULL)
-		idle_time = get_cpu_idle_time_jiffy(cpu, wall);
-	else if (!io_is_busy)
-		idle_time += get_cpu_iowait_time_us(cpu, wall);
-
-	return idle_time;
-}
-
 static void cpufreq_interactive_timer_resched(
 	struct cpufreq_interactive_cpuinfo *pcpu)
 {
-- 
1.7.9.5

[PATCH 60/70] cpufreq: interactive: hold reference on global cpufreq kobject if needed

[Bálint Czobor]

From: Greg Hackmann <ghackmann@google.com>

2361be23666232dbb4851a527f466c4cbf5340fc changed cpufreq to add the
global cpufreq kobject to sysfs on demand.

To ensure this happens, cpufreq_interactive must hold a reference on
this object on devices where it intends to use it (i.e., devices where
have_governor_per_policy() returns false).  Otherwise a parentless
kobject will be passed to sysfs_create_group() which will subsequently
BUG().

Change-Id: I7dd03956e1d3c6c3c0cc17c799882c235804ae09
Signed-off-by: Greg Hackmann <ghackmann@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index ff77b30..786fd01 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -1154,8 +1154,10 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		spin_lock_init(&tunables->above_hispeed_delay_lock);
 
 		policy->governor_data = tunables;
-		if (!have_governor_per_policy())
+		if (!have_governor_per_policy()) {
 			common_tunables = tunables;
+			WARN_ON(cpufreq_get_global_kobject());
+		}
 
 		rc = sysfs_create_group(get_governor_parent_kobj(policy),
 				get_sysfs_attr());
@@ -1185,6 +1187,10 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 
 			sysfs_remove_group(get_governor_parent_kobj(policy),
 					get_sysfs_attr());
+
+			if (!have_governor_per_policy())
+				cpufreq_put_global_kobject();
+
 			kfree(tunables);
 			common_tunables = NULL;
 		}
-- 
1.7.9.5

[PATCH 61/70] cpufreq: interactive: restructure CPUFREQ_GOV_LIMITS

[Bálint Czobor]

From: Badhri Jagan Sridharan <Badhri@google.com>

The cpufreq_interactive_timer gets cancelled and rescheduled
whenever the cpufreq_policy is changed. When the cpufreq policy is
changed at a rate faster than the sampling_rate of the interactive
governor, then the governor misses to change the target frequency
for long duration. The patch removes the need of cancelling the
timers when policy->min is changed.

Signed-off-by: Badhri Jagan Sridharan <Badhri@google.com>
Change-Id: Ibd98d151e1c73b8bd969484583ff98ee9f1135ef
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   48 ++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 13 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 786fd01..0e3e45e 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -44,8 +44,10 @@ struct cpufreq_interactive_cpuinfo {
 	u64 cputime_speedadj_timestamp;
 	struct cpufreq_policy *policy;
 	struct cpufreq_frequency_table *freq_table;
+	spinlock_t target_freq_lock; /*protects target freq */
 	unsigned int target_freq;
 	unsigned int floor_freq;
+	unsigned int max_freq;
 	u64 floor_validate_time;
 	u64 hispeed_validate_time;
 	struct rw_semaphore enable_sem;
@@ -358,6 +360,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	if (WARN_ON_ONCE(!delta_time))
 		goto rearm;
 
+	spin_lock_irqsave(&pcpu->target_freq_lock, flags);
 	do_div(cputime_speedadj, delta_time);
 	loadadjfreq = (unsigned int)cputime_speedadj * 100;
 	cpu_load = loadadjfreq / pcpu->target_freq;
@@ -383,6 +386,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 		trace_cpufreq_interactive_notyet(
 			data, cpu_load, pcpu->target_freq,
 			pcpu->policy->cur, new_freq);
+		spin_unlock_irqrestore(&pcpu->target_freq_lock, flags);
 		goto rearm;
 	}
 
@@ -390,8 +394,10 @@ static void cpufreq_interactive_timer(unsigned long data)
 
 	if (cpufreq_frequency_table_target(pcpu->policy, pcpu->freq_table,
 					   new_freq, CPUFREQ_RELATION_L,
-					   &index))
+					   &index)) {
+		spin_unlock_irqrestore(&pcpu->target_freq_lock, flags);
 		goto rearm;
+	}
 
 	new_freq = pcpu->freq_table[index].frequency;
 
@@ -405,6 +411,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 			trace_cpufreq_interactive_notyet(
 				data, cpu_load, pcpu->target_freq,
 				pcpu->policy->cur, new_freq);
+			spin_unlock_irqrestore(&pcpu->target_freq_lock, flags);
 			goto rearm;
 		}
 	}
@@ -426,6 +433,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 		trace_cpufreq_interactive_already(
 			data, cpu_load, pcpu->target_freq,
 			pcpu->policy->cur, new_freq);
+		spin_unlock_irqrestore(&pcpu->target_freq_lock, flags);
 		goto rearm_if_notmax;
 	}
 
@@ -433,6 +441,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 					 pcpu->policy->cur, new_freq);
 
 	pcpu->target_freq = new_freq;
+	spin_unlock_irqrestore(&pcpu->target_freq_lock, flags);
 	spin_lock_irqsave(&speedchange_cpumask_lock, flags);
 	cpumask_set_cpu(data, &speedchange_cpumask);
 	spin_unlock_irqrestore(&speedchange_cpumask_lock, flags);
@@ -576,16 +585,17 @@ static void cpufreq_interactive_boost(void)
 {
 	int i;
 	int anyboost = 0;
-	unsigned long flags;
+	unsigned long flags[2];
 	struct cpufreq_interactive_cpuinfo *pcpu;
 	struct cpufreq_interactive_tunables *tunables;
 
-	spin_lock_irqsave(&speedchange_cpumask_lock, flags);
+	spin_lock_irqsave(&speedchange_cpumask_lock, flags[0]);
 
 	for_each_online_cpu(i) {
 		pcpu = &per_cpu(cpuinfo, i);
 		tunables = pcpu->policy->governor_data;
 
+		spin_lock_irqsave(&pcpu->target_freq_lock, flags[1]);
 		if (pcpu->target_freq < tunables->hispeed_freq) {
 			pcpu->target_freq = tunables->hispeed_freq;
 			cpumask_set_cpu(i, &speedchange_cpumask);
@@ -601,9 +611,10 @@ static void cpufreq_interactive_boost(void)
 
 		pcpu->floor_freq = tunables->hispeed_freq;
 		pcpu->floor_validate_time = ktime_to_us(ktime_get());
+		spin_unlock_irqrestore(&pcpu->target_freq_lock, flags[1]);
 	}
 
-	spin_unlock_irqrestore(&speedchange_cpumask_lock, flags);
+	spin_unlock_irqrestore(&speedchange_cpumask_lock, flags[0]);
 
 	if (anyboost)
 		wake_up_process(speedchange_task);
@@ -1114,6 +1125,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 	struct cpufreq_interactive_cpuinfo *pcpu;
 	struct cpufreq_frequency_table *freq_table;
 	struct cpufreq_interactive_tunables *tunables;
+	unsigned long flags;
 
 	if (have_governor_per_policy())
 		tunables = policy->governor_data;
@@ -1215,6 +1227,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 				ktime_to_us(ktime_get());
 			pcpu->hispeed_validate_time =
 				pcpu->floor_validate_time;
+			pcpu->max_freq = policy->max;
 			down_write(&pcpu->enable_sem);
 			del_timer_sync(&pcpu->cpu_timer);
 			del_timer_sync(&pcpu->cpu_slack_timer);
@@ -1250,29 +1263,37 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		for_each_cpu(j, policy->cpus) {
 			pcpu = &per_cpu(cpuinfo, j);
 
-			/* hold write semaphore to avoid race */
-			down_write(&pcpu->enable_sem);
+			down_read(&pcpu->enable_sem);
 			if (pcpu->governor_enabled == 0) {
-				up_write(&pcpu->enable_sem);
+				up_read(&pcpu->enable_sem);
 				continue;
 			}
 
-			/* update target_freq firstly */
+			spin_lock_irqsave(&pcpu->target_freq_lock, flags);
 			if (policy->max < pcpu->target_freq)
 				pcpu->target_freq = policy->max;
 			else if (policy->min > pcpu->target_freq)
 				pcpu->target_freq = policy->min;
 
-			/* Reschedule timer.
+			spin_unlock_irqrestore(&pcpu->target_freq_lock, flags);
+			up_read(&pcpu->enable_sem);
+
+			/* Reschedule timer only if policy->max is raised.
 			 * Delete the timers, else the timer callback may
 			 * return without re-arm the timer when failed
 			 * acquire the semaphore. This race may cause timer
 			 * stopped unexpectedly.
 			 */
-			del_timer_sync(&pcpu->cpu_timer);
-			del_timer_sync(&pcpu->cpu_slack_timer);
-			cpufreq_interactive_timer_start(tunables, j);
-			up_write(&pcpu->enable_sem);
+
+			if (policy->max > pcpu->max_freq) {
+				down_write(&pcpu->enable_sem);
+				del_timer_sync(&pcpu->cpu_timer);
+				del_timer_sync(&pcpu->cpu_slack_timer);
+				cpufreq_interactive_timer_start(tunables, j);
+				up_write(&pcpu->enable_sem);
+			}
+
+			pcpu->max_freq = policy->max;
 		}
 		break;
 	}
@@ -1308,6 +1329,7 @@ static int __init cpufreq_interactive_init(void)
 		init_timer(&pcpu->cpu_slack_timer);
 		pcpu->cpu_slack_timer.function = cpufreq_interactive_nop_timer;
 		spin_lock_init(&pcpu->load_lock);
+		spin_lock_init(&pcpu->target_freq_lock);
 		init_rwsem(&pcpu->enable_sem);
 	}
 
-- 
1.7.9.5

[PATCH 62/70] cpufreq: interactive: turn boost_pulse off on boost off

[Bálint Czobor]

From: Ruchi Kandoi <kandoiruchi@google.com>

Change-Id: I36fe217fa047d68ea90e78b12c7db4537ea8010b
Signed-off-by: Ruchi Kandoi <kandoiruchi@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 0e3e45e..3daca21 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -908,6 +908,7 @@ static ssize_t store_boost(struct cpufreq_interactive_tunables *tunables,
 		trace_cpufreq_interactive_boost("on");
 		cpufreq_interactive_boost();
 	} else {
+		boostpulse_endtime = ktime_to_us(ktime_get());
 		trace_cpufreq_interactive_unboost("off");
 	}
 
-- 
1.7.9.5

[PATCH 63/70] cpufreq: interactive: remove compilation error from commit 49cc72365fb7ee87762a7ccc6a32ef68627216c5

[Bálint Czobor]

From: Ruchi Kandoi <kandoiruchi@google.com>

Change-Id: I068b18281d03ac879ef64d8ff36ed43367293767
Signed-off-by: Ruchi Kandoi <kandoiruchi@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 3daca21..6c4b6a8 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -908,7 +908,7 @@ static ssize_t store_boost(struct cpufreq_interactive_tunables *tunables,
 		trace_cpufreq_interactive_boost("on");
 		cpufreq_interactive_boost();
 	} else {
-		boostpulse_endtime = ktime_to_us(ktime_get());
+		tunables->boostpulse_endtime = ktime_to_us(ktime_get());
 		trace_cpufreq_interactive_unboost("off");
 	}
 
-- 
1.7.9.5

[PATCH 64/70] cpufreq: interactive: prevents the frequency to directly raise above the hispeed_freq from a lower frequency.

[Bálint Czobor]

From: Ruchi Kandoi <kandoiruchi@google.com>

When the load was below go_hispeed_load, there is a possibility that
choose_freq() would return a frequency which would be higher than the
hispeed_freq. According to the policy we should first jump to the
hispeed_freq, stay there for above_hispeed_delay and then be allowed to
raise higher than that.

Added a check to prevent the frequency to be directly raised to
something higher than the hispeed_freq.

Change-Id: Icda5d848dd9beadcc18835082ddf269732c75bd0
Signed-off-by: Ruchi Kandoi <kandoiruchi@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 6c4b6a8..f0e6fe2 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -377,6 +377,9 @@ static void cpufreq_interactive_timer(unsigned long data)
 		}
 	} else {
 		new_freq = choose_freq(pcpu, loadadjfreq);
+		if (new_freq > tunables->hispeed_freq &&
+				pcpu->target_freq < tunables->hispeed_freq)
+			new_freq = tunables->hispeed_freq;
 	}
 
 	if (pcpu->target_freq >= tunables->hispeed_freq &&
-- 
1.7.9.5

[PATCH 65/70] cpufreq: interactive: make common_tunables static

[Bálint Czobor]

From: Todd Poynor <toddpoynor@google.com>

From: Cylen Yao <cylen.yao@mediatek.com>

common_tunables should be static.

Change-Id: I502ee3062bece5082fea7861eff2f6237e25cede
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index f0e6fe2..a0a7c3a 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -115,7 +115,7 @@ struct cpufreq_interactive_tunables {
 };
 
 /* For cases where we have single governor instance for system */
-struct cpufreq_interactive_tunables *common_tunables;
+static struct cpufreq_interactive_tunables *common_tunables;
 
 static struct attribute_group *get_sysfs_attr(void);
 
-- 
1.7.9.5

[PATCH 66/70] cpufreq: interactive: Fix compile errors in accordance with changes from 3.14 to 3.18

[Bálint Czobor]

Signed-off-by: Ruchi Kandoi<kandoiruchi@google.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index a0a7c3a..9faa189 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -805,7 +805,7 @@ static ssize_t store_hispeed_freq(struct cpufreq_interactive_tunables *tunables,
 	int ret;
 	long unsigned int val;
 
-	ret = strict_strtoul(buf, 0, &val);
+	ret = kstrtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
 	tunables->hispeed_freq = val;
@@ -824,7 +824,7 @@ static ssize_t store_go_hispeed_load(struct cpufreq_interactive_tunables
 	int ret;
 	unsigned long val;
 
-	ret = strict_strtoul(buf, 0, &val);
+	ret = kstrtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
 	tunables->go_hispeed_load = val;
@@ -843,7 +843,7 @@ static ssize_t store_min_sample_time(struct cpufreq_interactive_tunables
 	int ret;
 	unsigned long val;
 
-	ret = strict_strtoul(buf, 0, &val);
+	ret = kstrtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
 	tunables->min_sample_time = val;
@@ -862,7 +862,7 @@ static ssize_t store_timer_rate(struct cpufreq_interactive_tunables *tunables,
 	int ret;
 	unsigned long val;
 
-	ret = strict_strtoul(buf, 0, &val);
+	ret = kstrtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
 	tunables->timer_rate = val;
-- 
1.7.9.5

[PATCH 67/70] cpufreq: interactive: Put global cpufreq kobject on failure

[Bálint Czobor]

From: Junjie Wu <junjiew@codeaurora.org>

Fix failure recovery path in cpufreq_governor_interactive(). Call
cpufreq_put_global_kobject() to release cpufreq global kobject upon
governor init failure.

Change-Id: I7a977070b7a3c75c90acccd2c117064ed1a10d0e
Signed-off-by: Junjie Wu <junjiew@codeaurora.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 9faa189..2055266 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -1180,8 +1180,10 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
 		if (rc) {
 			kfree(tunables);
 			policy->governor_data = NULL;
-			if (!have_governor_per_policy())
+			if (!have_governor_per_policy()) {
 				common_tunables = NULL;
+				cpufreq_put_global_kobject();
+			}
 			return rc;
 		}
 
-- 
1.7.9.5

[PATCH 68/70] subsystem: CPU FREQUENCY DRIVERS- Set cpu_load calculation on current frequency

[Bálint Czobor]

In timer, cpu_load is calcuated on target_freq.
cpu_load = loadadjfreq / pcpu->target_freq;
But cpu is actually running on current freq i.e. pcpu->policy->cur. So cpu_load
should be calculated on current frequency.
cpu_load = loadadjfreq / pcpu->policy->cur;

Change-Id: I89db6b68e9f82aa52077f6bf7d819dab74265790
Signed-off-by: rahul.khandelwal <rahul.khandelwal@spreadtrum.com>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 2055266..064ab56 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -363,7 +363,7 @@ static void cpufreq_interactive_timer(unsigned long data)
 	spin_lock_irqsave(&pcpu->target_freq_lock, flags);
 	do_div(cputime_speedadj, delta_time);
 	loadadjfreq = (unsigned int)cputime_speedadj * 100;
-	cpu_load = loadadjfreq / pcpu->target_freq;
+	cpu_load = loadadjfreq / pcpu->policy->cur;
 	boosted = tunables->boost_val || now < tunables->boostpulse_endtime;
 
 	if (cpu_load >= tunables->go_hispeed_load || boosted) {
-- 
1.7.9.5

[PATCH 69/70] cpufreq: interactive: Don't set floor_validate_time during boost

[Bálint Czobor]

From: Junjie Wu <junjiew@codeaurora.org>

Frequency selection algorithm guarantees its chosen frequency
is not lower than hispeed_freq as long as boost is enabled.

Setting floor_freq and floor_validate_time during boost could block
CPU frequency from going below hispeed_freq even after
boostpulse_duration expires, if min_sample_time is higher than
boostpulse_duration. This conflicts with the intention of commit
de091367ead15b6e95dd1d0743a18f0da5a07ee5
(cpufreq: interactive: specify duration of CPU speed boost pulse)
to allow CPU to ramp down immediately after boost expires. It also
makes boost behavior inconsistent since it depends on min_sample_time.

Avoid setting floor_freq and floor_validate_time when boost starts.

Change-Id: I12852998af46cfbfaf8661eb5e8d5301b6f631e7
Signed-off-by: Junjie Wu <junjiew@codeaurora.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |    8 --------
 1 file changed, 8 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 064ab56..95525b3 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -606,14 +606,6 @@ static void cpufreq_interactive_boost(void)
 				ktime_to_us(ktime_get());
 			anyboost = 1;
 		}
-
-		/*
-		 * Set floor freq and (re)start timer for when last
-		 * validated.
-		 */
-
-		pcpu->floor_freq = tunables->hispeed_freq;
-		pcpu->floor_validate_time = ktime_to_us(ktime_get());
 		spin_unlock_irqrestore(&pcpu->target_freq_lock, flags[1]);
 	}
 
-- 
1.7.9.5

[PATCH 70/70] cpufreq: interactive: Round up timer_rate to match jiffy

[Bálint Czobor]

From: Junjie Wu <junjiew@codeaurora.org>

Timers are scheduled in unit of jiffies. Round up timer_rate so that
it matches the actual sampling period.

Change-Id: I88386a5a448e40333f9a9b9f0cf72af58cb54656
Signed-off-by: Junjie Wu <junjiew@codeaurora.org>
Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
---
 drivers/cpufreq/cpufreq_interactive.c |   10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c
index 95525b3..a2fc98a 100644
--- a/drivers/cpufreq/cpufreq_interactive.c
+++ b/drivers/cpufreq/cpufreq_interactive.c
@@ -852,12 +852,18 @@ static ssize_t store_timer_rate(struct cpufreq_interactive_tunables *tunables,
 		const char *buf, size_t count)
 {
 	int ret;
-	unsigned long val;
+	unsigned long val, val_round;
 
 	ret = kstrtoul(buf, 0, &val);
 	if (ret < 0)
 		return ret;
-	tunables->timer_rate = val;
+
+	val_round = jiffies_to_usecs(usecs_to_jiffies(val));
+	if (val != val_round)
+		pr_warn("timer_rate not aligned to jiffy. Rounded up to %lu\n",
+			val_round);
+
+	tunables->timer_rate = val_round;
 	return count;
 }
 
-- 
1.7.9.5

Re: [PATCH 01/70] cpufreq: interactive: New 'interactive' governor

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 9606 bytes --]

Hi Mike,

[auto build test ERROR on pm/linux-next -- if it's inappropriate base, please suggest rules for selecting the more suitable base]

url:    https://github.com/0day-ci/linux/commits/B-lint-Czobor/cpufreq-interactive-New-interactive-governor/20151028-020715
config: sparc64-allyesconfig (attached as .config)
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc64 

All errors (new ones prefixed by >>):

   drivers/cpufreq/cpufreq_interactive.c: In function 'store_hispeed_freq':
   drivers/cpufreq/cpufreq_interactive.c:446:2: error: implicit declaration of function 'strict_strtoull' [-Werror=implicit-function-declaration]
     ret = strict_strtoull(buf, 0, &val);
     ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'store_go_hispeed_load':
   drivers/cpufreq/cpufreq_interactive.c:469:2: error: implicit declaration of function 'strict_strtoul' [-Werror=implicit-function-declaration]
     ret = strict_strtoul(buf, 0, &val);
     ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'cpufreq_interactive_idle_notifier':
>> drivers/cpufreq/cpufreq_interactive.c:623:7: error: 'IDLE_START' undeclared (first use in this function)
     case IDLE_START:
          ^
   drivers/cpufreq/cpufreq_interactive.c:623:7: note: each undeclared identifier is reported only once for each function it appears in
>> drivers/cpufreq/cpufreq_interactive.c:626:7: error: 'IDLE_END' undeclared (first use in this function)
     case IDLE_END:
          ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'cpufreq_interactive_init':
>> drivers/cpufreq/cpufreq_interactive.c:678:2: error: implicit declaration of function 'idle_notifier_register' [-Werror=implicit-function-declaration]
     idle_notifier_register(&cpufreq_interactive_idle_nb);
     ^
   cc1: some warnings being treated as errors

vim +/IDLE_START +623 drivers/cpufreq/cpufreq_interactive.c

   440					  struct attribute *attr, const char *buf,
   441					  size_t count)
   442	{
   443		int ret;
   444		u64 val;
   445	
 > 446		ret = strict_strtoull(buf, 0, &val);
   447		if (ret < 0)
   448			return ret;
   449		hispeed_freq = val;
   450		return count;
   451	}
   452	
   453	static struct global_attr hispeed_freq_attr = __ATTR(hispeed_freq, 0644,
   454			show_hispeed_freq, store_hispeed_freq);
   455	
   456	
   457	static ssize_t show_go_hispeed_load(struct kobject *kobj,
   458					     struct attribute *attr, char *buf)
   459	{
   460		return sprintf(buf, "%lu\n", go_hispeed_load);
   461	}
   462	
   463	static ssize_t store_go_hispeed_load(struct kobject *kobj,
   464				struct attribute *attr, const char *buf, size_t count)
   465	{
   466		int ret;
   467		unsigned long val;
   468	
 > 469		ret = strict_strtoul(buf, 0, &val);
   470		if (ret < 0)
   471			return ret;
   472		go_hispeed_load = val;
   473		return count;
   474	}
   475	
   476	static struct global_attr go_hispeed_load_attr = __ATTR(go_hispeed_load, 0644,
   477			show_go_hispeed_load, store_go_hispeed_load);
   478	
   479	static ssize_t show_min_sample_time(struct kobject *kobj,
   480					struct attribute *attr, char *buf)
   481	{
   482		return sprintf(buf, "%lu\n", min_sample_time);
   483	}
   484	
   485	static ssize_t store_min_sample_time(struct kobject *kobj,
   486				struct attribute *attr, const char *buf, size_t count)
   487	{
   488		int ret;
   489		unsigned long val;
   490	
   491		ret = strict_strtoul(buf, 0, &val);
   492		if (ret < 0)
   493			return ret;
   494		min_sample_time = val;
   495		return count;
   496	}
   497	
   498	static struct global_attr min_sample_time_attr = __ATTR(min_sample_time, 0644,
   499			show_min_sample_time, store_min_sample_time);
   500	
   501	static ssize_t show_timer_rate(struct kobject *kobj,
   502				struct attribute *attr, char *buf)
   503	{
   504		return sprintf(buf, "%lu\n", timer_rate);
   505	}
   506	
   507	static ssize_t store_timer_rate(struct kobject *kobj,
   508				struct attribute *attr, const char *buf, size_t count)
   509	{
   510		int ret;
   511		unsigned long val;
   512	
   513		ret = strict_strtoul(buf, 0, &val);
   514		if (ret < 0)
   515			return ret;
   516		timer_rate = val;
   517		return count;
   518	}
   519	
   520	static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644,
   521			show_timer_rate, store_timer_rate);
   522	
   523	static struct attribute *interactive_attributes[] = {
   524		&hispeed_freq_attr.attr,
   525		&go_hispeed_load_attr.attr,
   526		&min_sample_time_attr.attr,
   527		&timer_rate_attr.attr,
   528		NULL,
   529	};
   530	
   531	static struct attribute_group interactive_attr_group = {
   532		.attrs = interactive_attributes,
   533		.name = "interactive",
   534	};
   535	
   536	static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
   537			unsigned int event)
   538	{
   539		int rc;
   540		unsigned int j;
   541		struct cpufreq_interactive_cpuinfo *pcpu;
   542		struct cpufreq_frequency_table *freq_table;
   543	
   544		switch (event) {
   545		case CPUFREQ_GOV_START:
   546			if (!cpu_online(policy->cpu))
   547				return -EINVAL;
   548	
   549			freq_table =
   550				cpufreq_frequency_get_table(policy->cpu);
   551	
   552			for_each_cpu(j, policy->cpus) {
   553				pcpu = &per_cpu(cpuinfo, j);
   554				pcpu->policy = policy;
   555				pcpu->target_freq = policy->cur;
   556				pcpu->freq_table = freq_table;
   557				pcpu->freq_change_time_in_idle =
   558					get_cpu_idle_time_us(j,
   559						     &pcpu->freq_change_time);
   560				pcpu->governor_enabled = 1;
   561				smp_wmb();
   562			}
   563	
   564			if (!hispeed_freq)
   565				hispeed_freq = policy->max;
   566	
   567			/*
   568			 * Do not register the idle hook and create sysfs
   569			 * entries if we have already done so.
   570			 */
   571			if (atomic_inc_return(&active_count) > 1)
   572				return 0;
   573	
   574			rc = sysfs_create_group(cpufreq_global_kobject,
   575					&interactive_attr_group);
   576			if (rc)
   577				return rc;
   578	
   579			break;
   580	
   581		case CPUFREQ_GOV_STOP:
   582			for_each_cpu(j, policy->cpus) {
   583				pcpu = &per_cpu(cpuinfo, j);
   584				pcpu->governor_enabled = 0;
   585				smp_wmb();
   586				del_timer_sync(&pcpu->cpu_timer);
   587	
   588				/*
   589				 * Reset idle exit time since we may cancel the timer
   590				 * before it can run after the last idle exit time,
   591				 * to avoid tripping the check in idle exit for a timer
   592				 * that is trying to run.
   593				 */
   594				pcpu->idle_exit_time = 0;
   595			}
   596	
   597			flush_work(&freq_scale_down_work);
   598			if (atomic_dec_return(&active_count) > 0)
   599				return 0;
   600	
   601			sysfs_remove_group(cpufreq_global_kobject,
   602					&interactive_attr_group);
   603	
   604			break;
   605	
   606		case CPUFREQ_GOV_LIMITS:
   607			if (policy->max < policy->cur)
   608				__cpufreq_driver_target(policy,
   609						policy->max, CPUFREQ_RELATION_H);
   610			else if (policy->min > policy->cur)
   611				__cpufreq_driver_target(policy,
   612						policy->min, CPUFREQ_RELATION_L);
   613			break;
   614		}
   615		return 0;
   616	}
   617	
   618	static int cpufreq_interactive_idle_notifier(struct notifier_block *nb,
   619						     unsigned long val,
   620						     void *data)
   621	{
   622		switch (val) {
 > 623		case IDLE_START:
   624			cpufreq_interactive_idle_start();
   625			break;
 > 626		case IDLE_END:
   627			cpufreq_interactive_idle_end();
   628			break;
   629		}
   630	
   631		return 0;
   632	}
   633	
   634	static struct notifier_block cpufreq_interactive_idle_nb = {
   635		.notifier_call = cpufreq_interactive_idle_notifier,
   636	};
   637	
   638	static int __init cpufreq_interactive_init(void)
   639	{
   640		unsigned int i;
   641		struct cpufreq_interactive_cpuinfo *pcpu;
   642		struct sched_param param = { .sched_priority = MAX_RT_PRIO-1 };
   643	
   644		go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;
   645		min_sample_time = DEFAULT_MIN_SAMPLE_TIME;
   646		timer_rate = DEFAULT_TIMER_RATE;
   647	
   648		/* Initalize per-cpu timers */
   649		for_each_possible_cpu(i) {
   650			pcpu = &per_cpu(cpuinfo, i);
   651			init_timer(&pcpu->cpu_timer);
   652			pcpu->cpu_timer.function = cpufreq_interactive_timer;
   653			pcpu->cpu_timer.data = i;
   654		}
   655	
   656		up_task = kthread_create(cpufreq_interactive_up_task, NULL,
   657					 "kinteractiveup");
   658		if (IS_ERR(up_task))
   659			return PTR_ERR(up_task);
   660	
   661		sched_setscheduler_nocheck(up_task, SCHED_FIFO, &param);
   662		get_task_struct(up_task);
   663	
   664		/* No rescuer thread, bind to CPU queuing the work for possibly
   665		   warm cache (probably doesn't matter much). */
   666		down_wq = alloc_workqueue("knteractive_down", 0, 1);
   667	
   668		if (!down_wq)
   669			goto err_freeuptask;
   670	
   671		INIT_WORK(&freq_scale_down_work,
   672			  cpufreq_interactive_freq_down);
   673	
   674		spin_lock_init(&up_cpumask_lock);
   675		spin_lock_init(&down_cpumask_lock);
   676		mutex_init(&set_speed_lock);
   677	
 > 678		idle_notifier_register(&cpufreq_interactive_idle_nb);
   679	
   680		return cpufreq_register_governor(&cpufreq_gov_interactive);
   681	

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 43764 bytes --]

Re: [PATCH 01/70] cpufreq: interactive: New 'interactive' governor

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 9606 bytes --]

Hi Mike,

[auto build test ERROR on pm/linux-next -- if it's inappropriate base, please suggest rules for selecting the more suitable base]

url:    https://github.com/0day-ci/linux/commits/B-lint-Czobor/cpufreq-interactive-New-interactive-governor/20151028-020715
config: sparc64-allyesconfig (attached as .config)
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc64 

All errors (new ones prefixed by >>):

   drivers/cpufreq/cpufreq_interactive.c: In function 'store_hispeed_freq':
   drivers/cpufreq/cpufreq_interactive.c:446:2: error: implicit declaration of function 'strict_strtoull' [-Werror=implicit-function-declaration]
     ret = strict_strtoull(buf, 0, &val);
     ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'store_go_hispeed_load':
   drivers/cpufreq/cpufreq_interactive.c:469:2: error: implicit declaration of function 'strict_strtoul' [-Werror=implicit-function-declaration]
     ret = strict_strtoul(buf, 0, &val);
     ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'cpufreq_interactive_idle_notifier':
>> drivers/cpufreq/cpufreq_interactive.c:623:7: error: 'IDLE_START' undeclared (first use in this function)
     case IDLE_START:
          ^
   drivers/cpufreq/cpufreq_interactive.c:623:7: note: each undeclared identifier is reported only once for each function it appears in
>> drivers/cpufreq/cpufreq_interactive.c:626:7: error: 'IDLE_END' undeclared (first use in this function)
     case IDLE_END:
          ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'cpufreq_interactive_init':
>> drivers/cpufreq/cpufreq_interactive.c:678:2: error: implicit declaration of function 'idle_notifier_register' [-Werror=implicit-function-declaration]
     idle_notifier_register(&cpufreq_interactive_idle_nb);
     ^
   cc1: some warnings being treated as errors

vim +/IDLE_START +623 drivers/cpufreq/cpufreq_interactive.c

   440					  struct attribute *attr, const char *buf,
   441					  size_t count)
   442	{
   443		int ret;
   444		u64 val;
   445	
 > 446		ret = strict_strtoull(buf, 0, &val);
   447		if (ret < 0)
   448			return ret;
   449		hispeed_freq = val;
   450		return count;
   451	}
   452	
   453	static struct global_attr hispeed_freq_attr = __ATTR(hispeed_freq, 0644,
   454			show_hispeed_freq, store_hispeed_freq);
   455	
   456	
   457	static ssize_t show_go_hispeed_load(struct kobject *kobj,
   458					     struct attribute *attr, char *buf)
   459	{
   460		return sprintf(buf, "%lu\n", go_hispeed_load);
   461	}
   462	
   463	static ssize_t store_go_hispeed_load(struct kobject *kobj,
   464				struct attribute *attr, const char *buf, size_t count)
   465	{
   466		int ret;
   467		unsigned long val;
   468	
 > 469		ret = strict_strtoul(buf, 0, &val);
   470		if (ret < 0)
   471			return ret;
   472		go_hispeed_load = val;
   473		return count;
   474	}
   475	
   476	static struct global_attr go_hispeed_load_attr = __ATTR(go_hispeed_load, 0644,
   477			show_go_hispeed_load, store_go_hispeed_load);
   478	
   479	static ssize_t show_min_sample_time(struct kobject *kobj,
   480					struct attribute *attr, char *buf)
   481	{
   482		return sprintf(buf, "%lu\n", min_sample_time);
   483	}
   484	
   485	static ssize_t store_min_sample_time(struct kobject *kobj,
   486				struct attribute *attr, const char *buf, size_t count)
   487	{
   488		int ret;
   489		unsigned long val;
   490	
   491		ret = strict_strtoul(buf, 0, &val);
   492		if (ret < 0)
   493			return ret;
   494		min_sample_time = val;
   495		return count;
   496	}
   497	
   498	static struct global_attr min_sample_time_attr = __ATTR(min_sample_time, 0644,
   499			show_min_sample_time, store_min_sample_time);
   500	
   501	static ssize_t show_timer_rate(struct kobject *kobj,
   502				struct attribute *attr, char *buf)
   503	{
   504		return sprintf(buf, "%lu\n", timer_rate);
   505	}
   506	
   507	static ssize_t store_timer_rate(struct kobject *kobj,
   508				struct attribute *attr, const char *buf, size_t count)
   509	{
   510		int ret;
   511		unsigned long val;
   512	
   513		ret = strict_strtoul(buf, 0, &val);
   514		if (ret < 0)
   515			return ret;
   516		timer_rate = val;
   517		return count;
   518	}
   519	
   520	static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644,
   521			show_timer_rate, store_timer_rate);
   522	
   523	static struct attribute *interactive_attributes[] = {
   524		&hispeed_freq_attr.attr,
   525		&go_hispeed_load_attr.attr,
   526		&min_sample_time_attr.attr,
   527		&timer_rate_attr.attr,
   528		NULL,
   529	};
   530	
   531	static struct attribute_group interactive_attr_group = {
   532		.attrs = interactive_attributes,
   533		.name = "interactive",
   534	};
   535	
   536	static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
   537			unsigned int event)
   538	{
   539		int rc;
   540		unsigned int j;
   541		struct cpufreq_interactive_cpuinfo *pcpu;
   542		struct cpufreq_frequency_table *freq_table;
   543	
   544		switch (event) {
   545		case CPUFREQ_GOV_START:
   546			if (!cpu_online(policy->cpu))
   547				return -EINVAL;
   548	
   549			freq_table =
   550				cpufreq_frequency_get_table(policy->cpu);
   551	
   552			for_each_cpu(j, policy->cpus) {
   553				pcpu = &per_cpu(cpuinfo, j);
   554				pcpu->policy = policy;
   555				pcpu->target_freq = policy->cur;
   556				pcpu->freq_table = freq_table;
   557				pcpu->freq_change_time_in_idle =
   558					get_cpu_idle_time_us(j,
   559						     &pcpu->freq_change_time);
   560				pcpu->governor_enabled = 1;
   561				smp_wmb();
   562			}
   563	
   564			if (!hispeed_freq)
   565				hispeed_freq = policy->max;
   566	
   567			/*
   568			 * Do not register the idle hook and create sysfs
   569			 * entries if we have already done so.
   570			 */
   571			if (atomic_inc_return(&active_count) > 1)
   572				return 0;
   573	
   574			rc = sysfs_create_group(cpufreq_global_kobject,
   575					&interactive_attr_group);
   576			if (rc)
   577				return rc;
   578	
   579			break;
   580	
   581		case CPUFREQ_GOV_STOP:
   582			for_each_cpu(j, policy->cpus) {
   583				pcpu = &per_cpu(cpuinfo, j);
   584				pcpu->governor_enabled = 0;
   585				smp_wmb();
   586				del_timer_sync(&pcpu->cpu_timer);
   587	
   588				/*
   589				 * Reset idle exit time since we may cancel the timer
   590				 * before it can run after the last idle exit time,
   591				 * to avoid tripping the check in idle exit for a timer
   592				 * that is trying to run.
   593				 */
   594				pcpu->idle_exit_time = 0;
   595			}
   596	
   597			flush_work(&freq_scale_down_work);
   598			if (atomic_dec_return(&active_count) > 0)
   599				return 0;
   600	
   601			sysfs_remove_group(cpufreq_global_kobject,
   602					&interactive_attr_group);
   603	
   604			break;
   605	
   606		case CPUFREQ_GOV_LIMITS:
   607			if (policy->max < policy->cur)
   608				__cpufreq_driver_target(policy,
   609						policy->max, CPUFREQ_RELATION_H);
   610			else if (policy->min > policy->cur)
   611				__cpufreq_driver_target(policy,
   612						policy->min, CPUFREQ_RELATION_L);
   613			break;
   614		}
   615		return 0;
   616	}
   617	
   618	static int cpufreq_interactive_idle_notifier(struct notifier_block *nb,
   619						     unsigned long val,
   620						     void *data)
   621	{
   622		switch (val) {
 > 623		case IDLE_START:
   624			cpufreq_interactive_idle_start();
   625			break;
 > 626		case IDLE_END:
   627			cpufreq_interactive_idle_end();
   628			break;
   629		}
   630	
   631		return 0;
   632	}
   633	
   634	static struct notifier_block cpufreq_interactive_idle_nb = {
   635		.notifier_call = cpufreq_interactive_idle_notifier,
   636	};
   637	
   638	static int __init cpufreq_interactive_init(void)
   639	{
   640		unsigned int i;
   641		struct cpufreq_interactive_cpuinfo *pcpu;
   642		struct sched_param param = { .sched_priority = MAX_RT_PRIO-1 };
   643	
   644		go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;
   645		min_sample_time = DEFAULT_MIN_SAMPLE_TIME;
   646		timer_rate = DEFAULT_TIMER_RATE;
   647	
   648		/* Initalize per-cpu timers */
   649		for_each_possible_cpu(i) {
   650			pcpu = &per_cpu(cpuinfo, i);
   651			init_timer(&pcpu->cpu_timer);
   652			pcpu->cpu_timer.function = cpufreq_interactive_timer;
   653			pcpu->cpu_timer.data = i;
   654		}
   655	
   656		up_task = kthread_create(cpufreq_interactive_up_task, NULL,
   657					 "kinteractiveup");
   658		if (IS_ERR(up_task))
   659			return PTR_ERR(up_task);
   660	
   661		sched_setscheduler_nocheck(up_task, SCHED_FIFO, &param);
   662		get_task_struct(up_task);
   663	
   664		/* No rescuer thread, bind to CPU queuing the work for possibly
   665		   warm cache (probably doesn't matter much). */
   666		down_wq = alloc_workqueue("knteractive_down", 0, 1);
   667	
   668		if (!down_wq)
   669			goto err_freeuptask;
   670	
   671		INIT_WORK(&freq_scale_down_work,
   672			  cpufreq_interactive_freq_down);
   673	
   674		spin_lock_init(&up_cpumask_lock);
   675		spin_lock_init(&down_cpumask_lock);
   676		mutex_init(&set_speed_lock);
   677	
 > 678		idle_notifier_register(&cpufreq_interactive_idle_nb);
   679	
   680		return cpufreq_register_governor(&cpufreq_gov_interactive);
   681	

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 43764 bytes --]

Re: [PATCH 17/70] cpufreq: interactive: take idle notifications only when active

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 5657 bytes --]

Hi Sam,

[auto build test ERROR on pm/linux-next -- if it's inappropriate base, please suggest rules for selecting the more suitable base]

url:    https://github.com/0day-ci/linux/commits/B-lint-Czobor/cpufreq-interactive-New-interactive-governor/20151028-020715
config: sparc64-allyesconfig (attached as .config)
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc64 

All errors (new ones prefixed by >>):

   drivers/cpufreq/cpufreq_interactive.c: In function 'store_hispeed_freq':
   drivers/cpufreq/cpufreq_interactive.c:637:2: error: implicit declaration of function 'strict_strtoull' [-Werror=implicit-function-declaration]
     ret = strict_strtoull(buf, 0, &val);
     ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'store_go_hispeed_load':
   drivers/cpufreq/cpufreq_interactive.c:660:2: error: implicit declaration of function 'strict_strtoul' [-Werror=implicit-function-declaration]
     ret = strict_strtoul(buf, 0, &val);
     ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'cpufreq_interactive_idle_notifier':
   drivers/cpufreq/cpufreq_interactive.c:827:7: error: 'IDLE_START' undeclared (first use in this function)
     case IDLE_START:
          ^
   drivers/cpufreq/cpufreq_interactive.c:827:7: note: each undeclared identifier is reported only once for each function it appears in
   drivers/cpufreq/cpufreq_interactive.c:830:7: error: 'IDLE_END' undeclared (first use in this function)
     case IDLE_END:
          ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'cpufreq_governor_interactive':
   drivers/cpufreq/cpufreq_interactive.c:895:3: error: implicit declaration of function 'idle_notifier_register' [-Werror=implicit-function-declaration]
      idle_notifier_register(&cpufreq_interactive_idle_nb);
      ^
>> drivers/cpufreq/cpufreq_interactive.c:918:3: error: implicit declaration of function 'idle_notifier_unregister' [-Werror=implicit-function-declaration]
      idle_notifier_unregister(&cpufreq_interactive_idle_nb);
      ^
   cc1: some warnings being treated as errors

vim +/idle_notifier_unregister +918 drivers/cpufreq/cpufreq_interactive.c

   821	
   822	static int cpufreq_interactive_idle_notifier(struct notifier_block *nb,
   823						     unsigned long val,
   824						     void *data)
   825	{
   826		switch (val) {
 > 827		case IDLE_START:
   828			cpufreq_interactive_idle_start();
   829			break;
   830		case IDLE_END:
   831			cpufreq_interactive_idle_end();
   832			break;
   833		}
   834	
   835		return 0;
   836	}
   837	
   838	static struct notifier_block cpufreq_interactive_idle_nb = {
   839		.notifier_call = cpufreq_interactive_idle_notifier,
   840	};
   841	
   842	static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
   843			unsigned int event)
   844	{
   845		int rc;
   846		unsigned int j;
   847		struct cpufreq_interactive_cpuinfo *pcpu;
   848		struct cpufreq_frequency_table *freq_table;
   849	
   850		switch (event) {
   851		case CPUFREQ_GOV_START:
   852			if (!cpu_online(policy->cpu))
   853				return -EINVAL;
   854	
   855			freq_table =
   856				cpufreq_frequency_get_table(policy->cpu);
   857	
   858			for_each_cpu(j, policy->cpus) {
   859				pcpu = &per_cpu(cpuinfo, j);
   860				pcpu->policy = policy;
   861				pcpu->target_freq = policy->cur;
   862				pcpu->freq_table = freq_table;
   863				pcpu->target_set_time_in_idle =
   864					get_cpu_idle_time_us(j,
   865						     &pcpu->target_set_time);
   866				pcpu->floor_freq = pcpu->target_freq;
   867				pcpu->floor_validate_time =
   868					pcpu->target_set_time;
   869				pcpu->hispeed_validate_time =
   870					pcpu->target_set_time;
   871				pcpu->governor_enabled = 1;
   872				smp_wmb();
   873			}
   874	
   875			if (!hispeed_freq)
   876				hispeed_freq = policy->max;
   877	
   878			/*
   879			 * Do not register the idle hook and create sysfs
   880			 * entries if we have already done so.
   881			 */
   882			if (atomic_inc_return(&active_count) > 1)
   883				return 0;
   884	
   885			rc = sysfs_create_group(cpufreq_global_kobject,
   886					&interactive_attr_group);
   887			if (rc)
   888				return rc;
   889	
   890			rc = input_register_handler(&cpufreq_interactive_input_handler);
   891			if (rc)
   892				pr_warn("%s: failed to register input handler\n",
   893					__func__);
   894	
   895			idle_notifier_register(&cpufreq_interactive_idle_nb);
   896			break;
   897	
   898		case CPUFREQ_GOV_STOP:
   899			for_each_cpu(j, policy->cpus) {
   900				pcpu = &per_cpu(cpuinfo, j);
   901				pcpu->governor_enabled = 0;
   902				smp_wmb();
   903				del_timer_sync(&pcpu->cpu_timer);
   904	
   905				/*
   906				 * Reset idle exit time since we may cancel the timer
   907				 * before it can run after the last idle exit time,
   908				 * to avoid tripping the check in idle exit for a timer
   909				 * that is trying to run.
   910				 */
   911				pcpu->idle_exit_time = 0;
   912			}
   913	
   914			flush_work(&freq_scale_down_work);
   915			if (atomic_dec_return(&active_count) > 0)
   916				return 0;
   917	
 > 918			idle_notifier_unregister(&cpufreq_interactive_idle_nb);
   919			input_unregister_handler(&cpufreq_interactive_input_handler);
   920			sysfs_remove_group(cpufreq_global_kobject,
   921					&interactive_attr_group);

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 43764 bytes --]

Re: [PATCH 17/70] cpufreq: interactive: take idle notifications only when active

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 5657 bytes --]

Hi Sam,

[auto build test ERROR on pm/linux-next -- if it's inappropriate base, please suggest rules for selecting the more suitable base]

url:    https://github.com/0day-ci/linux/commits/B-lint-Czobor/cpufreq-interactive-New-interactive-governor/20151028-020715
config: sparc64-allyesconfig (attached as .config)
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc64 

All errors (new ones prefixed by >>):

   drivers/cpufreq/cpufreq_interactive.c: In function 'store_hispeed_freq':
   drivers/cpufreq/cpufreq_interactive.c:637:2: error: implicit declaration of function 'strict_strtoull' [-Werror=implicit-function-declaration]
     ret = strict_strtoull(buf, 0, &val);
     ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'store_go_hispeed_load':
   drivers/cpufreq/cpufreq_interactive.c:660:2: error: implicit declaration of function 'strict_strtoul' [-Werror=implicit-function-declaration]
     ret = strict_strtoul(buf, 0, &val);
     ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'cpufreq_interactive_idle_notifier':
   drivers/cpufreq/cpufreq_interactive.c:827:7: error: 'IDLE_START' undeclared (first use in this function)
     case IDLE_START:
          ^
   drivers/cpufreq/cpufreq_interactive.c:827:7: note: each undeclared identifier is reported only once for each function it appears in
   drivers/cpufreq/cpufreq_interactive.c:830:7: error: 'IDLE_END' undeclared (first use in this function)
     case IDLE_END:
          ^
   drivers/cpufreq/cpufreq_interactive.c: In function 'cpufreq_governor_interactive':
   drivers/cpufreq/cpufreq_interactive.c:895:3: error: implicit declaration of function 'idle_notifier_register' [-Werror=implicit-function-declaration]
      idle_notifier_register(&cpufreq_interactive_idle_nb);
      ^
>> drivers/cpufreq/cpufreq_interactive.c:918:3: error: implicit declaration of function 'idle_notifier_unregister' [-Werror=implicit-function-declaration]
      idle_notifier_unregister(&cpufreq_interactive_idle_nb);
      ^
   cc1: some warnings being treated as errors

vim +/idle_notifier_unregister +918 drivers/cpufreq/cpufreq_interactive.c

   821	
   822	static int cpufreq_interactive_idle_notifier(struct notifier_block *nb,
   823						     unsigned long val,
   824						     void *data)
   825	{
   826		switch (val) {
 > 827		case IDLE_START:
   828			cpufreq_interactive_idle_start();
   829			break;
   830		case IDLE_END:
   831			cpufreq_interactive_idle_end();
   832			break;
   833		}
   834	
   835		return 0;
   836	}
   837	
   838	static struct notifier_block cpufreq_interactive_idle_nb = {
   839		.notifier_call = cpufreq_interactive_idle_notifier,
   840	};
   841	
   842	static int cpufreq_governor_interactive(struct cpufreq_policy *policy,
   843			unsigned int event)
   844	{
   845		int rc;
   846		unsigned int j;
   847		struct cpufreq_interactive_cpuinfo *pcpu;
   848		struct cpufreq_frequency_table *freq_table;
   849	
   850		switch (event) {
   851		case CPUFREQ_GOV_START:
   852			if (!cpu_online(policy->cpu))
   853				return -EINVAL;
   854	
   855			freq_table =
   856				cpufreq_frequency_get_table(policy->cpu);
   857	
   858			for_each_cpu(j, policy->cpus) {
   859				pcpu = &per_cpu(cpuinfo, j);
   860				pcpu->policy = policy;
   861				pcpu->target_freq = policy->cur;
   862				pcpu->freq_table = freq_table;
   863				pcpu->target_set_time_in_idle =
   864					get_cpu_idle_time_us(j,
   865						     &pcpu->target_set_time);
   866				pcpu->floor_freq = pcpu->target_freq;
   867				pcpu->floor_validate_time =
   868					pcpu->target_set_time;
   869				pcpu->hispeed_validate_time =
   870					pcpu->target_set_time;
   871				pcpu->governor_enabled = 1;
   872				smp_wmb();
   873			}
   874	
   875			if (!hispeed_freq)
   876				hispeed_freq = policy->max;
   877	
   878			/*
   879			 * Do not register the idle hook and create sysfs
   880			 * entries if we have already done so.
   881			 */
   882			if (atomic_inc_return(&active_count) > 1)
   883				return 0;
   884	
   885			rc = sysfs_create_group(cpufreq_global_kobject,
   886					&interactive_attr_group);
   887			if (rc)
   888				return rc;
   889	
   890			rc = input_register_handler(&cpufreq_interactive_input_handler);
   891			if (rc)
   892				pr_warn("%s: failed to register input handler\n",
   893					__func__);
   894	
   895			idle_notifier_register(&cpufreq_interactive_idle_nb);
   896			break;
   897	
   898		case CPUFREQ_GOV_STOP:
   899			for_each_cpu(j, policy->cpus) {
   900				pcpu = &per_cpu(cpuinfo, j);
   901				pcpu->governor_enabled = 0;
   902				smp_wmb();
   903				del_timer_sync(&pcpu->cpu_timer);
   904	
   905				/*
   906				 * Reset idle exit time since we may cancel the timer
   907				 * before it can run after the last idle exit time,
   908				 * to avoid tripping the check in idle exit for a timer
   909				 * that is trying to run.
   910				 */
   911				pcpu->idle_exit_time = 0;
   912			}
   913	
   914			flush_work(&freq_scale_down_work);
   915			if (atomic_dec_return(&active_count) > 0)
   916				return 0;
   917	
 > 918			idle_notifier_unregister(&cpufreq_interactive_idle_nb);
   919			input_unregister_handler(&cpufreq_interactive_input_handler);
   920			sysfs_remove_group(cpufreq_global_kobject,
   921					&interactive_attr_group);

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 43764 bytes --]

Re: [PATCH 01/70] cpufreq: interactive: New 'interactive' governor

[Rafael J. Wysocki]

Hi,

On Tuesday, October 27, 2015 06:29:49 PM Bálint Czobor wrote:
> From: Mike Chan <mike@android.com>
> 
> This governor is designed for latency-sensitive workloads, such as
> interactive user interfaces.  The interactive governor aims to be
> significantly more responsive to ramp CPU quickly up when CPU-intensive
> activity begins.
> 
> Existing governors sample CPU load at a particular rate, typically
> every X ms.  This can lead to under-powering UI threads for the period of
> time during which the user begins interacting with a previously-idle system
> until the next sample period happens.
> 
> The 'interactive' governor uses a different approach. Instead of sampling
> the CPU at a specified rate, the governor will check whether to scale the
> CPU frequency up soon after coming out of idle.  When the CPU comes out of
> idle, a timer is configured to fire within 1-2 ticks.  If the CPU is very
> busy from exiting idle to when the timer fires then we assume the CPU is
> underpowered and ramp to MAX speed.
> 
> If the CPU was not sufficiently busy to immediately ramp to MAX speed, then
> the governor evaluates the CPU load since the last speed adjustment,
> choosing the highest value between that longer-term load or the short-term
> load since idle exit to determine the CPU speed to ramp to.
> 
> A realtime thread is used for scaling up, giving the remaining tasks the
> CPU performance benefit, unlike existing governors which are more likely to
> schedule rampup work to occur after your performance starved tasks have
> completed.
> 
> The tuneables for this governor are:
> /sys/devices/system/cpu/cpufreq/interactive/min_sample_time:
> 	The minimum amount of time to spend at the current frequency before
> 	ramping down. This is to ensure that the governor has seen enough
> 	historic CPU load data to determine the appropriate workload.
> 	Default is 80000 uS.
> /sys/devices/system/cpu/cpufreq/interactive/go_maxspeed_load
> 	The CPU load at which to ramp to max speed.  Default is 85.
> 
> Change-Id: Ib2b362607c62f7c56d35f44a9ef3280f98c17585
> Signed-off-by: Mike Chan <mike@android.com>
> Signed-off-by: Todd Poynor <toddpoynor@google.com>
> Bug: 3152864
> Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>

It's good to see that submitted, but it'll have to go through a detailed
review which is going to take some time.

One my observation after a cursory look at it is that at least some later
patches of the series modify drivers/cpufreq/cpufreq_interactive.c which is
a new file added by the first patch.  Is there any particular reason to
avoid folding all of those patches into the first one?

Thanks,
Rafael

Re: [PATCH 66/70] cpufreq: interactive: Fix compile errors in accordance with changes from 3.14 to 3.18

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

Hi Bálint,

[auto build test ERROR on pm/linux-next -- if it's inappropriate base, please suggest rules for selecting the more suitable base]

url:    https://github.com/0day-ci/linux/commits/B-lint-Czobor/cpufreq-interactive-New-interactive-governor/20151028-020715
config: i386-allmodconfig (attached as .config)
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

   ERROR: "sched_setscheduler_nocheck" undefined!
>> ERROR: "idle_notifier_unregister" undefined!
>> ERROR: "idle_notifier_register" undefined!

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 51611 bytes --]

Re: [PATCH 66/70] cpufreq: interactive: Fix compile errors in accordance with changes from 3.14 to 3.18

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

Hi Bálint,

[auto build test ERROR on pm/linux-next -- if it's inappropriate base, please suggest rules for selecting the more suitable base]

url:    https://github.com/0day-ci/linux/commits/B-lint-Czobor/cpufreq-interactive-New-interactive-governor/20151028-020715
config: i386-allmodconfig (attached as .config)
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

   ERROR: "sched_setscheduler_nocheck" undefined!
>> ERROR: "idle_notifier_unregister" undefined!
>> ERROR: "idle_notifier_register" undefined!

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 51611 bytes --]

Re: [PATCH 01/70] cpufreq: interactive: New 'interactive' governor

[Viresh Kumar]

Hi Bálint,

On 28-10-15, 01:59, Rafael J. Wysocki wrote:
> On Tuesday, October 27, 2015 06:29:49 PM Bálint Czobor wrote:
> > From: Mike Chan <mike@android.com>
> > 
> > This governor is designed for latency-sensitive workloads, such as
> > interactive user interfaces.  The interactive governor aims to be
> > significantly more responsive to ramp CPU quickly up when CPU-intensive
> > activity begins.
> > 
> > Existing governors sample CPU load at a particular rate, typically
> > every X ms.  This can lead to under-powering UI threads for the period of
> > time during which the user begins interacting with a previously-idle system
> > until the next sample period happens.
> > 
> > The 'interactive' governor uses a different approach. Instead of sampling
> > the CPU at a specified rate, the governor will check whether to scale the
> > CPU frequency up soon after coming out of idle.  When the CPU comes out of
> > idle, a timer is configured to fire within 1-2 ticks.  If the CPU is very
> > busy from exiting idle to when the timer fires then we assume the CPU is
> > underpowered and ramp to MAX speed.
> > 
> > If the CPU was not sufficiently busy to immediately ramp to MAX speed, then
> > the governor evaluates the CPU load since the last speed adjustment,
> > choosing the highest value between that longer-term load or the short-term
> > load since idle exit to determine the CPU speed to ramp to.
> > 
> > A realtime thread is used for scaling up, giving the remaining tasks the
> > CPU performance benefit, unlike existing governors which are more likely to
> > schedule rampup work to occur after your performance starved tasks have
> > completed.
> > 
> > The tuneables for this governor are:
> > /sys/devices/system/cpu/cpufreq/interactive/min_sample_time:
> > 	The minimum amount of time to spend at the current frequency before
> > 	ramping down. This is to ensure that the governor has seen enough
> > 	historic CPU load data to determine the appropriate workload.
> > 	Default is 80000 uS.
> > /sys/devices/system/cpu/cpufreq/interactive/go_maxspeed_load
> > 	The CPU load at which to ramp to max speed.  Default is 85.
> > 
> > Change-Id: Ib2b362607c62f7c56d35f44a9ef3280f98c17585
> > Signed-off-by: Mike Chan <mike@android.com>
> > Signed-off-by: Todd Poynor <toddpoynor@google.com>
> > Bug: 3152864
> > Signed-off-by: Bálint Czobor <czoborbalint@gmail.com>
> 
> It's good to see that submitted, but it'll have to go through a detailed
> review which is going to take some time.

These are the last memories I have around upstreaming this governor:
http://marc.info/?l=linux-kernel&m=132867057910479&w=2

Has anything changed after that? Or we decided to go ahead with it and
upstream ?

> One my observation after a cursory look at it is that at least some later
> patches of the series modify drivers/cpufreq/cpufreq_interactive.c which is
> a new file added by the first patch.  Is there any particular reason to
> avoid folding all of those patches into the first one?

Right, not to discourage you from upstreaming stuff, but there were
few things that went wrong here:

- I don't see the idle_notifier_register thing present in upstream
  kernel for anything other than x86, and I am sure you were
  targeting ARM here :)
- And I don't see the same stuff present in your patchset as well,
  sorry if I am wrong. And if I am write, you didn't even base this of
  a mainline release and compile tested it.
- 70 patches in a series are really really huge. I wouldn't expect
  people to review any of that. Though to be fair enough, I must admit
  of doing similar things in the past :)
- There is something called as a cover-letter (in case you aren't
  aware about it), which should have been written here to give a
  really nice summary about what you are trying to do here. There are
  non-ARM people, who wouldn't have known about this stuff at all
  until now.
- Mainline doesn't care about history of commits that are present in a
  non-mainline repository, even if it is android. Yeah, you screw up a
  bit here as people loose their credits for what they have worked on.
  But it should really have been 1-3 patches ONLY for the governor.
  And maybe a patchset of 5-7 patches, but it can be much lesser as
  well.

Cc'ing Peter/Ingo for their inputs..

-- 
viresh

Re: [PATCH 01/70] cpufreq: interactive: New 'interactive' governor

[Viresh Kumar]

On 28-10-15, 23:01, Bálint Czobor wrote:
> Please ignore that single patch, doesn't even build. I'll submit it again soon.

Can you please reply to the queries I posted to you first, before
sending any new updates ?

-- 
viresh

Re: [PATCH 01/70] cpufreq: interactive: New 'interactive' governor

[Peter Zijlstra]

On Wed, Oct 28, 2015 at 08:30:42AM +0530, Viresh Kumar wrote:
> These are the last memories I have around upstreaming this governor:
> http://marc.info/?l=linux-kernel&m=132867057910479&w=2
> 
> Has anything changed after that? Or we decided to go ahead with it and
> upstream ?

Urgh, I thought we were trying to kill all the various governors by
integrating the lot with the scheduler !?

Re: [PATCH 01/70] cpufreq: interactive: New 'interactive' governor

[Steve Muckle]

On 11/02/2015 06:02 AM, Peter Zijlstra wrote:
> On Wed, Oct 28, 2015 at 08:30:42AM +0530, Viresh Kumar wrote:
>> These are the last memories I have around upstreaming this governor:
>> http://marc.info/?l=linux-kernel&m=132867057910479&w=2
>>
>> Has anything changed after that? Or we decided to go ahead with it and
>> upstream ?
> 
> Urgh, I thought we were trying to kill all the various governors by
> integrating the lot with the scheduler !?

That's the goal but it may be a while. There are major unanswered questions IMO about the viability of replacing all the governers with a single scheduler-driven implementation. Is per-entity load tracking responsive enough to drive CPU frequency? Can we replicate the functionality available and being used today with the proposed design? These aren't necessarily showstoppers but they could require considerable extra work (and debate).

If we move forward without ensuring these concerns are addressed I believe it would make things even worse than they are now since there will be people/vendors remaining behind on cpufreq governors, forward porting them if necessary, and there will be the fundamentally different sched-driven solution to support and maintain as well.

Having the interactive governor in the tree would help in the evaluation and comparison process, and also immediately address a widely used component being out of tree.