* [PATCH 00/13] High performance balancing logic for big.LITTLE
@ 2015-11-06 12:02 Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 01/13] hperf_hmp: add new config for arm and arm64 Arseniy Krasnov
` (13 more replies)
0 siblings, 14 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz; +Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel
Prologue
The patch set introduces an extension to the default Linux CPU scheduler (CFS).
The main purpose of the extension is utilization of a big.LITTLE CPU for maximum
performance. Such solution may be useful for users of OdroidXU-3 board
(supporting 8 cores) who doesn't care about power efficiency.
Maximum utilization was reached using the following policies:
1) A15 cores must be utilized as much as possible e.g. idle A15 cores always
pull some task from A7 core.
2) After execution of a task on A7 core for some period of time it should be
swapped with an appropriate task from A15 cluster in order to achieve
fairness.
3) Load of big and little clusters is balanced according to frequency and
A15/A7 slowdown coefficient.
Approach Description
The scheduler creates a hierarchy of two domains: MC and HMP. The MC domain is a
default domain for SCHED_MC config. The HMP domain contains two clusters: A15
and A7 CPUs. Balancing between HMP domains is performed by the new logic, in MC
domains, in turn, balancing is done by the default logic of the 'load_balance()'
function.
To perform balancing between HMP domains, the load of each cluster is calculated
in scheduler's softirq handler. Then, this value is scaled according to each
cluster's frequency and slowdown coefficient which is a ratio of busy-loop
performance on A15 and A7. There are three ways of migration between two
clusters: from A15 cluster to A7 cluster (if load on A15 cluster is too high),
from A7 cluster to A15 cluster (otherwise) and task swapping when load on both
clusters is the same. To migrate some task from one cluster to another firstly
this task should be selected. To find a task suitable for migration the
scheduler uses a special per-task metric called 'druntime'. It is based on CFS's
vruntime metric but its grow direction depends on a core where the task is
executed: for A15 core it grows up, for A7 core, in turn, it goes down. So,
being the druntime value close to zero means that the task is executed on both
clusters for the same amount of time. As a result, to get a task for migration
it scans each runqueue to find a task with highest/lowest druntime depending on
which cluster is scanned; after, when the task is found, it is moved to another
cluster. These balancing steps are performed in each scheduler balancing
operation executed by softirq.
To get maximum performance A15 cores must be fully utilized; this means that
idle A15 cores are always able to pull tasks from A7 cores while A7 cores cannot
do that from A15 cores.
An finally, let's look to fairness - it is provided by swapping of tasks during
every softirq balancing: when balance is broken it tries to repair the balance
moving tasks from one cluster to another, then when the clusters are balanced,
the tasks are swapped during each softirq balancing. In addition to this logic,
'select_task_rq_fair' was modified in order to place woken tasks to least loaded
CPU, because it won't break the balance between A15 and A7 cores.
Test results
Several test kits were used for performance measurement of the solution.
All comparision is done against the Linaro MP scheduler.
The first test case is a parsec benchmark suite. It contains different types of
tasks like cluster searching or pattern recognition in order to test scheduler
performance. Results of some benchmarks are listed in the text below (in
seconds):
Streamcluster:
Developed by Princeton University and solves the online clustering problem.
Streamcluster was included in the PARSEC benchmark suite because of the
importance of data mining algorithms and the prevalence of problems with
streaming characteristics.
Threads HPERF_HMP Linaro MP
1 27,333 27,422
2 14,162 14,197
3 10,099 10,168
4 8,227 8,332
5 10,922 23,349
6 10,85 22,507
7 11,39 22,041
8 12,307 21,181
9 20,339 22,115
10 21,33 23,746
11 23,289 24,831
12 25,363 26,699
13 34,091 34,84
14 34,758 38,661
15 35,743 38,688
16 38,1 44,735
17 41,165 77,098
18 44,223 102,633
19 46,177 113,748
20 48,22 119,146
21 52,372 135,499
22 54,319 136,454
23 56,218 141,924
24 57,843 145,727
25 61,759 158,754
26 63,179 163,915
27 64,987 167,559
28 67,329 171,203
29 70,489 185,171
30 73,084 189,303
31 75,264 192,487
32 77,015 197,27
avg 40,373 87,543
Bodytrack:
This computer vision application is an Intel RMS workload which tracks a human
body with multiple cameras through an image sequence. This benchmark was
included due to the increasing significance of computer vision algorithms in
areas such as video surveillance, character animation and computer interfaces.
Threads HPERF_HMP Linaro MP
1 15,884 16,632
2 8,536 9,42
3 6,037 7,257
4 4,84 6,076
5 8,835 5,739
6 4,437 5,513
7 4,119 5,474
8 3,992 5,115
9 3,854 5,164
10 3,92 4,911
11 3,854 4,932
12 3,83 4,816
13 3,839 5,643
14 3,861 4,816
15 3,889 4,896
16 3,845 4,854
17 3,872 4,837
18 3,852 4,876
19 4,304 4,868
20 3,915 4,928
21 3,87 4,841
22 3,858 4,995
23 3,881 4,97
24 3,876 4,899
25 3,854 4,96
26 3,869 4,902
27 3,874 4,979
28 3,88 4,928
29 3,914 5,008
30 3,889 5,216
31 3,898 5,242
32 3,894 5,199
avg 4,689 5,653
Blackscholes:
This application is an Intel RMS benchmark. It calculates the prices for a
portfolio of European options analytically with the Black-Scholes partial
differential equation. There is no closed-form expression for the blackscholes
equation and as such it must be computed numerically.
Threads HPERF_HMP Linaro MP
1 7,293 6,807
2 3,886 4,044
3 2,906 2,911
4 2,429 2,427
5 2,58 2,985
6 2,401 2,672
7 2,205 2,411
8 2,132 2,293
9 2,074 2,41
10 2,067 2,264
11 2,054 2,205
12 2,091 2,222
13 2,042 2,28
14 2,035 2,222
15 2,026 2,25
16 2,024 2,177
17 2,021 2,173
18 2,033 2,09
19 2,03 2,05
20 2,024 2,158
21 2,002 2,175
22 2,026 2,179
23 2,017 2,134
24 2,01 2,156
25 2,009 2,155
26 2,013 2,179
27 2,017 2,177
28 2,019 2,189
29 2,013 2,158
30 2,002 2,162
31 2,016 2,16
32 2,012 2,159
avg 2,328 2,469
Also, well known Antutu benchmark was executed on Exynos 5433 board:
HPERF_HMP Linaro MP
Integral benchmark result 42400 36860
Result: hperf_hmp is 15% better.
Arseniy Krasnov (13):
hperf_hmp: add new config for arm and arm64.
hperf_hmp: introduce hew domain flag.
hperf_hmp: add sched domains initialization.
hperf_hmp: scheduler initialization routines.
hperf_hmp: introduce druntime metric.
hperf_hmp: is_hmp_imbalance introduced.
hperf_hmp: migration auxiliary functions.
hperf_hmp: swap tasks function.
hperf_hmp: one way balancing function.
hperf_hmp: idle pull function.
hperf_hmp: task CPU selection logic.
hperf_hmp: rest of logic.
hperf_hmp: cpufreq routines.
arch/arm/Kconfig | 21 +
arch/arm/kernel/topology.c | 6 +-
arch/arm64/Kconfig | 21 +
include/linux/sched.h | 17 +
kernel/sched/core.c | 65 +-
kernel/sched/fair.c | 1553 ++++++++++++++++++++++++++++++++++++++++----
kernel/sched/sched.h | 16 +
7 files changed, 1586 insertions(+), 113 deletions(-)
--
1.9.1
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 01/13] hperf_hmp: add new config for arm and arm64.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 02/13] hperf_hmp: introduce hew domain flag Arseniy Krasnov
` (12 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
New config option which enables new scheduler logic: HPERF_HMP. Also
adds the following options:
'HPERF_HMP_DEBUG': enables extra runtime checks of balancing parameteres.
'HMP_FAST_CPU_MASK': CPU mask of A15 cluster(in hex string).
'HMP_SLOW_CPU_MASK': CPU mask of A7 cluster(in hex string).
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
arch/arm/Kconfig | 21 +++++++++++++++++++++
arch/arm64/Kconfig | 21 +++++++++++++++++++++
2 files changed, 42 insertions(+)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 72ad724..0581914 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1387,6 +1387,27 @@ config SCHED_MC
making when dealing with multi-core CPU chips at a cost of slightly
increased overhead in some places. If unsure say N here.
+config HPERF_HMP
+ bool "HPERF_HMP load balancing enhancements for ARM big.LITTLE"
+ select SCHED_MC
+ help
+ Uses HPERF_HMP load balancing algorithm between A7 and A15 CPU domains.
+
+config HPERF_HMP_DEBUG
+ bool "Additional HPERF_HMP runtime debug checks"
+ depends on HPERF_HMP
+ default n
+
+config HMP_FAST_CPU_MASK
+ string "Fast (Cortex-A15) CPU mask for HPERF_HMP"
+ default ""
+ depends on HPERF_HMP
+
+config HMP_SLOW_CPU_MASK
+ string "Slow (Cortex-A7) CPU mask for HPERF_HMP"
+ default ""
+ depends on HPERF_HMP
+
config SCHED_SMT
bool "SMT scheduler support"
depends on ARM_CPU_TOPOLOGY
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 07d1811..71a8983 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -412,6 +412,27 @@ config SCHED_MC
making when dealing with multi-core CPU chips at a cost of slightly
increased overhead in some places. If unsure say N here.
+config HPERF_HMP
+ bool "HPERF_HMP load balancing enhancements for ARM big.LITTLE"
+ select SCHED_MC
+ help
+ Uses HPERF_HMP load balancing algorithm between A7 and A15 CPU domains.
+
+config HPERF_HMP_DEBUG
+ bool "Additional HPERF_HMP runtime debug checks"
+ depends on HPERF_HMP
+ default n
+
+config HMP_FAST_CPU_MASK
+ string "Fast (Cortex-A15) CPU mask for HPERF_HMP"
+ default ""
+ depends on HPERF_HMP
+
+config HMP_SLOW_CPU_MASK
+ string "Slow (Cortex-A7) CPU mask for HPERF_HMP"
+ default ""
+ depends on HPERF_HMP
+
config SCHED_SMT
bool "SMT scheduler support"
help
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 02/13] hperf_hmp: introduce hew domain flag.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 01/13] hperf_hmp: add new config for arm and arm64 Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 03/13] hperf_hmp: add sched domains initialization Arseniy Krasnov
` (11 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
New scheduler domain type: HMP. Each big.LITTLE cluster is detected by
scheduler as HMP domain. HPERF_HMP logic works between two HMP domains, the
default CFS logic, in turn, works inside the HMP domain.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
arch/arm/kernel/topology.c | 6 +++++-
include/linux/sched.h | 4 ++++
kernel/sched/core.c | 9 ++++++++-
3 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/arch/arm/kernel/topology.c b/arch/arm/kernel/topology.c
index 08b7847..7fcc5fe 100644
--- a/arch/arm/kernel/topology.c
+++ b/arch/arm/kernel/topology.c
@@ -285,7 +285,11 @@ static struct sched_domain_topology_level arm_topology[] = {
{ cpu_corepower_mask, cpu_corepower_flags, SD_INIT_NAME(GMC) },
{ cpu_coregroup_mask, cpu_core_flags, SD_INIT_NAME(MC) },
#endif
- { cpu_cpu_mask, SD_INIT_NAME(DIE) },
+ { cpu_cpu_mask,
+#ifdef CONFIG_HPERF_HMP
+ .flags = SD_HMP_BALANCE,
+#endif
+ SD_INIT_NAME(DIE)},
{ NULL, },
};
diff --git a/include/linux/sched.h b/include/linux/sched.h
index b7b9501..eb084df 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -990,6 +990,10 @@ extern void wake_up_q(struct wake_q_head *head);
#define SD_OVERLAP 0x2000 /* sched_domains of this level overlap */
#define SD_NUMA 0x4000 /* cross-node balancing */
+#ifdef CONFIG_HPERF_HMP
+#define SD_HMP_BALANCE 0x8000 /* Use HMP load balancing algorithm */
+#endif
+
#ifdef CONFIG_SCHED_SMT
static inline int cpu_smt_flags(void)
{
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index bcd214e..16092e0 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6410,6 +6410,9 @@ sd_init(struct sched_domain_topology_level *tl, int cpu)
| 0*SD_PREFER_SIBLING
| 0*SD_NUMA
| sd_flags
+#ifdef CONFIG_HPERF_HMP
+ | (tl->flags & SD_HMP_BALANCE)
+#endif
,
.last_balance = jiffies,
@@ -6472,7 +6475,11 @@ static struct sched_domain_topology_level default_topology[] = {
#ifdef CONFIG_SCHED_MC
{ cpu_coregroup_mask, cpu_core_flags, SD_INIT_NAME(MC) },
#endif
- { cpu_cpu_mask, SD_INIT_NAME(DIE) },
+ { cpu_cpu_mask,
+#ifdef CONFIG_HPERF_HMP
+ .flags = SD_HMP_BALANCE,
+#endif
+ SD_INIT_NAME(DIE)},
{ NULL, },
};
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 03/13] hperf_hmp: add sched domains initialization.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 01/13] hperf_hmp: add new config for arm and arm64 Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 02/13] hperf_hmp: introduce hew domain flag Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 04/13] hperf_hmp: scheduler initialization routines Arseniy Krasnov
` (10 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
Attaching CPU clusters as 'sched_group' to HMP domains. Each HMP domain
has two pointers to A15 and A7 scheduling groups(struct sched_group).
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
include/linux/sched.h | 4 ++++
kernel/sched/core.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 53 insertions(+)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index eb084df..aa72125 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1057,6 +1057,10 @@ struct sched_domain {
u64 max_newidle_lb_cost;
unsigned long next_decay_max_lb_cost;
+#ifdef CONFIG_HPERF_HMP
+ struct sched_group *a15_group;
+ struct sched_group *a7_group;
+#endif
#ifdef CONFIG_SCHEDSTATS
/* load_balance() stats */
unsigned int lb_count[CPU_MAX_IDLE_TYPES];
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 16092e0..e3a632f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -90,6 +90,16 @@
#define CREATE_TRACE_POINTS
#include <trace/events/sched.h>
+#ifdef CONFIG_HPERF_HMP
+/* cpumask for A15 cpus */
+static DECLARE_BITMAP(cpu_fastest_bits, CONFIG_NR_CPUS);
+struct cpumask *cpu_fastest_mask = to_cpumask(cpu_fastest_bits);
+
+/* cpumask for A7 cpus */
+static DECLARE_BITMAP(cpu_slowest_bits, CONFIG_NR_CPUS);
+struct cpumask *cpu_slowest_mask = to_cpumask(cpu_slowest_bits);
+#endif
+
DEFINE_MUTEX(sched_domains_mutex);
DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
@@ -6971,6 +6981,45 @@ static int build_sched_domains(const struct cpumask *cpu_map,
sd = *per_cpu_ptr(d.sd, i);
cpu_attach_domain(sd, d.rd, i);
}
+
+#ifdef CONFIG_HPERF_HMP
+ for (i = nr_cpumask_bits - 1; i >= 0; i--) {
+ if (!cpumask_test_cpu(i, cpu_map))
+ continue;
+
+ for (sd = *per_cpu_ptr(d.sd, i); sd; sd = sd->parent) {
+ struct sched_group *sg;
+ sd->a7_group = NULL;
+ sd->a15_group = NULL;
+
+ /* Process only HMP domains */
+ if (!(sd->flags & SD_HMP_BALANCE))
+ continue;
+
+ /*
+ * Process sched groups of this domain.
+ * Attach sg to hmp domains.
+ */
+ sg = sd->groups;
+ do {
+ if (!sg->sgc)
+ goto next_sg;
+#ifdef CONFIG_SCHED_DEBUG
+ printk(KERN_EMERG "Attaching CPUs 0x%08lX to domain %s\n",
+ sched_group_cpus(sg)->bits[0], sd->name);
+#endif
+ if (cpumask_intersects(sched_group_cpus(sg),
+ cpu_fastest_mask))
+ sd->a15_group = sg;
+ else
+ sd->a7_group = sg;
+next_sg:
+ sg = sg->next;
+ } while (sg != sd->groups);
+ }
+ }
+#endif /* CONFIG_HPERF_HMP */
+
rcu_read_unlock();
ret = 0;
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 04/13] hperf_hmp: scheduler initialization routines.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (2 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 03/13] hperf_hmp: add sched domains initialization Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 05/13] hperf_hmp: introduce druntime metric Arseniy Krasnov
` (9 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
Adds new fields to 'rq' structure and routine called during fair class
setup, which initializes some HMP scheduler variables: big and little cluster
masks. They are read from kernel config(if set), else default values are used.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
kernel/sched/core.c | 4 ++++
kernel/sched/fair.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
kernel/sched/sched.h | 15 +++++++++++++++
3 files changed, 65 insertions(+)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e3a632f..8747e06 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7488,6 +7488,10 @@ void __init sched_init(void)
#endif
init_rq_hrtick(rq);
atomic_set(&rq->nr_iowait, 0);
+#ifdef CONFIG_HPERF_HMP
+ rq->druntime_sum = 0;
+ rq->nr_hmp_tasks = 0;
+#endif
}
set_load_weight(&init_task);
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 9a5e60f..c57007f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -100,6 +100,11 @@ const_debug unsigned int sysctl_sched_migration_cost = 500000UL;
*/
unsigned int __read_mostly sysctl_sched_shares_window = 10000000UL;
+#ifdef CONFIG_HPERF_HMP
+extern void hmp_set_cpu_masks(struct cpumask *, struct cpumask *);
+static unsigned int freq_scale_cpu_power[CONFIG_NR_CPUS];
+#endif /* CONFIG_HPERF_HMP */
+
#ifdef CONFIG_CFS_BANDWIDTH
/*
* Amount of runtime to allocate from global (tg) to local (per-cfs_rq) pool
@@ -8305,8 +8310,38 @@ void show_numa_stats(struct task_struct *p, struct seq_file *m)
#endif /* CONFIG_NUMA_BALANCING */
#endif /* CONFIG_SCHED_DEBUG */
+#ifdef CONFIG_HPERF_HMP
+static unsigned long default_fast_mask = 0x0F;
+static unsigned long default_slow_mask = 0xF0;
+
+void hmp_set_cpu_masks(struct cpumask *fast_mask, struct cpumask *slow_mask)
+{
+ cpumask_clear(fast_mask);
+ cpumask_clear(slow_mask);
+
+ /* try to parse CPU masks from config */
+ if (strlen(CONFIG_HMP_FAST_CPU_MASK) &&
+ strlen(CONFIG_HMP_SLOW_CPU_MASK)) {
+ if (cpumask_parse(CONFIG_HMP_FAST_CPU_MASK, fast_mask) ||
+ cpumask_parse(CONFIG_HMP_SLOW_CPU_MASK, slow_mask))
+ pr_err("hperf_hmp: Failed to get CPU masks from config!\n");
+ else
+ return;
+ }
+
+ pr_err("hperf_hmp: Fast mask will be: %08lX, slow mask: %08lX\n",
+ default_fast_mask, default_slow_mask);
+
+ fast_mask->bits[0] = default_fast_mask;
+ slow_mask->bits[0] = default_slow_mask;
+}
+#endif
+
__init void init_sched_fair_class(void)
{
+#ifdef CONFIG_HPERF_HMP
+ int cpu;
+#endif
#ifdef CONFIG_SMP
open_softirq(SCHED_SOFTIRQ, run_rebalance_domains);
@@ -8315,6 +8350,17 @@ __init void init_sched_fair_class(void)
zalloc_cpumask_var(&nohz.idle_cpus_mask, GFP_NOWAIT);
cpu_notifier(sched_ilb_notifier, 0);
#endif
+
+#ifdef CONFIG_HPERF_HMP
+ for_each_possible_cpu(cpu)
+ freq_scale_cpu_power[cpu] = SCHED_CAPACITY_SCALE;
+ hmp_set_cpu_masks(cpu_fastest_mask, cpu_slowest_mask);
+ pr_info("hperf_hmp: fast CPUs mask: %08X\n",
+ (unsigned int)cpumask_bits(cpu_fastest_mask)[0]);
+ pr_info("hperf_hmp: slow CPUs mask: %08X\n",
+ (unsigned int)cpumask_bits(cpu_slowest_mask)[0]);
+#endif
+
#endif /* SMP */
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 6d2a119..94828dc 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -597,6 +597,11 @@ struct rq {
*/
unsigned long nr_uninterruptible;
+#ifdef CONFIG_HPERF_HMP
+ /* shows the amount of accumulated unfairness by tasks of this rq */
+ long druntime_sum;
+ unsigned int nr_hmp_tasks;
+#endif
struct task_struct *curr, *idle, *stop;
unsigned long next_balance;
struct mm_struct *prev_mm;
@@ -892,6 +897,16 @@ static inline unsigned int group_first_cpu(struct sched_group *group)
extern int group_balance_cpu(struct sched_group *sg);
+#ifdef CONFIG_HPERF_HMP
+extern struct cpumask *cpu_fastest_mask;
+extern struct cpumask *cpu_slowest_mask;
+
+static inline bool cpu_is_fastest(int cpu)
+{
+ return cpumask_test_cpu(cpu, cpu_fastest_mask);
+}
+#endif
+
#else
static inline void sched_ttwu_pending(void) { }
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 05/13] hperf_hmp: introduce druntime metric.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (3 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 04/13] hperf_hmp: scheduler initialization routines Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 06/13] hperf_hmp: is_hmp_imbalance introduced Arseniy Krasnov
` (8 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
This patch adds special per-task metric to look for candidate for
migration between HMP domains(clusters). 'druntime' grows up when task runs on
A7 cluster, and goes down on A15 cluster. Also druntime is scaled according load
on little cluster in order to align its value with big cluster's total druntime.
For migration from big/little to little/big cluster task with lowest/highest
'druntime' chosen. 'druntime' is used to execute each task on each cluster
approximately same amount of time. 'druntime' is calculated each call of default
'update_curr' function.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
include/linux/sched.h | 3 ++
kernel/sched/core.c | 3 ++
kernel/sched/fair.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 121 insertions(+)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index aa72125..89c1bf3 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1257,6 +1257,9 @@ struct sched_entity {
struct list_head group_node;
unsigned int on_rq;
+#ifdef CONFIG_HPERF_HMP
+ long druntime;
+#endif
u64 exec_start;
u64 sum_exec_runtime;
u64 vruntime;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8747e06..6883a00 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2085,6 +2085,9 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
p->se.prev_sum_exec_runtime = 0;
p->se.nr_migrations = 0;
p->se.vruntime = 0;
+#ifdef CONFIG_HPERF_HMP
+ p->se.druntime = 0;
+#endif
INIT_LIST_HEAD(&p->se.group_node);
#ifdef CONFIG_SCHEDSTATS
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c57007f..e94fab4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -102,6 +102,10 @@ unsigned int __read_mostly sysctl_sched_shares_window = 10000000UL;
#ifdef CONFIG_HPERF_HMP
extern void hmp_set_cpu_masks(struct cpumask *, struct cpumask *);
+static atomic_t a15_nr_hmp_busy = ATOMIC_INIT(0);
+static atomic_t a7_nr_hmp_busy = ATOMIC_INIT(0);
+static atomic_t hmp_imbalance = ATOMIC_INIT(0);
+
static unsigned int freq_scale_cpu_power[CONFIG_NR_CPUS];
#endif /* CONFIG_HPERF_HMP */
@@ -660,6 +664,115 @@ static u64 sched_vslice(struct cfs_rq *cfs_rq, struct sched_entity *se)
return calc_delta_fair(sched_slice(cfs_rq, se), se);
}
+#ifdef CONFIG_HPERF_HMP
+static bool
+is_task_hmp(struct task_struct *task, const struct cpumask *task_cpus)
+{
+ if (!task_cpus)
+ task_cpus = tsk_cpus_allowed(task);
+
+ /*
+ * Check if a task has cpus_allowed only for one CPU domain (A15 or A7)
+ */
+ return !(cpumask_intersects(task_cpus, cpu_fastest_mask) ^
+ cpumask_intersects(task_cpus, cpu_slowest_mask));
+}
+
+#ifdef CONFIG_HPERF_HMP_DEBUG
+static inline void check_druntime_sum(struct rq *rq, long druntime_sum)
+{
+ BUG_ON(rq->cfs.h_nr_running == 0 && druntime_sum != 0);
+
+ if (cpu_is_fastest(rq->cpu))
+ BUG_ON(druntime_sum > 0);
+ else
+ BUG_ON(druntime_sum < 0);
+}
+#else
+static inline void check_druntime_sum(struct rq *rq, long druntime_sum)
+{
+}
+#endif
+
+static inline void add_druntime_sum(struct rq *rq, long delta)
+{
+ rq->druntime_sum += delta;
+ check_druntime_sum(rq, rq->druntime_sum);
+}
+/* Updates druntime for a task */
+static inline void
+update_hmp_stat(struct cfs_rq *cfs_rq, struct sched_entity *curr,
+ unsigned long delta_exec)
+{
+ long to_add;
+ unsigned int hmp_fairness_threshold = 240;
+ struct rq *rq = rq_of(cfs_rq);
+ int a7_nr_hmp_busy_tmp;
+
+ if (atomic_read(&hmp_imbalance) == 0)
+ return;
+
+ if (!curr->on_rq)
+ return;
+
+ if (!entity_is_task(curr))
+ return;
+
+ if (!task_of(curr)->on_rq)
+ return;
+
+ if (!cfs_rq->h_nr_running)
+ return;
+
+ if (!is_task_hmp(task_of(curr), NULL))
+ return;
+
+ delta_exec = delta_exec >> 10;
+
+ if (cpu_is_fastest(rq->cpu))
+ to_add = -delta_exec;
+ else
+ to_add = delta_exec;
+
+ to_add -= curr->druntime;
+
+ /* Avoid values with the different sign */
+ if ((cpu_is_fastest(rq->cpu) && to_add >= 0) ||
+ (!cpu_is_fastest(rq->cpu) && to_add <= 0))
+ return;
+
+ to_add /= (long)(2 + 4 * hmp_fairness_threshold /
+ (cfs_rq->h_nr_running + 1));
+
+ a7_nr_hmp_busy_tmp = atomic_read(&a7_nr_hmp_busy);
+ /* druntime balancing between the domains */
+ if (!cpu_is_fastest(rq->cpu) && a7_nr_hmp_busy_tmp) {
+ to_add *= atomic_read(&a15_nr_hmp_busy);
+ to_add /= a7_nr_hmp_busy_tmp;
+ }
+
+ if (cpu_is_fastest(rq->cpu)) {
+ if (curr->druntime < 0)
+ add_druntime_sum(rq, to_add);
+ else if ((curr->druntime + to_add) < 0)
+ add_druntime_sum(rq, curr->druntime + to_add);
+ } else {
+ if (curr->druntime > 0)
+ add_druntime_sum(rq, to_add);
+ else if ((curr->druntime + to_add) > 0)
+ add_druntime_sum(rq, curr->druntime + to_add);
+ }
+
+ curr->druntime += to_add;
+}
+#else
+static inline void
+update_hmp_stat(struct cfs_rq *cfs_rq, struct sched_entity *curr,
+ unsigned long delta_exec)
+{
+}
+#endif /* CONFIG_HPERF_HMP */
+
#ifdef CONFIG_SMP
static int select_idle_sibling(struct task_struct *p, int cpu);
static unsigned long task_h_load(struct task_struct *p);
@@ -735,6 +848,8 @@ static void update_curr(struct cfs_rq *cfs_rq)
}
account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+ update_hmp_stat(cfs_rq, curr, delta_exec);
}
static void update_curr_fair(struct rq *rq)
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 06/13] hperf_hmp: is_hmp_imbalance introduced.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (4 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 05/13] hperf_hmp: introduce druntime metric Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 07/13] hperf_hmp: migration auxiliary functions Arseniy Krasnov
` (7 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
'is_hmp_imbalance' function calculates imbalance between clusters, four
cases are possible: balancing from/to one of clusters, task swap(when clusters
are balanced) or skip rebalance. Function calculates load difference between two
cluster(cluster load / cluster power) and threshold when balancing is needed.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
kernel/sched/fair.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 103 insertions(+)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e94fab4..3ab39b6 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -104,9 +104,21 @@ unsigned int __read_mostly sysctl_sched_shares_window = 10000000UL;
extern void hmp_set_cpu_masks(struct cpumask *, struct cpumask *);
static atomic_t a15_nr_hmp_busy = ATOMIC_INIT(0);
static atomic_t a7_nr_hmp_busy = ATOMIC_INIT(0);
+
+/* Total weight of all running tasks on A15 and A7 CPU domains */
+static atomic_long_t a15_total_weight = ATOMIC_LONG_INIT(0);
+static atomic_long_t a7_total_weight = ATOMIC_LONG_INIT(0);
+
static atomic_t hmp_imbalance = ATOMIC_INIT(0);
static unsigned int freq_scale_cpu_power[CONFIG_NR_CPUS];
+
+enum hmp_balance_actions {
+ SWAP_TASKS,
+ A15_TO_A7,
+ A7_TO_A15,
+ SKIP_REBALANCE,
+};
#endif /* CONFIG_HPERF_HMP */
#ifdef CONFIG_CFS_BANDWIDTH
@@ -7016,6 +7028,97 @@ static int should_we_balance(struct lb_env *env)
*/
return balance_cpu == env->dst_cpu;
}
+#ifdef CONFIG_HPERF_HMP
+/**
+ * is_hmp_imbalance(): Calculates imbalance between HMP domains.
+ * @sd: Current sched domain.
+ *
+ * Returns migration direction(see SWAP_TASKS, A15_TO_A7, A7_TO_A15,
+ * SKIP_REBALANCE).
+ *
+ * Imbalance depends on load of tasks on A15 cores and A7 cores,
+ * current CPU's frequencies, and A7 slowdown coefficient which is about 2.4.
+ */
+static int is_hmp_imbalance(struct sched_domain *sd)
+{
+ int imbalance, cpu;
+ int a15_group_power = 0, a7_group_power = 0,
+ hmp_imbalance_min_threshold;
+ int a15_group_load, a7_group_load, a15_a7_group_power;
+ unsigned int a7_balanced_num;
+ int reminder, divisor;
+ unsigned int a15_balanced_num;
+ long long int hmp_imbalance_threshold;
+
+ if (!sd->a15_group) {
+ return SKIP_REBALANCE;
+ }
+
+ if (!sd->a7_group) {
+ return SKIP_REBALANCE;
+ }
+ for_each_online_cpu(cpu) {
+ if (cpu_is_fastest(cpu))
+ a15_group_power += freq_scale_cpu_power[cpu];
+ else
+ a7_group_power += freq_scale_cpu_power[cpu];
+ }
+
+ if (a15_group_power == 0 || a7_group_power == 0) {
+ return SKIP_REBALANCE;
+ }
+
+ a15_balanced_num = 0;
+ a7_balanced_num = 0;
+
+ for_each_online_cpu(cpu) {
+ if (cpu_rq(cpu)->cfs.h_nr_running <= 1) {
+ if (cpu_is_fastest(cpu))
+ a15_balanced_num++;
+ else
+ a7_balanced_num++;
+ }
+ }
+
+ a7_group_load = atomic_long_read(&a7_total_weight);
+
+ if (atomic_long_read(&a7_total_weight) == 0 &&
+ (a15_balanced_num == sd->a15_group->group_weight)) {
+ return SKIP_REBALANCE;
+ }
+
+ a15_group_load = atomic_long_read(&a15_total_weight);
+ a15_a7_group_power = a15_group_power + a7_group_power;
+
+ imbalance = (a15_group_load * 1024) / (a15_group_power) -
+ (a7_group_load * 1024) / (a7_group_power);
+ hmp_imbalance_threshold = ((long long int)NICE_0_LOAD *
+ 1024 * a15_a7_group_power);
+ divisor = 2 * a15_group_power * a7_group_power;
+ hmp_imbalance_threshold = div_s64_rem(hmp_imbalance_threshold,
+ divisor, &reminder);
+ hmp_imbalance_min_threshold = hmp_imbalance_threshold >> 3;
+
+ if (imbalance < hmp_imbalance_min_threshold &&
+ imbalance > -hmp_imbalance_min_threshold) {
+ atomic_set(&hmp_imbalance, 0);
+ return SKIP_REBALANCE;
+ }
+
+ if (imbalance > hmp_imbalance_threshold) {
+ return A15_TO_A7;
+ } else {
+ if (imbalance < -hmp_imbalance_threshold) {
+ if (a7_balanced_num == sd->a7_group->group_weight)
+ return SWAP_TASKS;
+ else
+ return A7_TO_A15;
+ } else {
+ return SWAP_TASKS;
+ }
+ }
+}
+#endif /* CONFIG_HPERF_HMP */
/*
* Check this_cpu to ensure it is balanced within domain. Attempt to move
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 07/13] hperf_hmp: migration auxiliary functions.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (5 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 06/13] hperf_hmp: is_hmp_imbalance introduced Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 13:03 ` kbuild test robot
2015-11-06 12:02 ` [PATCH 08/13] hperf_hmp: swap tasks function Arseniy Krasnov
` (6 subsequent siblings)
13 siblings, 1 reply; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
Adds functions used for migration: scanning every runqueue from another
cluster for migration process, searching task to migrate from runqueue mentioned
above and function to move task from one CPU to another.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
include/linux/sched.h | 6 +
kernel/sched/fair.c | 301 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 307 insertions(+)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 89c1bf3..dafda4b 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1259,6 +1259,12 @@ struct sched_entity {
#ifdef CONFIG_HPERF_HMP
long druntime;
+
+ /* Time of last migration between HMP domains (in jiffies)*/
+ unsigned long last_migration;
+
+ /* If set, don't touch for migration */
+ int migrate_candidate;
#endif
u64 exec_start;
u64 sum_exec_runtime;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3ab39b6..ff05364 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7118,6 +7118,307 @@ static int is_hmp_imbalance(struct sched_domain *sd)
}
}
}
+
+/**
+ * hmp_can_migrate_task(): Checks whether specified task could be migrated.
+ * @p: task to check.
+ * @env: migration parameters.
+ *
+ * Returns 1 if migration possible, else 0.
+ */
+static int hmp_can_migrate_task(struct task_struct *p, struct lb_env *env)
+{
+ if (!cpumask_test_cpu(env->dst_cpu, tsk_cpus_allowed(p))) {
+ schedstat_inc(p, se.statistics.nr_failed_migrations_affine);
+ return 0;
+ }
+ env->flags &= ~LBF_ALL_PINNED;
+
+ if (task_running(env->src_rq, p)) {
+ schedstat_inc(p, se.statistics.nr_failed_migrations_running);
+ return 0;
+ }
+ return 1;
+}
+
+/**
+ * detach_specified_task(): Detaches specified task.
+ * @pm: Task to move.
+ * @env: Migration parameters.
+ *
+ * Returns moved task.
+ */
+static struct task_struct *
+detach_specified_task(struct task_struct *p, struct lb_env *env)
+{
+ lockdep_assert_held(&env->src_rq->lock);
+
+ /* If task to move falls asleep, so don't scan runqueue and return */
+ if (p->se.migrate_candidate == 0)
+ return 0;
+
+ if (throttled_lb_pair(task_group(p), env->src_rq->cpu, env->dst_cpu))
+ goto exit;
+
+ if (!hmp_can_migrate_task(p, env))
+ goto exit;
+
+ detach_task(p, env);
+ /*
+ * Right now, this is only the third place move_task()
+ * is called, so we can safely collect move_task()
+ * stats here rather than inside move_task().
+ */
+ schedstat_inc(env->sd, lb_gained[env->idle]);
+ return p;
+exit:
+ p->se.migrate_candidate = 0;
+
+ return NULL;
+}
+
+/**
+ * migrate_runnable_task(): Moves task that isn't running to destination CPU.
+ * @migrate_task: Task to migrate.
+ * @destination_cpu: Destination CPU.
+ *
+ * Returns moved weight.
+ *
+ * Runqueue's of @migrate_task and @destination_cpu must be locked.
+ */
+static unsigned migrate_runnable_task(struct task_struct *migrate_task,
+ int destination_cpu)
+{
+ struct sched_domain *sd = NULL;
+ int src_cpu = task_cpu(migrate_task);
+ struct rq *src_rq = task_rq(migrate_task);
+ int dst_cpu = destination_cpu;
+ struct rq *dst_rq = cpu_rq(dst_cpu);
+ unsigned int ld_moved = 0;
+ struct task_struct *p = NULL;
+
+#ifdef CONFIG_HPERF_HMP_DEBUG
+ BUG_ON(src_rq == dst_rq);
+#else
+ if (WARN_ON(src_rq == dst_rq))
+ return 0;
+#endif
+
+ rcu_read_lock();
+ for_each_domain(dst_cpu, sd) {
+ if (cpumask_test_cpu(src_cpu, sched_domain_span(sd)))
+ break;
+ }
+ if (likely(sd)) {
+ struct lb_env env = {
+ .sd = sd,
+ .dst_cpu = dst_cpu,
+ .dst_rq = dst_rq,
+ .src_cpu = src_cpu,
+ .src_rq = src_rq,
+ .idle = CPU_NOT_IDLE,
+ };
+
+ schedstat_inc(sd, alb_count);
+ p = detach_specified_task(migrate_task, &env);
+ if (p) {
+ migrate_task->se.last_migration = jiffies;
+ schedstat_inc(sd, alb_pushed);
+ ld_moved = migrate_task->se.load.weight;
+ } else
+ schedstat_inc(sd, alb_failed);
+ }
+ rcu_read_unlock();
+
+ if (p)
+ attach_task(dst_rq, p);
+
+ if (migrate_task->se.migrate_candidate)
+ migrate_task->se.migrate_candidate = 0;
+ return ld_moved;
+}
+
+/* A task can't be migrated more often than 4 ms between A7 and A15 CPUs */
+static int se_is_old(struct sched_entity *se)
+{
+ const unsigned int migration_delay = 4; /* ms */
+
+ return time_after(jiffies,
+ se->last_migration + msecs_to_jiffies(migration_delay));
+}
+
+/**
+ * get_opposite_group(): Gets A15 of A7 group of domain.
+ * @sd: Current sched domain.
+ * @domain: Flag, which group is needed.
+ *
+ * Returns pointer to sched group.
+ */
+static struct sched_group *get_opposite_group(struct sched_domain *sd,
+ int domain)
+{
+ if (!domain)
+ return sd->a15_group;
+ else
+ return sd->a7_group;
+}
+
+/**
+ * get_unfair_rq(): Returns runqueue which most fits for HMP migration.
+ * @sd: Current sched_domain.
+ * @this_cpu: without NO_HZ same as smp_processor_id().
+ *
+ * Returns struct rq*.
+ *
+ * Returned runqueue will be locked.
+ */
+static struct rq *get_unfair_rq(struct sched_domain *sd, int this_cpu)
+{
+ struct rq *unfair_rq = NULL;
+ struct sched_group *opposite_sg;
+ struct cpumask *opposite_mask;
+ int druntime;
+ int cpu;
+
+ opposite_sg = get_opposite_group(sd, cpu_is_fastest(this_cpu));
+
+ if (!opposite_sg)
+ return NULL;
+
+ opposite_mask = sched_group_cpus(opposite_sg);
+ druntime = cpu_is_fastest(this_cpu) ? INT_MIN : INT_MAX;
+
+ /* Check rq's of opposite domain */
+ for_each_cpu_and(cpu, opposite_mask, cpu_online_mask) {
+ struct rq *rq = cpu_rq(cpu);
+ long tmp_druntime;
+
+ /*
+ * Note: the value is read without a spinlock and can be
+ * outdated. But it is fine in the long run.
+ */
+ tmp_druntime = rq->druntime_sum;
+
+ /* Skip empty rqs or rqs waiting for stopper */
+ if (rq->active_balance || !rq->cfs.h_nr_running)
+ continue;
+
+ if (cpu_is_fastest(cpu)) {
+ if (tmp_druntime < druntime) {
+ druntime = tmp_druntime;
+ unfair_rq = rq;
+ }
+ } else {
+ if (tmp_druntime > druntime) {
+ druntime = tmp_druntime;
+ unfair_rq = rq;
+ }
+ }
+ }
+
+ if (unfair_rq) {
+ raw_spin_lock(&unfair_rq->lock);
+ if (!unfair_rq->cfs.h_nr_running || unfair_rq->active_balance) {
+ raw_spin_unlock(&unfair_rq->lock);
+ return NULL;
+ }
+ }
+
+ return unfair_rq;
+}
+
+/**
+ * get_migration_candidate(): Get task which most fits for HMP migration.
+ * @sd: Current sched domain.
+ * @unfair_rq: Runqueue to scan for migration task.
+ * @idle_flag: Determines unfair_rq is idle for not. If 1, then ignore task's
+ * @destination_cpu: Destination CPU for task from @unfair_rq
+ * druntime and last migration time.
+ *
+ * Returns struct task_struct*.
+ *
+ * @unfair_rq must be locked. @sd must have SD_HMP_BALANCE flag.
+ */
+static struct task_struct *get_migration_candidate(struct sched_domain *sd,
+ struct rq *unfair_rq,
+ int idle_flag,
+ int destination_cpu)
+{
+ long druntime;
+ struct task_struct *p;
+ struct list_head *tasks;
+ struct task_struct *candidate = NULL;
+ unsigned int count = sched_nr_migrate_break;
+
+ if (unfair_rq->cfs.h_nr_running < count)
+ count = unfair_rq->cfs.h_nr_running;
+
+ tasks = &unfair_rq->cfs_tasks;
+ druntime = cpu_is_fastest(unfair_rq->cpu) ? LONG_MAX : LONG_MIN;
+
+ while (!list_empty(tasks)) {
+ p = list_first_entry(tasks, struct task_struct, se.group_node);
+
+ if (!count)
+ break;
+
+ count--;
+ /* this task pinned by someone else for HMP migration */
+ if (p->se.migrate_candidate)
+ goto next;
+
+ /* if task can't run on destination cpu, skip */
+ if (!cpumask_test_cpu(destination_cpu, tsk_cpus_allowed(p)))
+ goto next;
+
+ /* check for 4ms timestamp, if idle_pull then don't care*/
+ if (!se_is_old(&p->se) && !idle_flag)
+ goto next;
+
+ if (cpu_is_fastest(unfair_rq->cpu)) {
+ if (p->se.druntime < druntime &&
+ (p->se.druntime < 0 || idle_flag)) {
+ candidate = p;
+ druntime = p->se.druntime;
+ }
+ } else {
+ if (p->se.druntime > druntime &&
+ (p->se.druntime > 0 || idle_flag)) {
+ candidate = p;
+ druntime = p->se.druntime;
+ }
+ }
+
+next:
+ list_move_tail(&p->se.group_node, tasks);
+ }
+
+ if (candidate)
+ candidate->se.migrate_candidate = 1;
+
+ return candidate;
+}
+
+/**
+ * try_to_move_task(): Migrates task if it isn't running.
+ * @migrate_task: Task to migrate.
+ * @destination_cpu: Destination cpu for @migrate_task.
+ * @stopper_needed: Flag which show that stopper thread needed to migrate task.
+ *
+ * Returns moved weight and flag that stopper needed or not.
+ *
+ * Runqueues of @migrate_task and @destination_cpu must be locked.
+ */
+static unsigned int try_to_move_task(struct task_struct *migrate_task,
+ int destination_cpu, int *stopper_needed)
+{
+ if (task_running(task_rq(migrate_task), migrate_task)) {
+ *stopper_needed = 1;
+ return migrate_task->se.load.weight;
+ }
+
+ return migrate_runnable_task(migrate_task, destination_cpu);
+}
#endif /* CONFIG_HPERF_HMP */
/*
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 08/13] hperf_hmp: swap tasks function.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (6 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 07/13] hperf_hmp: migration auxiliary functions Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 09/13] hperf_hmp: one way balancing function Arseniy Krasnov
` (5 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
'swap_tasks' performs migration between current CPU and CPU from another
cluster. It scans two runqueues looking for tasks using 'druntime' metric. When
both tasks are found it pulls task from another cluster, and push task from the
current CPU.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
kernel/sched/fair.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++++++
kernel/sched/sched.h | 1 +
2 files changed, 101 insertions(+)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ff05364..028d329 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7419,6 +7419,106 @@ static unsigned int try_to_move_task(struct task_struct *migrate_task,
return migrate_runnable_task(migrate_task, destination_cpu);
}
+
+/**
+ * swap_tasks(): swaps two tasks from different HMP domains
+ * @sd: Current sched domain
+ * @this_cpu: without NO_HZ same as smp_processor_id().
+ *
+ * Returns weight of migrated tasks.
+ */
+static unsigned int swap_tasks(struct sched_domain *sd, int this_cpu)
+{
+ unsigned int ld_moved = 0;
+ int local_stopper = 0;
+ int foreign_stopper = 0;
+ struct rq *local_rq = cpu_rq(this_cpu);
+ struct rq *foreign_rq = NULL;
+ struct task_struct *local_task = NULL;
+ struct task_struct *foreign_task = NULL;
+ unsigned long local_flags;
+
+ local_irq_save(local_flags);
+ foreign_rq = get_unfair_rq(sd, this_cpu);
+
+ if (!foreign_rq) {
+ local_irq_restore(local_flags);
+ return 0;
+ }
+
+ double_lock_balance(foreign_rq, local_rq);
+
+ /* rq's waiting for stopper execution, return */
+ if (foreign_rq->active_balance)
+ goto unlock;
+
+ if (local_rq->active_balance)
+ goto unlock;
+
+ foreign_task = get_migration_candidate(sd, foreign_rq, 0, this_cpu);
+
+ if (!foreign_task)
+ goto unlock;
+
+ /* Get local task for migration */
+ local_task = get_migration_candidate(sd, local_rq, 0, foreign_rq->cpu);
+
+ if (!local_task) {
+ foreign_task->se.migrate_candidate = 0;
+ goto unlock;
+ }
+ /* First try to push local task */
+ ld_moved = try_to_move_task(local_task, foreign_rq->cpu,
+ &local_stopper);
+
+ /* If failed to move, then return, don't try to move foreign task */
+ if (!ld_moved) {
+ local_task->se.migrate_candidate = 0;
+ foreign_task->se.migrate_candidate = 0;
+ goto unlock;
+ }
+
+ /*
+ * Migration is possible, but task is running,
+ * so mark rq to run stopper.
+ */
+ if (local_stopper) {
+ local_rq->push_cpu = foreign_rq->cpu;
+ local_rq->migrate_task = local_task;
+ local_rq->active_balance = 1;
+ }
+
+ /* Now try to pull task from another cpu */
+ ld_moved = try_to_move_task(foreign_task, this_cpu,
+ &foreign_stopper);
+
+ /* Failed to move foreign_task */
+ if (!ld_moved)
+ foreign_task->se.migrate_candidate = 0;
+
+ /* Migration is possible, mark rq to run stopper */
+ if (foreign_stopper) {
+ foreign_rq->push_cpu = this_cpu;
+ foreign_rq->migrate_task = foreign_task;
+ foreign_rq->active_balance = 1;
+ }
+
+unlock:
+ double_rq_unlock(local_rq, foreign_rq);
+ local_irq_restore(local_flags);
+
+ if (local_stopper)
+ stop_one_cpu_nowait(local_rq->cpu,
+ active_load_balance_cpu_stop, local_rq,
+ &local_rq->active_balance_work);
+
+ if (foreign_stopper)
+ stop_one_cpu_nowait(foreign_rq->cpu,
+ active_load_balance_cpu_stop, foreign_rq,
+ &foreign_rq->active_balance_work);
+
+ return ld_moved;
+}
#endif /* CONFIG_HPERF_HMP */
/*
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 94828dc..47e9605 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -598,6 +598,7 @@ struct rq {
unsigned long nr_uninterruptible;
#ifdef CONFIG_HPERF_HMP
+ struct task_struct *migrate_task; /* task from this rq for migration */
/* shows the amount of accumulated unfairness by tasks of this rq */
long druntime_sum;
unsigned int nr_hmp_tasks;
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 09/13] hperf_hmp: one way balancing function.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (7 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 08/13] hperf_hmp: swap tasks function Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 10/13] hperf_hmp: idle pull function Arseniy Krasnov
` (4 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
Almost identical functions which push/pull task from/to current CPU
to/from another cluster. Called when balancing between clusters is broken and we
need to fix it.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
kernel/sched/fair.c | 254 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 254 insertions(+)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 028d329..4fda1ec 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7519,6 +7519,260 @@ unlock:
return ld_moved;
}
+
+/* Get idlest cpu from opposite domain of this_cpu */
+static int get_idlest_cpu(struct sched_domain *sd, int this_cpu)
+{
+ struct sched_group *opposite_sg;
+ struct cpumask *opposite_mask;
+ unsigned long load = ULONG_MAX;
+ int idlest_cpu = -1;
+ int cpu;
+
+ opposite_sg = get_opposite_group(sd, cpu_is_fastest(this_cpu));
+ opposite_mask = sched_group_cpus(opposite_sg);
+
+ for_each_cpu_and(cpu, opposite_mask, cpu_online_mask) {
+ if (cpu_rq(cpu)->load.weight < load) {
+ load = cpu_rq(cpu)->load.weight;
+ idlest_cpu = cpu;
+ }
+ }
+ return idlest_cpu;
+}
+
+/**
+ * move_a15_to_a7(): Moves one task from A15 to A7.
+ * @sd: Current sched domain.
+ * @this_cpu: without NO_HZ same as smp_processor_id().
+ *
+ * Returns moved weight.
+ *
+ * Chooses task to migrate by druntime.
+ */
+static unsigned int move_a15_to_a7(struct sched_domain *sd, int this_cpu)
+{
+ struct task_struct *task_to_move;
+ struct rq *local_rq = NULL;
+ struct rq *foreign_rq = NULL;
+ int local_stopper_flag = 0;
+ int foreign_stopper_flag = 0;
+ unsigned long local_flags;
+ unsigned int ld_moved = 0;
+
+ local_rq = cpu_rq(this_cpu);
+ local_irq_save(local_flags);
+
+ if (!cpu_is_fastest(this_cpu)) {
+ /* this A7 pulls task from A15 */
+ foreign_rq = get_unfair_rq(sd, this_cpu);
+
+ if (!foreign_rq) {
+ local_irq_restore(local_flags);
+ return 0;
+ }
+
+ double_lock_balance(foreign_rq, local_rq);
+
+ if (foreign_rq->active_balance)
+ goto unlock;
+
+ if (local_rq->active_balance)
+ goto unlock;
+
+ if (foreign_rq->cfs.h_nr_running <= 1)
+ goto unlock;
+
+ task_to_move = get_migration_candidate(sd, foreign_rq, 0,
+ this_cpu);
+
+ if (!task_to_move)
+ goto unlock;
+
+ ld_moved = try_to_move_task(task_to_move, this_cpu,
+ &foreign_stopper_flag);
+
+ if (!ld_moved) {
+ task_to_move->se.migrate_candidate = 0;
+ goto unlock;
+ }
+
+ if (foreign_stopper_flag) {
+ foreign_rq->active_balance = 1;
+ foreign_rq->push_cpu = this_cpu;
+ foreign_rq->migrate_task = task_to_move;
+ }
+ } else {
+ /* this A15 push task to A7 */
+ int dst_cpu = get_idlest_cpu(sd, this_cpu);
+
+ if (dst_cpu == -1) {
+ local_irq_restore(local_flags);
+ return 0;
+ }
+
+ foreign_rq = cpu_rq(dst_cpu);
+ raw_spin_lock(&foreign_rq->lock);
+ double_lock_balance(foreign_rq, local_rq);
+
+ if (local_rq->cfs.h_nr_running <= 1)
+ goto unlock;
+
+ if (foreign_rq->active_balance)
+ goto unlock;
+
+ if (local_rq->active_balance)
+ goto unlock;
+
+ task_to_move = get_migration_candidate(sd, local_rq, 0,
+ foreign_rq->cpu);
+
+ if (!task_to_move)
+ goto unlock;
+
+ ld_moved = try_to_move_task(task_to_move, dst_cpu,
+ &local_stopper_flag);
+
+ if (!ld_moved) {
+ task_to_move->se.migrate_candidate = 0;
+ goto unlock;
+ }
+
+ if (local_stopper_flag) {
+ local_rq->active_balance = 1;
+ local_rq->push_cpu = dst_cpu;
+ local_rq->migrate_task = task_to_move;
+ }
+ }
+unlock:
+ double_rq_unlock(local_rq, foreign_rq);
+ local_irq_restore(local_flags);
+
+ if (foreign_stopper_flag)
+ stop_one_cpu_nowait(foreign_rq->cpu,
+ active_load_balance_cpu_stop, foreign_rq,
+ &foreign_rq->active_balance_work);
+
+ if (local_stopper_flag)
+ stop_one_cpu_nowait(local_rq->cpu,
+ active_load_balance_cpu_stop, local_rq,
+ &local_rq->active_balance_work);
+
+ return ld_moved;
+}
+
+/**
+ * move_a7_to_a15(): Moves one task from A7 to A15.
+ * @sd: Current sched domain.
+ * @this_cpu: without NO_HZ same as smp_processor_id().
+ *
+ * Returns moved weight.
+ *
+ * Chooses task to migrate by druntime.
+ */
+static unsigned int move_a7_to_a15(struct sched_domain *sd, int this_cpu)
+{
+ struct task_struct *task_to_move;
+ struct rq *local_rq = NULL;
+ struct rq *foreign_rq = NULL;
+ int local_stopper_flag = 0;
+ int foreign_stopper_flag = 0;
+ unsigned long local_flags;
+ unsigned int ld_moved = 0;
+
+ local_rq = cpu_rq(this_cpu);
+ local_irq_save(local_flags);
+
+ if (cpu_is_fastest(this_cpu)) {
+ /* this A15 pulls task from A7 */
+ foreign_rq = get_unfair_rq(sd, this_cpu);
+
+ if (!foreign_rq) {
+ local_irq_restore(local_flags);
+ return 0;
+ }
+ double_lock_balance(foreign_rq, local_rq);
+
+ if (local_rq->active_balance)
+ goto unlock;
+
+ if (foreign_rq->active_balance)
+ goto unlock;
+
+ task_to_move = get_migration_candidate(sd, foreign_rq, 0,
+ this_cpu);
+
+ if (!task_to_move)
+ goto unlock;
+
+ ld_moved = try_to_move_task(task_to_move, this_cpu,
+ &foreign_stopper_flag);
+
+ if (!ld_moved) {
+ task_to_move->se.migrate_candidate = 0;
+ goto unlock;
+ }
+
+ if (foreign_stopper_flag) {
+ foreign_rq->active_balance = 1;
+ foreign_rq->push_cpu = this_cpu;
+ foreign_rq->migrate_task = task_to_move;
+ }
+ } else {
+ /* this A7 push task to A15*/
+ int dst_cpu = get_idlest_cpu(sd, this_cpu);
+
+ if (dst_cpu == -1) {
+ local_irq_restore(local_flags);
+ return 0;
+ }
+
+ foreign_rq = cpu_rq(dst_cpu);
+ raw_spin_lock(&foreign_rq->lock);
+ double_lock_balance(foreign_rq, local_rq);
+
+ if (foreign_rq->active_balance)
+ goto unlock;
+
+ if (local_rq->active_balance)
+ goto unlock;
+
+ task_to_move = get_migration_candidate(sd, local_rq, 0,
+ foreign_rq->cpu);
+
+ if (!task_to_move)
+ goto unlock;
+
+ ld_moved = try_to_move_task(task_to_move, dst_cpu,
+ &local_stopper_flag);
+
+ if (!ld_moved) {
+ task_to_move->se.migrate_candidate = 0;
+ goto unlock;
+ }
+
+ if (local_stopper_flag) {
+ local_rq->active_balance = 1;
+ local_rq->push_cpu = dst_cpu;
+ local_rq->migrate_task = task_to_move;
+ }
+ }
+unlock:
+ double_rq_unlock(local_rq, foreign_rq);
+ local_irq_restore(local_flags);
+
+ if (foreign_stopper_flag)
+ stop_one_cpu_nowait(foreign_rq->cpu,
+ active_load_balance_cpu_stop, foreign_rq,
+ &foreign_rq->active_balance_work);
+
+ if (local_stopper_flag)
+ stop_one_cpu_nowait(local_rq->cpu,
+ active_load_balance_cpu_stop, local_rq,
+ &local_rq->active_balance_work);
+
+ return ld_moved;
+}
#endif /* CONFIG_HPERF_HMP */
/*
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 10/13] hperf_hmp: idle pull function.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (8 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 09/13] hperf_hmp: one way balancing function Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 11/13] hperf_hmp: task CPU selection logic Arseniy Krasnov
` (3 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
HMP idle pull is triggered when CPU becomes idle. It tries to pull task
from another cluster when it is overloaded. Also A7 can't pull alone task from
A15, but A15 can do that with A7 core. Task for migration is chosen in the same
way as for other HMP migration cases - using 'druntime' metric. Only difference
is that migration task doesn't need to run 5ms on its cluster before migration.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
kernel/sched/fair.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 4fda1ec..fd16729 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7421,6 +7421,72 @@ static unsigned int try_to_move_task(struct task_struct *migrate_task,
}
/**
+ * hmp_idle_pull(): Pulls task from opposite domain of this_cpu to this_cpu.
+ * @sd: Current sched domain.
+ * @this_cpu: without NO_HZ same as smp_processor_id().
+ *
+ * Returns moved weight.
+ *
+ * Chooses task by its druntime. Ignores task's druntime and
+ * time of last HMP migration. Also A7 can't pulls task from A15
+ * if A15 become idle.
+ */
+static unsigned int hmp_idle_pull(struct sched_domain *sd, int this_cpu)
+{
+ unsigned int ld_moved = 0;
+ struct task_struct *task_to_pull;
+ unsigned long local_flags;
+ int idle_stopper = 0;
+ struct rq *local_rq;
+ struct rq *rq;
+
+ local_irq_save(local_flags);
+ local_rq = cpu_rq(this_cpu);
+ rq = get_unfair_rq(sd, this_cpu);
+
+ if (!rq) {
+ local_irq_restore(local_flags);
+ return 0;
+ }
+ double_lock_balance(rq, local_rq);
+
+ if (rq->active_balance)
+ goto unlock;
+
+ if (local_rq->active_balance)
+ goto unlock;
+
+ /* Forbids secondary CPUs to pull alone task from primary CPUs */
+ if (!cpu_is_fastest(this_cpu) && rq->cfs.h_nr_running <= 1)
+ goto unlock;
+
+ /* Get task to pull from opposite domain to this_cpu */
+ task_to_pull = get_migration_candidate(sd, rq, 1, this_cpu);
+
+ if (!task_to_pull)
+ goto unlock;
+
+ ld_moved = try_to_move_task(task_to_pull, this_cpu, &idle_stopper);
+
+ if (idle_stopper) {
+ rq->push_cpu = this_cpu;
+ rq->active_balance = 1;
+ rq->migrate_task = task_to_pull;
+ }
+
+unlock:
+ double_rq_unlock(local_rq, rq);
+ local_irq_restore(local_flags);
+
+ if (idle_stopper)
+ stop_one_cpu_nowait(rq->cpu, active_load_balance_cpu_stop,
+ rq, &rq->active_balance_work);
+
+ return ld_moved;
+}
+
+
+/**
* swap_tasks(): swaps two tasks from different HMP domains
* @sd: Current sched domain
* @this_cpu: without NO_HZ same as smp_processor_id().
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 11/13] hperf_hmp: task CPU selection logic.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (9 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 10/13] hperf_hmp: idle pull function Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:29 ` kbuild test robot
2015-11-06 12:02 ` [PATCH 12/13] hperf_hmp: rest of logic Arseniy Krasnov
` (2 subsequent siblings)
13 siblings, 1 reply; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
Adds new runqueue selection logic. If task is newly woken(fork or exec)
or it is not WF_SYNC wakeup, idlest CPU from both clusters is selected. Else,
default wake up logic is used('want_affine'). If it fails, idlest CPU from both
clusters is selected.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
kernel/sched/fair.c | 132 ++++++++++++++++++++++++++++++++++++++++------------
1 file changed, 101 insertions(+), 31 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index fd16729..79be023 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4798,6 +4798,62 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)
return 1;
}
+#ifdef CONFIG_HPERF_HMP
+/**
+ * hmp_select_task_rq_fair(): selects cpu for task.
+ * @p: task which needs cpu
+ *
+ * Returns cpu for task.
+ *
+ * Selects idlest cpu for task @p.
+ */
+static int
+hmp_select_task_rq_fair(struct task_struct *p)
+{
+ int cpu;
+ int new_cpu;
+ unsigned long load;
+ unsigned long scaled_load;
+
+ new_cpu = task_cpu(p);
+
+ load = ULONG_MAX;
+ /* First check primary cpus */
+ for_each_cpu_and(cpu, cpu_online_mask, cpu_fastest_mask) {
+ if (cpumask_test_cpu(cpu, tsk_cpus_allowed(p))) {
+ /* Select idle cpu if it exists */
+ if (idle_cpu(cpu))
+ return cpu;
+ /* Otherwise select the least loaded cpu */
+ scaled_load = (weighted_cpuload(cpu) *
+ SCHED_CAPACITY_SCALE) /
+ freq_scale_cpu_power[cpu];
+ if (scaled_load < load) {
+ new_cpu = cpu;
+ load = scaled_load;
+ }
+ }
+ }
+
+ /* Then check secondary cpus */
+ for_each_cpu_and(cpu, cpu_online_mask, cpu_slowest_mask) {
+ if (cpumask_test_cpu(cpu, tsk_cpus_allowed(p))) {
+ if (idle_cpu(cpu))
+ return cpu;
+ scaled_load = (weighted_cpuload(cpu) *
+ SCHED_CAPACITY_SCALE) /
+ freq_scale_cpu_power[cpu];
+ if (scaled_load < load) {
+ new_cpu = cpu;
+ load = scaled_load;
+ }
+ }
+ }
+
+ return new_cpu;
+}
+
+#else /* CONFIG_HPERF_HMP */
/*
* find_idlest_group finds and returns the least busy CPU group within the
* domain.
@@ -4905,6 +4961,7 @@ find_idlest_cpu(struct sched_group *group, struct task_struct *p, int this_cpu)
return shallowest_idle_cpu != -1 ? shallowest_idle_cpu : least_loaded_cpu;
}
+#endif /* CONFIG_HPERF_HMP */
/*
* Try and locate an idle CPU in the sched_domain.
*/
@@ -4998,6 +5055,11 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int sd_flag, int wake_f
int want_affine = 0;
int sync = wake_flags & WF_SYNC;
+#ifdef CONFIG_HPERF_HMP
+ if (!(sd_flag & SD_BALANCE_WAKE) || !sync)
+ return hmp_select_task_rq_fair(p);
+#endif
+
if (sd_flag & SD_BALANCE_WAKE)
want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, tsk_cpus_allowed(p));
@@ -5030,41 +5092,49 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int sd_flag, int wake_f
if (!sd) {
if (sd_flag & SD_BALANCE_WAKE) /* XXX always ? */
- new_cpu = select_idle_sibling(p, new_cpu);
-
- } else while (sd) {
- struct sched_group *group;
- int weight;
+ if (IS_ENABLED(CONFIG_HPERF_HMP) && sync)
+ new_cpu = prev_cpu;
+ else
+ new_cpu = select_idle_sibling(p, prev_cpu);
+ } else {
+#ifdef CONFIG_HPERF_HMP
+ new_cpu = hmp_select_task_rq_fair(p);
+#else
+ while (sd) {
+ struct sched_group *group;
+ int weight;
- if (!(sd->flags & sd_flag)) {
- sd = sd->child;
- continue;
- }
+ if (!(sd->flags & sd_flag)) {
+ sd = sd->child;
+ continue;
+ }
- group = find_idlest_group(sd, p, cpu, sd_flag);
- if (!group) {
- sd = sd->child;
- continue;
- }
+ group = find_idlest_group(sd, p, cpu, sd_flag);
+ if (!group) {
+ sd = sd->child;
+ continue;
+ }
- new_cpu = find_idlest_cpu(group, p, cpu);
- if (new_cpu == -1 || new_cpu == cpu) {
- /* Now try balancing at a lower domain level of cpu */
- sd = sd->child;
- continue;
- }
+ new_cpu = find_idlest_cpu(group, p, cpu);
+ if (new_cpu == -1 || new_cpu == cpu) {
+ /* Now try balancing at a lower domain level of cpu */
+ sd = sd->child;
+ continue;
+ }
- /* Now try balancing at a lower domain level of new_cpu */
- cpu = new_cpu;
- weight = sd->span_weight;
- sd = NULL;
- for_each_domain(cpu, tmp) {
- if (weight <= tmp->span_weight)
- break;
- if (tmp->flags & sd_flag)
- sd = tmp;
- }
- /* while loop will break here if sd == NULL */
+ /* Now try balancing at a lower domain level of new_cpu */
+ cpu = new_cpu;
+ weight = sd->span_weight;
+ sd = NULL;
+ for_each_domain(cpu, tmp) {
+ if (weight <= tmp->span_weight)
+ break;
+ if (tmp->flags & sd_flag)
+ sd = tmp;
+ }
+ /* while loop will break here if sd == NULL */
+ }
+#endif
}
rcu_read_unlock();
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 12/13] hperf_hmp: rest of logic.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (10 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 11/13] hperf_hmp: task CPU selection logic Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 13/13] hperf_hmp: cpufreq routines Arseniy Krasnov
2015-11-07 9:52 ` [PATCH 00/13] High performance balancing logic for big.LITTLE Peter Zijlstra
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
Inserts call to main logic from 'load_balance', balance parameters
calculation during enqueue/dequeue task from runqueue and affinity mask
change callback for fair scheduling class.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
kernel/sched/fair.c | 204 +++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 202 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 79be023..06f6518 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -677,6 +677,16 @@ static u64 sched_vslice(struct cfs_rq *cfs_rq, struct sched_entity *se)
}
#ifdef CONFIG_HPERF_HMP
+static void hmp_calculate_imbalance(void)
+{
+ if (atomic_long_read(&a7_total_weight) == 0) {
+ atomic_set(&hmp_imbalance, 0);
+ return;
+ }
+
+ atomic_set(&hmp_imbalance, 1);
+}
+
static bool
is_task_hmp(struct task_struct *task, const struct cpumask *task_cpus)
{
@@ -711,6 +721,13 @@ static inline void add_druntime_sum(struct rq *rq, long delta)
rq->druntime_sum += delta;
check_druntime_sum(rq, rq->druntime_sum);
}
+
+static inline void sub_druntime_sum(struct rq *rq, long delta)
+{
+ rq->druntime_sum -= delta;
+ check_druntime_sum(rq, rq->druntime_sum);
+}
+
/* Updates druntime for a task */
static inline void
update_hmp_stat(struct cfs_rq *cfs_rq, struct sched_entity *curr,
@@ -861,7 +878,9 @@ static void update_curr(struct cfs_rq *cfs_rq)
account_cfs_rq_runtime(cfs_rq, delta_exec);
+#ifdef CONFIG_HPERF_HMP
update_hmp_stat(cfs_rq, curr, delta_exec);
+#endif
}
static void update_curr_fair(struct rq *rq)
@@ -4200,6 +4219,66 @@ static inline void hrtick_update(struct rq *rq)
}
#endif
+#ifdef CONFIG_HPERF_HMP
+#ifdef CONFIG_HPERF_HMP_DEBUG
+static void check_nr_hmp_tasks(struct rq *rq)
+{
+ if (rq->nr_hmp_tasks > rq->cfs.h_nr_running) {
+ pr_emerg("HMP BUG: rq->nr_hmp_tasks = %u, "
+ "rq->cfs.h_nr_running = %u\n", rq->nr_hmp_tasks,
+ rq->cfs.h_nr_running);
+ BUG();
+ }
+}
+#else
+static void check_nr_hmp_tasks(struct rq *rq) { }
+#endif
+
+static void nr_hmp_tasks_inc(struct rq *rq)
+{
+ if (!rq->nr_hmp_tasks) {
+ if (cpu_is_fastest(rq->cpu))
+ atomic_inc(&a15_nr_hmp_busy);
+ else
+ atomic_inc(&a7_nr_hmp_busy);
+ }
+
+ rq->nr_hmp_tasks++;
+ check_nr_hmp_tasks(rq);
+}
+
+static void nr_hmp_tasks_dec(struct rq *rq)
+{
+ rq->nr_hmp_tasks--;
+
+ if (!rq->nr_hmp_tasks) {
+ if (cpu_is_fastest(rq->cpu))
+ atomic_dec(&a15_nr_hmp_busy);
+ else
+ atomic_dec(&a7_nr_hmp_busy);
+ }
+ check_nr_hmp_tasks(rq);
+}
+
+static void
+set_cpus_allowed_hmp(struct task_struct *p, const struct cpumask *new_mask)
+{
+ bool is_hmp_before, is_hmp_after;
+
+ cpumask_copy(&p->cpus_allowed, new_mask);
+ p->nr_cpus_allowed = cpumask_weight(new_mask);
+ is_hmp_before = is_task_hmp(p, NULL);
+ is_hmp_after = is_task_hmp(p, new_mask);
+
+ if (!p->on_cpu && p->se.on_rq && (is_hmp_before != is_hmp_after)) {
+ if (is_hmp_after)
+ nr_hmp_tasks_inc(rq_of(cfs_rq_of(&p->se)));
+ else
+ nr_hmp_tasks_dec(rq_of(cfs_rq_of(&p->se)));
+ }
+}
+#endif
+
/*
* The enqueue_task method is called before nr_running is
* increased. Here we update the fair scheduling stats and
@@ -4241,8 +4320,24 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
update_cfs_shares(cfs_rq);
}
- if (!se)
+ if (!se) {
add_nr_running(rq, 1);
+#ifdef CONFIG_HPERF_HMP
+ if (is_task_hmp(p, NULL))
+ nr_hmp_tasks_inc(rq);
+
+ if (cpu_is_fastest(rq->cpu)) {
+ atomic_long_add(p->se.load.weight, &a15_total_weight);
+ if (p->se.druntime < 0)
+ add_druntime_sum(rq, p->se.druntime);
+ } else {
+ atomic_long_add(p->se.load.weight, &a7_total_weight);
+ if (p->se.druntime > 0)
+ add_druntime_sum(rq, p->se.druntime);
+ }
+ hmp_calculate_imbalance();
+#endif
+ }
hrtick_update(rq);
}
@@ -4301,8 +4396,30 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
update_cfs_shares(cfs_rq);
}
- if (!se)
+ if (!se) {
sub_nr_running(rq, 1);
+#ifdef CONFIG_HPERF_HMP
+ if (is_task_hmp(p, NULL))
+ nr_hmp_tasks_dec(rq);
+
+ /*
+ * Set this field to 0 because if task selected for migration
+ * fall asleep it will never be selected again for migration.
+ */
+ p->se.migrate_candidate = 0;
+
+ if (cpu_is_fastest(rq->cpu)) {
+ atomic_long_sub(p->se.load.weight, &a15_total_weight);
+ if (p->se.druntime < 0)
+ sub_druntime_sum(rq, p->se.druntime);
+ } else {
+ atomic_long_sub(p->se.load.weight, &a7_total_weight);
+ if (p->se.druntime > 0)
+ sub_druntime_sum(rq, p->se.druntime);
+ }
+ hmp_calculate_imbalance();
+#endif
+ }
hrtick_update(rq);
}
@@ -5848,6 +5965,11 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
}
schedstat_inc(p, se.statistics.nr_failed_migrations_hot);
+
+#ifdef CONFIG_HPERF_HMP
+ if (env->src_rq->migrate_task) /*idle pull*/
+ return 1;
+#endif
return 0;
}
@@ -5879,6 +6001,10 @@ static struct task_struct *detach_one_task(struct lb_env *env)
if (!can_migrate_task(p, env))
continue;
+#ifdef CONFIG_HPERF_HMP
+ if (p->se.migrate_candidate)
+ continue;
+#endif
detach_task(p, env);
/*
@@ -5946,6 +6072,10 @@ static int detach_tasks(struct lb_env *env)
if ((load / 2) > env->imbalance)
goto next;
+#ifdef CONFIG_HPERF_HMP
+ if (p->se.migrate_candidate)
+ goto next;
+#endif
detach_task(p, env);
list_add(&p->se.group_node, &env->tasks);
@@ -7909,6 +8039,44 @@ unlock:
return ld_moved;
}
+
+/**
+ * hmp_do_rebalance(): Checks imbalance in HMP domain and performs balancing.
+ *
+ * @sd: Current sched domain.
+ * @this_cpu: without NO_HZ same as smp_processor_id().
+ *
+ * Returns moved weight.
+ */
+static unsigned int hmp_do_rebalance(struct sched_domain *sd, int this_cpu)
+{
+ unsigned int ld_moved = 0;
+ switch (is_hmp_imbalance(sd)) {
+ case SWAP_TASKS:
+ ld_moved = swap_tasks(sd, this_cpu);
+ break;
+ case A15_TO_A7:
+ ld_moved = move_a15_to_a7(sd, this_cpu);
+ break;
+ case A7_TO_A15:
+ ld_moved = move_a7_to_a15(sd, this_cpu);
+ break;
+ case SKIP_REBALANCE:
+ default:
+ break;
+ }
+ return ld_moved;
+}
+
+/* HMP balancing entry point */
+static unsigned int hmp_load_balance(struct sched_domain *sd,
+ enum cpu_idle_type idle, int this_cpu)
+{
+ if (idle == CPU_NEWLY_IDLE || idle == CPU_IDLE)
+ return hmp_idle_pull(sd, this_cpu);
+ else
+ return hmp_do_rebalance(sd, this_cpu);
+}
#endif /* CONFIG_HPERF_HMP */
/*
@@ -7938,6 +8106,11 @@ static int load_balance(int this_cpu, struct rq *this_rq,
.tasks = LIST_HEAD_INIT(env.tasks),
};
+#ifdef CONFIG_HPERF_HMP
+ /* It is HMP domain, so branch to HPERF_HMP logic */
+ if (sd->flags & SD_HMP_BALANCE)
+ return hmp_load_balance(sd, idle, this_cpu);
+#endif
/*
* For NEWLY_IDLE load_balancing, we don't need to consider
* other cpus in our group
@@ -8322,6 +8495,14 @@ static int active_load_balance_cpu_stop(void *data)
struct sched_domain *sd;
struct task_struct *p = NULL;
+#ifdef CONFIG_HPERF_HMP_DEBUG
+ if (!cpumask_test_cpu(target_cpu, cpu_fastest_mask) &&
+ !cpumask_test_cpu(target_cpu, cpu_slowest_mask)) {
+ pr_emerg("hperf_hmp: %s: for CPU#%i target_cpu is invalid: %i!\n",
+ __func__, busiest_cpu, target_cpu);
+ BUG();
+ }
+#endif
raw_spin_lock_irq(&busiest_rq->lock);
/* make sure the requested cpu hasn't gone down in the meantime */
@@ -8349,6 +8530,9 @@ static int active_load_balance_cpu_stop(void *data)
}
if (likely(sd)) {
+#ifdef CONFIG_HPERF_HMP
+ struct task_struct *migrate_task;
+#endif
struct lb_env env = {
.sd = sd,
.dst_cpu = target_cpu,
@@ -8360,7 +8544,19 @@ static int active_load_balance_cpu_stop(void *data)
schedstat_inc(sd, alb_count);
+#ifdef CONFIG_HPERF_HMP
+ if (env.src_rq->migrate_task) {
+ migrate_task = env.src_rq->migrate_task;
+ p = detach_specified_task(migrate_task, &env);
+ if (p)
+ migrate_task->se.last_migration = jiffies;
+ env.src_rq->migrate_task = NULL;
+ } else {
+ p = detach_one_task(&env);
+ }
+#else
p = detach_one_task(&env);
+#endif
if (p)
schedstat_inc(sd, alb_pushed);
else
@@ -9267,8 +9463,12 @@ const struct sched_class fair_sched_class = {
.task_waking = task_waking_fair,
.task_dead = task_dead_fair,
+#ifdef CONFIG_HPERF_HMP
+ .set_cpus_allowed = set_cpus_allowed_hmp,
+#else
.set_cpus_allowed = set_cpus_allowed_common,
#endif
+#endif
.set_curr_task = set_curr_task_fair,
.task_tick = task_tick_fair,
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 13/13] hperf_hmp: cpufreq routines.
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (11 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 12/13] hperf_hmp: rest of logic Arseniy Krasnov
@ 2015-11-06 12:02 ` Arseniy Krasnov
2015-11-07 9:52 ` [PATCH 00/13] High performance balancing logic for big.LITTLE Peter Zijlstra
13 siblings, 0 replies; 17+ messages in thread
From: Arseniy Krasnov @ 2015-11-06 12:02 UTC (permalink / raw)
To: linux, mingo, peterz
Cc: a.krasnov, v.tyrtov, s.rogachev, linux-kernel, Tarek Dakhran,
Sergey Dyasly, Dmitriy Safonov, Ilya Maximets
Adds CPU frequency change notifier in fair scheduling class. Every time
when governor changes frequency, it calls callback from this patch. Frequency of
each CPU is used for imbalance calculation.
Signed-off-by: Tarek Dakhran <t.dakhran@samsung.com>
Signed-off-by: Sergey Dyasly <s.dyasly@samsung.com>
Signed-off-by: Dmitriy Safonov <d.safonov@partner.samsung.com>
Signed-off-by: Arseniy Krasnov <a.krasnov@samsung.com>
Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---
kernel/sched/fair.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 76 insertions(+)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 06f6518..87dc0db 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -33,6 +33,10 @@
#include <trace/events/sched.h>
+#ifdef CONFIG_HPERF_HMP
+#include <linux/cpufreq.h>
+#endif
+
#include "sched.h"
/*
@@ -101,6 +105,11 @@ const_debug unsigned int sysctl_sched_migration_cost = 500000UL;
unsigned int __read_mostly sysctl_sched_shares_window = 10000000UL;
#ifdef CONFIG_HPERF_HMP
+/*
+ * Log level of hperf_hmp messages. Bigger means more messages.
+ * Maximum level is 3.
+ */
+unsigned int sysctl_sched_hperf_hmp_log_level;
extern void hmp_set_cpu_masks(struct cpumask *, struct cpumask *);
static atomic_t a15_nr_hmp_busy = ATOMIC_INIT(0);
static atomic_t a7_nr_hmp_busy = ATOMIC_INIT(0);
@@ -7229,6 +7238,73 @@ static int should_we_balance(struct lb_env *env)
return balance_cpu == env->dst_cpu;
}
#ifdef CONFIG_HPERF_HMP
+static void hperf_hmp_vprint(unsigned int log_level, const char *format,
+ va_list ap)
+{
+ if (sysctl_sched_hperf_hmp_log_level < log_level)
+ return;
+ vprintk(format, ap);
+}
+
+static void hperf_hmp_print(unsigned int log_level, const char *format, ...)
+{
+ va_list ap;
+
+ va_start(ap, format);
+ hperf_hmp_vprint(log_level, format, ap);
+ va_end(ap);
+}
+
+/* Called when frequency is changed */
+static int hmp_cpufreq_callback(struct notifier_block *nb,
+ unsigned long event, void *data)
+{
+ struct cpufreq_freqs *new_freq = data;
+
+ /* recount power only after change of frequency */
+ if (event != CPUFREQ_POSTCHANGE)
+ return NOTIFY_DONE;
+
+ if (!new_freq)
+ return NOTIFY_DONE;
+
+ freq_scale_cpu_power[new_freq->cpu] = (new_freq->new >> 10);
+
+ /* Apply slowdown coefficient of 1.9 for A7 CPUs */
+ if (!cpu_is_fastest(new_freq->cpu)) {
+ freq_scale_cpu_power[new_freq->cpu] *= 10;
+ freq_scale_cpu_power[new_freq->cpu] /= 19;
+ }
+
+ hperf_hmp_print(2, KERN_INFO "hperf_hmp: CPU#%i new frequency is: %u MHz\n",
+ new_freq->cpu, new_freq->new / 1000);
+
+ return NOTIFY_OK;
+}
+
+static struct notifier_block cpufreq_notifier = {
+ .notifier_call = hmp_cpufreq_callback
+};
+
+static int __init register_sched_cpufreq_notifier(void)
+{
+ int err = 0;
+ int cpu;
+
+ for_each_online_cpu(cpu)
+ freq_scale_cpu_power[cpu] = capacity_of(cpu);
+
+ err = cpufreq_register_notifier(&cpufreq_notifier,
+ CPUFREQ_TRANSITION_NOTIFIER);
+ if (!err)
+ pr_info("hperf_hmp: registered cpufreq transition notifier\n");
+ else
+ pr_info("hperf_hmp: failed to register cpufreq notifier!\n");
+
+ return err;
+}
+core_initcall(register_sched_cpufreq_notifier);
+
/**
* is_hmp_imbalance(): Calculates imbalance between HMP domains.
* @sd: Current sched domain.
--
1.9.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH 11/13] hperf_hmp: task CPU selection logic.
2015-11-06 12:02 ` [PATCH 11/13] hperf_hmp: task CPU selection logic Arseniy Krasnov
@ 2015-11-06 12:29 ` kbuild test robot
0 siblings, 0 replies; 17+ messages in thread
From: kbuild test robot @ 2015-11-06 12:29 UTC (permalink / raw)
To: Arseniy Krasnov
Cc: kbuild-all, linux, mingo, peterz, a.krasnov, v.tyrtov,
s.rogachev, linux-kernel, Tarek Dakhran, Sergey Dyasly,
Dmitriy Safonov, Ilya Maximets
[-- Attachment #1: Type: text/plain, Size: 3195 bytes --]
Hi Arseniy,
[auto build test WARNING on tip/sched/core]
[also build test WARNING on v4.3 next-20151106]
url: https://github.com/0day-ci/linux/commits/Arseniy-Krasnov/High-performance-balancing-logic-for-big-LITTLE/20151106-200901
config: x86_64-randconfig-x018-11051832 (attached as .config)
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
All warnings (new ones prefixed by >>):
kernel/sched/fair.c: In function 'select_task_rq_fair':
>> kernel/sched/fair.c:5159:6: warning: suggest explicit braces to avoid ambiguous 'else' [-Wparentheses]
if (sd_flag & SD_BALANCE_WAKE) /* XXX always ? */
^
vim +/else +5159 kernel/sched/fair.c
29cd8bae kernel/sched_fair.c Peter Zijlstra 2009-09-17 5143 break;
f03542a7 kernel/sched/fair.c Alex Shi 2012-07-26 5144 }
29cd8bae kernel/sched_fair.c Peter Zijlstra 2009-09-17 5145
f03542a7 kernel/sched/fair.c Alex Shi 2012-07-26 5146 if (tmp->flags & sd_flag)
c88d5910 kernel/sched_fair.c Peter Zijlstra 2009-09-10 5147 sd = tmp;
63b0e9ed kernel/sched/fair.c Mike Galbraith 2015-07-14 5148 else if (!want_affine)
63b0e9ed kernel/sched/fair.c Mike Galbraith 2015-07-14 5149 break;
c88d5910 kernel/sched_fair.c Peter Zijlstra 2009-09-10 5150 }
4ae7d5ce kernel/sched_fair.c Ingo Molnar 2008-03-19 5151
63b0e9ed kernel/sched/fair.c Mike Galbraith 2015-07-14 5152 if (affine_sd) {
63b0e9ed kernel/sched/fair.c Mike Galbraith 2015-07-14 5153 sd = NULL; /* Prefer wake_affine over balance flags */
63b0e9ed kernel/sched/fair.c Mike Galbraith 2015-07-14 5154 if (cpu != prev_cpu && wake_affine(affine_sd, p, sync))
63b0e9ed kernel/sched/fair.c Mike Galbraith 2015-07-14 5155 new_cpu = cpu;
8b911acd kernel/sched_fair.c Mike Galbraith 2010-03-11 5156 }
3b640894 kernel/sched_fair.c Peter Zijlstra 2009-09-16 5157
63b0e9ed kernel/sched/fair.c Mike Galbraith 2015-07-14 5158 if (!sd) {
63b0e9ed kernel/sched/fair.c Mike Galbraith 2015-07-14 @5159 if (sd_flag & SD_BALANCE_WAKE) /* XXX always ? */
9b7aaf11 kernel/sched/fair.c Arseniy Krasnov 2015-11-06 5160 if (IS_ENABLED(CONFIG_HPERF_HMP) && sync)
9b7aaf11 kernel/sched/fair.c Arseniy Krasnov 2015-11-06 5161 new_cpu = prev_cpu;
9b7aaf11 kernel/sched/fair.c Arseniy Krasnov 2015-11-06 5162 else
9b7aaf11 kernel/sched/fair.c Arseniy Krasnov 2015-11-06 5163 new_cpu = select_idle_sibling(p, prev_cpu);
9b7aaf11 kernel/sched/fair.c Arseniy Krasnov 2015-11-06 5164 } else {
9b7aaf11 kernel/sched/fair.c Arseniy Krasnov 2015-11-06 5165 #ifdef CONFIG_HPERF_HMP
9b7aaf11 kernel/sched/fair.c Arseniy Krasnov 2015-11-06 5166 new_cpu = hmp_select_task_rq_fair(p);
9b7aaf11 kernel/sched/fair.c Arseniy Krasnov 2015-11-06 5167 #else
:::::: The code at line 5159 was first introduced by commit
:::::: 63b0e9edceec10fa41ec33393a1515a5ff444277 sched/fair: Beef up wake_wide()
:::::: TO: Mike Galbraith <umgwanakikbuti@gmail.com>
:::::: CC: Ingo Molnar <mingo@kernel.org>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 21230 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 07/13] hperf_hmp: migration auxiliary functions.
2015-11-06 12:02 ` [PATCH 07/13] hperf_hmp: migration auxiliary functions Arseniy Krasnov
@ 2015-11-06 13:03 ` kbuild test robot
0 siblings, 0 replies; 17+ messages in thread
From: kbuild test robot @ 2015-11-06 13:03 UTC (permalink / raw)
To: Arseniy Krasnov
Cc: kbuild-all, linux, mingo, peterz, a.krasnov, v.tyrtov,
s.rogachev, linux-kernel, Tarek Dakhran, Sergey Dyasly,
Dmitriy Safonov, Ilya Maximets
[-- Attachment #1: Type: text/plain, Size: 6314 bytes --]
Hi Arseniy,
[auto build test WARNING on tip/sched/core]
[also build test WARNING on v4.3 next-20151106]
url: https://github.com/0day-ci/linux/commits/Arseniy-Krasnov/High-performance-balancing-logic-for-big-LITTLE/20151106-200901
reproduce: make htmldocs
All warnings (new ones prefixed by >>):
include/linux/init.h:1: warning: no structured comments found
>> kernel/sched/fair.c:7198: warning: No description found for parameter 'p'
>> kernel/sched/fair.c:7198: warning: Excess function parameter 'pm' description in 'detach_specified_task'
kernel/sys.c:1: warning: no structured comments found
drivers/dma-buf/seqno-fence.c:1: warning: no structured comments found
drivers/dma-buf/reservation.c:1: warning: no structured comments found
include/linux/reservation.h:1: warning: no structured comments found
include/media/v4l2-dv-timings.h:29: warning: cannot understand function prototype: 'const struct v4l2_dv_timings v4l2_dv_timings_presets[]; '
include/media/v4l2-dv-timings.h:147: warning: No description found for parameter 'frame_height'
include/media/v4l2-dv-timings.h:147: warning: No description found for parameter 'hfreq'
include/media/v4l2-dv-timings.h:147: warning: No description found for parameter 'vsync'
include/media/v4l2-dv-timings.h:147: warning: No description found for parameter 'active_width'
include/media/v4l2-dv-timings.h:147: warning: No description found for parameter 'polarities'
include/media/v4l2-dv-timings.h:147: warning: No description found for parameter 'interlaced'
include/media/v4l2-dv-timings.h:147: warning: No description found for parameter 'fmt'
include/media/v4l2-dv-timings.h:171: warning: No description found for parameter 'frame_height'
include/media/v4l2-dv-timings.h:171: warning: No description found for parameter 'hfreq'
include/media/v4l2-dv-timings.h:171: warning: No description found for parameter 'vsync'
include/media/v4l2-dv-timings.h:171: warning: No description found for parameter 'polarities'
include/media/v4l2-dv-timings.h:171: warning: No description found for parameter 'interlaced'
include/media/v4l2-dv-timings.h:171: warning: No description found for parameter 'aspect'
include/media/v4l2-dv-timings.h:171: warning: No description found for parameter 'fmt'
include/media/v4l2-dv-timings.h:184: warning: No description found for parameter 'hor_landscape'
include/media/v4l2-dv-timings.h:184: warning: No description found for parameter 'vert_portrait'
include/media/videobuf2-core.h:112: warning: No description found for parameter 'get_dmabuf'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_alloc'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_put'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_get_dmabuf'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_get_userptr'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_put_userptr'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_prepare'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_finish'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_attach_dmabuf'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_detach_dmabuf'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_map_dmabuf'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_unmap_dmabuf'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_vaddr'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_cookie'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_num_users'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_mem_mmap'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_buf_init'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_buf_prepare'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_buf_finish'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_buf_cleanup'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_buf_queue'
include/media/videobuf2-core.h:233: warning: No description found for parameter 'cnt_buf_done'
drivers/media/dvb-core/dvbdev.h:199: warning: Excess function parameter 'device' description in 'dvb_register_device'
drivers/media/dvb-core/dvbdev.h:199: warning: Excess function parameter 'adapter_nums' description in 'dvb_register_device'
include/linux/hsi/hsi.h:150: warning: Excess struct/union/enum/typedef member 'e_handler' description in 'hsi_client'
include/linux/hsi/hsi.h:150: warning: Excess struct/union/enum/typedef member 'pclaimed' description in 'hsi_client'
include/linux/hsi/hsi.h:150: warning: Excess struct/union/enum/typedef member 'nb' description in 'hsi_client'
vim +/p +7198 kernel/sched/fair.c
7182 if (task_running(env->src_rq, p)) {
7183 schedstat_inc(p, se.statistics.nr_failed_migrations_running);
7184 return 0;
7185 }
7186 return 1;
7187 }
7188
7189 /**
7190 * detach_specified_task(): Detaches specified task.
7191 * @pm: Task to move.
7192 * @env: Migration parameters.
7193 *
7194 * Returns moved task.
7195 */
7196 static struct task_struct *
7197 detach_specified_task(struct task_struct *p, struct lb_env *env)
> 7198 {
7199 lockdep_assert_held(&env->src_rq->lock);
7200
7201 /* If task to move falls asleep, so don't scan runqueue and return */
7202 if (p->se.migrate_candidate == 0)
7203 return 0;
7204
7205 if (throttled_lb_pair(task_group(p), env->src_rq->cpu, env->dst_cpu))
7206 goto exit;
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 6062 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 00/13] High performance balancing logic for big.LITTLE
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
` (12 preceding siblings ...)
2015-11-06 12:02 ` [PATCH 13/13] hperf_hmp: cpufreq routines Arseniy Krasnov
@ 2015-11-07 9:52 ` Peter Zijlstra
13 siblings, 0 replies; 17+ messages in thread
From: Peter Zijlstra @ 2015-11-07 9:52 UTC (permalink / raw)
To: Arseniy Krasnov; +Cc: linux, mingo, v.tyrtov, s.rogachev, linux-kernel
No, no, no, no.
This is horrible and exactly what I've been telling people I do not want
to see.
This is very arch specific scheduler code, and very badly done. It
doesn't even call the groups big and little, it goes so far as to put a7
and a15 in sched domain member names.
It doesn't get topology information from device tree but from hard coded
CONFIG strings.
It introduces a swap_task function while we already have one.
It doesn't integrate with any of the other bits that make up and
influence energy consumption such as cpuidle and cpufreq (very minor one
way).
It doesn't mention the existing energy-aware-scheduling effort, nor how
that approach cannot be made to work for this.
It has a completely broken SoB chain.
It introduces new metrics (like druntime) without first defining them;
and in general very poor Changelogs.
In general, this makes me very sad. Please start by participating in the
existing discussion.
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2015-11-07 9:52 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-06 12:02 [PATCH 00/13] High performance balancing logic for big.LITTLE Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 01/13] hperf_hmp: add new config for arm and arm64 Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 02/13] hperf_hmp: introduce hew domain flag Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 03/13] hperf_hmp: add sched domains initialization Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 04/13] hperf_hmp: scheduler initialization routines Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 05/13] hperf_hmp: introduce druntime metric Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 06/13] hperf_hmp: is_hmp_imbalance introduced Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 07/13] hperf_hmp: migration auxiliary functions Arseniy Krasnov
2015-11-06 13:03 ` kbuild test robot
2015-11-06 12:02 ` [PATCH 08/13] hperf_hmp: swap tasks function Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 09/13] hperf_hmp: one way balancing function Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 10/13] hperf_hmp: idle pull function Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 11/13] hperf_hmp: task CPU selection logic Arseniy Krasnov
2015-11-06 12:29 ` kbuild test robot
2015-11-06 12:02 ` [PATCH 12/13] hperf_hmp: rest of logic Arseniy Krasnov
2015-11-06 12:02 ` [PATCH 13/13] hperf_hmp: cpufreq routines Arseniy Krasnov
2015-11-07 9:52 ` [PATCH 00/13] High performance balancing logic for big.LITTLE Peter Zijlstra
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.