[PATCH v5 0/7] Add latency priority for CFS class

[PATCH v5 0/7] Add latency priority for CFS class

[Vincent Guittot]

This patchset restarts the work about adding a latency priority to describe
the latency tolerance of cfs tasks.

The patches [1-3] have been done by Parth:
https://lore.kernel.org/lkml/20200228090755.22829-1-parth@linux.ibm.com/

I have just rebased and moved the set of latency priority outside the
priority update. I have removed the reviewed tag because the patches
are 2 years old.

This aims to be a generic interface and the following patches is one use
of it to improve the scheduling latency of cfs tasks.

The patch [4] uses latency nice priority to define a latency offset
and then decide if a cfs task can or should preempt the current
running task. The patch gives some tests results with cyclictests and
hackbench to highlight the benefit of latency priority for short
interactive task or long intensive tasks.

Patch [5] adds the support of latency nice priority to task group by
adding a cpu.latency.nice field. The range is [-20:19] as for setting task
latency priority.

Patch [6] makes sched_core taking into account the latency offset.

Patch [7] adds a rb tree to cover some corner cases where the latency
sensitive task (priority < 0) is preempted by high priority task (RT/DL)
or fails to preempt them. This patch ensures that tasks will have at least
a slice of sched_min_granularity in priority at wakeup. The patch gives
results to show the benefit in addition to patch 4.

I have also backported the patchset on a dragonboard RB3 with an android
mainline kernel based on v5.18 for a quick test. I have used the
TouchLatency app which is part of AOSP and described to be a very good
test to highlight jitter and jank frame sources of a system [1].
In addition to the app, I have added some short running tasks waking-up
regularly (to use the 8 cpus for 4 ms every 37777us) to stress the system
without overloading it (and disabling EAS). The 1st results shows that the
patchset helps to reduce the missed deadline frames from 5% to less than
0.1% when the cpu.latency.nice of task group are set.

I have also tested the patchset with the modified version of the alsa
latency test that has been shared by Tim. The test quickly xruns with
default latency nice priority 0 but is able to run without underuns with
a latency -20 and hackbench running simultaneously.


[1] https://source.android.com/docs/core/debug/eval_perf#touchlatency

Change since v4:
- Removed permission checks to set latency priority. This enables user
  without elevated priviledge like audio application to set their latency
  priority as requested by Tim.
- Removed cpu.latency and replaced it by cpu.latency.nice so we keep a
  generic interface not tied to latency_offset which can be used to
  implement other latency features.
- Added an entry in Documentation/admin-guide/cgroup-v2.rst to describe
  cpu.latency.nice.
- Fix some typos.

Change since v3:
- Fix 2 compilation warnings raised by kernel test robot <lkp@intel.com>

Change since v2:
- Set a latency_offset field instead of saving a weight and computing it
  on the fly.
- Make latency_offset available for task group: cpu.latency
- Fix some corner cases to make latency sensitive tasks schedule first and
  add a rb tree for latency sensitive task.

Change since v1:
- fix typo
- move some codes in the right patch to make bisect happy
- simplify and fixed how the weight is computed
- added support of sched core patch 7

Parth Shah (3):
  sched: Introduce latency-nice as a per-task attribute
  sched/core: Propagate parent task's latency requirements to the child
    task
  sched: Allow sched_{get,set}attr to change latency_nice of the task

Vincent Guittot (4):
  sched/fair: Take into account latency priority at wakeup
  sched/fair: Add sched group latency support
  sched/core: Support latency priority with sched core
  sched/fair: Add latency list

 Documentation/admin-guide/cgroup-v2.rst |   8 +
 include/linux/sched.h                   |   5 +
 include/uapi/linux/sched.h              |   4 +-
 include/uapi/linux/sched/types.h        |  19 +++
 init/init_task.c                        |   1 +
 kernel/sched/core.c                     | 106 +++++++++++++
 kernel/sched/debug.c                    |   1 +
 kernel/sched/fair.c                     | 189 +++++++++++++++++++++++-
 kernel/sched/sched.h                    |  37 ++++-
 tools/include/uapi/linux/sched.h        |   4 +-
 10 files changed, 366 insertions(+), 8 deletions(-)

-- 
2.17.1

[PATCH v5 1/7] sched: Introduce latency-nice as a per-task attribute

[Vincent Guittot]

From: Parth Shah <parth@linux.ibm.com>

Latency-nice indicates the latency requirements of a task with respect
to the other tasks in the system. The value of the attribute can be within
the range of [-20, 19] both inclusive to be in-line with the values just
like task nice values.

latency_nice = -20 indicates the task to have the least latency as
compared to the tasks having latency_nice = +19.

The latency_nice may affect only the CFS SCHED_CLASS by getting
latency requirements from the userspace.

Additionally, add debugging bits for newly added latency_nice attribute.

Signed-off-by: Parth Shah <parth@linux.ibm.com>
[rebase]
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 include/linux/sched.h |  1 +
 kernel/sched/debug.c  |  1 +
 kernel/sched/sched.h  | 18 ++++++++++++++++++
 3 files changed, 20 insertions(+)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 15e3bd96e4ce..6805f378a9c3 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -783,6 +783,7 @@ struct task_struct {
 	int				static_prio;
 	int				normal_prio;
 	unsigned int			rt_priority;
+	int				latency_nice;
 
 	struct sched_entity		se;
 	struct sched_rt_entity		rt;
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index bb3d63bdf4ae..a3f7876217a6 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1042,6 +1042,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 #endif
 	P(policy);
 	P(prio);
+	P(latency_nice);
 	if (task_has_dl_policy(p)) {
 		P(dl.runtime);
 		P(dl.deadline);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 1fc198be1ffd..eeb6efb0b610 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -125,6 +125,24 @@ extern int sched_rr_timeslice;
  */
 #define NS_TO_JIFFIES(TIME)	((unsigned long)(TIME) / (NSEC_PER_SEC / HZ))
 
+/*
+ * Latency nice is meant to provide scheduler hints about the relative
+ * latency requirements of a task with respect to other tasks.
+ * Thus a task with latency_nice == 19 can be hinted as the task with no
+ * latency requirements, in contrast to the task with latency_nice == -20
+ * which should be given priority in terms of lower latency.
+ */
+#define MAX_LATENCY_NICE	19
+#define MIN_LATENCY_NICE	-20
+
+#define LATENCY_NICE_WIDTH	\
+	(MAX_LATENCY_NICE - MIN_LATENCY_NICE + 1)
+
+/*
+ * Default tasks should be treated as a task with latency_nice = 0.
+ */
+#define DEFAULT_LATENCY_NICE	0
+
 /*
  * Increase resolution of nice-level calculations for 64-bit architectures.
  * The extra resolution improves shares distribution and load balancing of
-- 
2.17.1

[PATCH v5 2/7] sched/core: Propagate parent task's latency requirements to the child task

[Vincent Guittot]

From: Parth Shah <parth@linux.ibm.com>

Clone parent task's latency_nice attribute to the forked child task.

Reset the latency_nice value to default value when the child task is
set to sched_reset_on_fork.

Also, initialize init_task.latency_nice value with DEFAULT_LATENCY_NICE
value

Signed-off-by: Parth Shah <parth@linux.ibm.com>
[rebase]
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 init/init_task.c    | 1 +
 kernel/sched/core.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/init/init_task.c b/init/init_task.c
index ff6c4b9bfe6b..7dd71dd2d261 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -78,6 +78,7 @@ struct task_struct init_task
 	.prio		= MAX_PRIO - 20,
 	.static_prio	= MAX_PRIO - 20,
 	.normal_prio	= MAX_PRIO - 20,
+	.latency_nice	= DEFAULT_LATENCY_NICE,
 	.policy		= SCHED_NORMAL,
 	.cpus_ptr	= &init_task.cpus_mask,
 	.user_cpus_ptr	= NULL,
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 4fa4a3ddb4f4..ada2d05bd894 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4559,6 +4559,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
 		p->prio = p->normal_prio = p->static_prio;
 		set_load_weight(p, false);
 
+		p->latency_nice = DEFAULT_LATENCY_NICE;
 		/*
 		 * We don't need the reset flag anymore after the fork. It has
 		 * fulfilled its duty:
-- 
2.17.1

[PATCH v5 3/7] sched: Allow sched_{get,set}attr to change latency_nice of the task

[Vincent Guittot]

From: Parth Shah <parth@linux.ibm.com>

Introduce the latency_nice attribute to sched_attr and provide a
mechanism to change the value with the use of sched_setattr/sched_getattr
syscall.

Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in
latency_nice of the task on every sched_setattr syscall.

Signed-off-by: Parth Shah <parth@linux.ibm.com>
[rebase and add a dedicated __setscheduler_latency ]
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 include/uapi/linux/sched.h       |  4 +++-
 include/uapi/linux/sched/types.h | 19 +++++++++++++++++++
 kernel/sched/core.c              | 24 ++++++++++++++++++++++++
 tools/include/uapi/linux/sched.h |  4 +++-
 4 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h
index 3bac0a8ceab2..b2e932c25be6 100644
--- a/include/uapi/linux/sched.h
+++ b/include/uapi/linux/sched.h
@@ -132,6 +132,7 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_LATENCY_NICE		0x80
 
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
@@ -143,6 +144,7 @@ struct clone_args {
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_LATENCY_NICE)
 
 #endif /* _UAPI_LINUX_SCHED_H */
diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/types.h
index f2c4589d4dbf..db1e8199e8c8 100644
--- a/include/uapi/linux/sched/types.h
+++ b/include/uapi/linux/sched/types.h
@@ -10,6 +10,7 @@ struct sched_param {
 
 #define SCHED_ATTR_SIZE_VER0	48	/* sizeof first published struct */
 #define SCHED_ATTR_SIZE_VER1	56	/* add: util_{min,max} */
+#define SCHED_ATTR_SIZE_VER2	60	/* add: latency_nice */
 
 /*
  * Extended scheduling parameters data structure.
@@ -98,6 +99,22 @@ struct sched_param {
  * scheduled on a CPU with no more capacity than the specified value.
  *
  * A task utilization boundary can be reset by setting the attribute to -1.
+ *
+ * Latency Tolerance Attributes
+ * ===========================
+ *
+ * A subset of sched_attr attributes allows to specify the relative latency
+ * requirements of a task with respect to the other tasks running/queued in the
+ * system.
+ *
+ * @ sched_latency_nice	task's latency_nice value
+ *
+ * The latency_nice of a task can have any value in a range of
+ * [MIN_LATENCY_NICE..MAX_LATENCY_NICE].
+ *
+ * A task with latency_nice with the value of LATENCY_NICE_MIN can be
+ * taken for a task requiring a lower latency as opposed to the task with
+ * higher latency_nice.
  */
 struct sched_attr {
 	__u32 size;
@@ -120,6 +137,8 @@ struct sched_attr {
 	__u32 sched_util_min;
 	__u32 sched_util_max;
 
+	/* latency requirement hints */
+	__s32 sched_latency_nice;
 };
 
 #endif /* _UAPI_LINUX_SCHED_TYPES_H */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ada2d05bd894..6a6116ea4c2c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7318,6 +7318,14 @@ static void __setscheduler_params(struct task_struct *p,
 	p->rt_priority = attr->sched_priority;
 	p->normal_prio = normal_prio(p);
 	set_load_weight(p, true);
+
+}
+
+static void __setscheduler_latency(struct task_struct *p,
+		const struct sched_attr *attr)
+{
+	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
+		p->latency_nice = attr->sched_latency_nice;
 }
 
 /*
@@ -7460,6 +7468,13 @@ static int __sched_setscheduler(struct task_struct *p,
 			return retval;
 	}
 
+	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
+		if (attr->sched_latency_nice > MAX_LATENCY_NICE)
+			return -EINVAL;
+		if (attr->sched_latency_nice < MIN_LATENCY_NICE)
+			return -EINVAL;
+	}
+
 	if (pi)
 		cpuset_read_lock();
 
@@ -7494,6 +7509,9 @@ static int __sched_setscheduler(struct task_struct *p,
 			goto change;
 		if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
 			goto change;
+		if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
+		    attr->sched_latency_nice != p->latency_nice)
+			goto change;
 
 		p->sched_reset_on_fork = reset_on_fork;
 		retval = 0;
@@ -7582,6 +7600,7 @@ static int __sched_setscheduler(struct task_struct *p,
 		__setscheduler_params(p, attr);
 		__setscheduler_prio(p, newprio);
 	}
+	__setscheduler_latency(p, attr);
 	__setscheduler_uclamp(p, attr);
 
 	if (queued) {
@@ -7792,6 +7811,9 @@ static int sched_copy_attr(struct sched_attr __user *uattr, struct sched_attr *a
 	    size < SCHED_ATTR_SIZE_VER1)
 		return -EINVAL;
 
+	if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) &&
+	    size < SCHED_ATTR_SIZE_VER2)
+		return -EINVAL;
 	/*
 	 * XXX: Do we want to be lenient like existing syscalls; or do we want
 	 * to be strict and return an error on out-of-bounds values?
@@ -8029,6 +8051,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
 	get_params(p, &kattr);
 	kattr.sched_flags &= SCHED_FLAG_ALL;
 
+	kattr.sched_latency_nice = p->latency_nice;
+
 #ifdef CONFIG_UCLAMP_TASK
 	/*
 	 * This could race with another potential updater, but this is fine
diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sched.h
index 3bac0a8ceab2..ecc4884bfe4b 100644
--- a/tools/include/uapi/linux/sched.h
+++ b/tools/include/uapi/linux/sched.h
@@ -132,6 +132,7 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_LATENCY_NICE		0X80
 
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
@@ -143,6 +144,7 @@ struct clone_args {
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_LATENCY_NICE)
 
 #endif /* _UAPI_LINUX_SCHED_H */
-- 
2.17.1

[PATCH v5 4/7] sched/fair: Take into account latency priority at wakeup

[Vincent Guittot]

Take into account the latency priority of a thread when deciding to
preempt the current running thread. We don't want to provide more CPU
bandwidth to a thread but reorder the scheduling to run latency sensitive
task first whenever possible.

As long as a thread didn't use its bandwidth, it will be able to preempt
the current thread.

At the opposite, a thread with a low latency priority will preempt current
thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will
wait for the tick to get its sched slice.

                                   curr vruntime
                                       |
                      sysctl_sched_wakeup_granularity
                                   <-->
----------------------------------|----|-----------------------|---------------
                                  |    |<--------------------->
                                  |    .  sysctl_sched_latency
                                  |    .
default/current latency entity    |    .
                                  |    .
1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-
se preempts curr at wakeup ------>|<- se doesn't preempt curr -----------------
                                  |    .
                                  |    .
                                  |    .
low latency entity                |    .
                                   ---------------------->|
                               % of sysctl_sched_latency  |
1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-
preempt ------------------------------------------------->|<- do not preempt --
                                  |    .
                                  |    .
                                  |    .
high latency entity               |    .
         |<-----------------------|    .
         | % of sysctl_sched_latency   .
111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1
preempt->|<- se doesn't preempt curr ------------------------------------------

Tests results of nice latency impact on heavy load like hackbench:

hackbench -l (2560 / group) -g group
group        latency 0             latency 19
1            1.350(+/- 1.10%)      1.287(+/- 1.60%) + 5%
4            1.392(+/- 1.67%)      1.248(+/- 1.51%) +11%
8            1.294(+/- 1.56%)      1.254(+/- 1.96%) + 3%
16           1.315(+/- 1.02%)      1.288(+/- 1.53%) + 2%

hackbench -p -l (2560 / group) -g group
group
1            1.768(+/- 13%)      0.805(+/- 2%) +54%
4            1.634(+/- 13%)      0.765(+/- 1%) +53%
8            1.305(+/-  4%)      0.745(+/- 2%) +43%
16           0.786(+/-  4%)      0.705(+/- 2%) +10%

By deacreasing the latency prio, we reduce the number of preemption at
wakeup and help hackbench making progress.

Test results of nice latency impact on short live load like cyclictest
while competing with heavy load like hackbench:

hackbench -l 10000 -g $group &
cyclictest --policy other -D 5 -q -n
        latency 0           latency -20
group   min  avg    max     min  avg    max
0       17    18     28      16   18     28
1       67   319   7369      63   76   2479
4       64   453  51699      45   95   8788
8       65   814  51699      63  116  19535
16      64  1275  37744      63  159  51134

group = 0 means that hackbench is not running.

The avg is significantly improved with nice latency -20 especially with
large number of groups but min and max remain quite similar. If we add the
histogram parameter to get details of latency, we have :

hackbench -l 10000 -g 16 &
cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
              latency 0    latency -20
Min Latencies:    63           63
Avg Latencies:  1459          214
Max Latencies: 58274        82358
50% latencies:   164           87
75% latencies:   848           91
85% latencies:  1246           94
90% latencies:  2149           96
95% latencies:  8463           99
99% latencies:>20000          120

With percentile details, we see the benefit of nice latency -20 as
only 1% of the latencies stays above 120us whereas the default latency
has got 25% are above 848us, 10% over the 2ms and 1% above 20ms.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 include/linux/sched.h |  4 +++-
 init/init_task.c      |  2 +-
 kernel/sched/core.c   | 38 +++++++++++++++++++++++++----
 kernel/sched/debug.c  |  2 +-
 kernel/sched/fair.c   | 56 +++++++++++++++++++++++++++++++++++++++++--
 kernel/sched/sched.h  | 14 ++++++++++-
 6 files changed, 105 insertions(+), 11 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 6805f378a9c3..a74cad08e91e 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -567,6 +567,8 @@ struct sched_entity {
 	/* cached value of my_q->h_nr_running */
 	unsigned long			runnable_weight;
 #endif
+	/* preemption offset in ns */
+	long				latency_offset;
 
 #ifdef CONFIG_SMP
 	/*
@@ -783,7 +785,7 @@ struct task_struct {
 	int				static_prio;
 	int				normal_prio;
 	unsigned int			rt_priority;
-	int				latency_nice;
+	int				latency_prio;
 
 	struct sched_entity		se;
 	struct sched_rt_entity		rt;
diff --git a/init/init_task.c b/init/init_task.c
index 7dd71dd2d261..b8ddf403bc62 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -78,7 +78,7 @@ struct task_struct init_task
 	.prio		= MAX_PRIO - 20,
 	.static_prio	= MAX_PRIO - 20,
 	.normal_prio	= MAX_PRIO - 20,
-	.latency_nice	= DEFAULT_LATENCY_NICE,
+	.latency_prio	= NICE_WIDTH - 20,
 	.policy		= SCHED_NORMAL,
 	.cpus_ptr	= &init_task.cpus_mask,
 	.user_cpus_ptr	= NULL,
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 6a6116ea4c2c..00fa2da12506 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1284,6 +1284,16 @@ static void set_load_weight(struct task_struct *p, bool update_load)
 	}
 }
 
+static void set_latency_offset(struct task_struct *p)
+{
+	long weight = sched_latency_to_weight[p->latency_prio];
+	s64 offset;
+
+	offset = sysctl_sched_latency * weight;
+	offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX);
+	p->se.latency_offset = (long)offset;
+}
+
 #ifdef CONFIG_UCLAMP_TASK
 /*
  * Serializes updates of utilization clamp values
@@ -4559,7 +4569,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
 		p->prio = p->normal_prio = p->static_prio;
 		set_load_weight(p, false);
 
-		p->latency_nice = DEFAULT_LATENCY_NICE;
+		p->latency_prio = NICE_TO_LATENCY(0);
+		set_latency_offset(p);
+
 		/*
 		 * We don't need the reset flag anymore after the fork. It has
 		 * fulfilled its duty:
@@ -7324,8 +7336,10 @@ static void __setscheduler_params(struct task_struct *p,
 static void __setscheduler_latency(struct task_struct *p,
 		const struct sched_attr *attr)
 {
-	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
-		p->latency_nice = attr->sched_latency_nice;
+	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
+		p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice);
+		set_latency_offset(p);
+	}
 }
 
 /*
@@ -7510,7 +7524,7 @@ static int __sched_setscheduler(struct task_struct *p,
 		if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
 			goto change;
 		if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
-		    attr->sched_latency_nice != p->latency_nice)
+		    attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio))
 			goto change;
 
 		p->sched_reset_on_fork = reset_on_fork;
@@ -8051,7 +8065,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
 	get_params(p, &kattr);
 	kattr.sched_flags &= SCHED_FLAG_ALL;
 
-	kattr.sched_latency_nice = p->latency_nice;
+	kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio);
 
 #ifdef CONFIG_UCLAMP_TASK
 	/*
@@ -11204,6 +11218,20 @@ const u32 sched_prio_to_wmult[40] = {
  /*  15 */ 119304647, 148102320, 186737708, 238609294, 286331153,
 };
 
+/*
+ * latency weight for wakeup preemption
+ */
+const int sched_latency_to_weight[40] = {
+ /* -20 */     -1024,     -973,     -922,      -870,      -819,
+ /* -15 */      -768,     -717,     -666,      -614,      -563,
+ /* -10 */      -512,     -461,     -410,      -358,      -307,
+ /*  -5 */      -256,     -205,     -154,      -102,       -51,
+ /*   0 */         0,       51,      102,       154,       205,
+ /*   5 */       256,      307,      358,       410,       461,
+ /*  10 */       512,      563,      614,       666,       717,
+ /*  15 */       768,      819,      870,       922,       973,
+};
+
 void call_trace_sched_update_nr_running(struct rq *rq, int count)
 {
         trace_sched_update_nr_running_tp(rq, count);
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index a3f7876217a6..06aaa0c81d4b 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1042,7 +1042,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 #endif
 	P(policy);
 	P(prio);
-	P(latency_nice);
+	P(latency_prio);
 	if (task_has_dl_policy(p)) {
 		P(dl.runtime);
 		P(dl.deadline);
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 5ffec4370602..c3f857630dcf 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4574,6 +4574,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 		update_idle_cfs_rq_clock_pelt(cfs_rq);
 }
 
+static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se);
+
 /*
  * Preempt the current task with a newly woken task if needed:
  */
@@ -4606,6 +4608,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 
 	se = __pick_first_entity(cfs_rq);
 	delta = curr->vruntime - se->vruntime;
+	delta -= wakeup_latency_gran(curr, se);
 
 	if (delta < 0)
 		return;
@@ -5732,6 +5735,35 @@ static int sched_idle_cpu(int cpu)
 }
 #endif
 
+static void set_next_buddy(struct sched_entity *se);
+
+static void check_preempt_from_others(struct cfs_rq *cfs, struct sched_entity *se)
+{
+	struct sched_entity *next;
+
+	if (se->latency_offset >= 0)
+		return;
+
+	if (cfs->nr_running <= 1)
+		return;
+	/*
+	 * When waking from another class, we don't need to check to preempt at
+	 * wakeup and don't set next buddy as a candidate for being picked in
+	 * priority.
+	 * In case of simultaneous wakeup when current is another class, the
+	 * latency sensitive tasks lost opportunity to preempt non sensitive
+	 * tasks which woke up simultaneously.
+	 */
+
+	if (cfs->next)
+		next = cfs->next;
+	else
+		next = __pick_first_entity(cfs);
+
+	if (next && wakeup_preempt_entity(next, se) == 1)
+		set_next_buddy(se);
+}
+
 /*
  * The enqueue_task method is called before nr_running is
  * increased. Here we update the fair scheduling stats and
@@ -5818,14 +5850,15 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 	if (!task_new)
 		update_overutilized_status(rq);
 
+	if (rq->curr->sched_class != &fair_sched_class)
+		check_preempt_from_others(cfs_rq_of(&p->se), &p->se);
+
 enqueue_throttle:
 	assert_list_leaf_cfs_rq(rq);
 
 	hrtick_update(rq);
 }
 
-static void set_next_buddy(struct sched_entity *se);
-
 /*
  * The dequeue_task method is called before nr_running is
  * decreased. We remove the task from the rbtree and
@@ -7144,6 +7177,22 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 }
 #endif /* CONFIG_SMP */
 
+static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se)
+{
+	long latency_offset = se->latency_offset;
+
+	/*
+	 * A negative latency offset means that the sched_entity has latency
+	 * requirement that needs to be evaluated versus other entity.
+	 * Otherwise, use the latency weight to evaluate how much scheduling
+	 * delay is acceptable by se.
+	 */
+	if ((se->latency_offset < 0) || (curr->latency_offset < 0))
+		latency_offset -= curr->latency_offset;
+
+	return latency_offset;
+}
+
 static unsigned long wakeup_gran(struct sched_entity *se)
 {
 	unsigned long gran = sysctl_sched_wakeup_granularity;
@@ -7183,6 +7232,9 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
 {
 	s64 gran, vdiff = curr->vruntime - se->vruntime;
 
+	/* Take into account latency priority */
+	vdiff -= wakeup_latency_gran(curr, se);
+
 	if (vdiff <= 0)
 		return -1;
 
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index eeb6efb0b610..80c4d2f5827f 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -142,6 +142,17 @@ extern int sched_rr_timeslice;
  * Default tasks should be treated as a task with latency_nice = 0.
  */
 #define DEFAULT_LATENCY_NICE	0
+#define DEFAULT_LATENCY_PRIO	(DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2)
+
+/*
+ * Convert user-nice values [ -20 ... 0 ... 19 ]
+ * to static latency [ 0..39 ],
+ * and back.
+ */
+#define NICE_TO_LATENCY(nice)	((nice) + DEFAULT_LATENCY_PRIO)
+#define LATENCY_TO_NICE(prio)	((prio) - DEFAULT_LATENCY_PRIO)
+#define NICE_LATENCY_SHIFT	(SCHED_FIXEDPOINT_SHIFT)
+#define NICE_LATENCY_WEIGHT_MAX	(1L << NICE_LATENCY_SHIFT)
 
 /*
  * Increase resolution of nice-level calculations for 64-bit architectures.
@@ -2116,6 +2127,7 @@ static_assert(WF_TTWU == SD_BALANCE_WAKE);
 
 extern const int		sched_prio_to_weight[40];
 extern const u32		sched_prio_to_wmult[40];
+extern const int		sched_latency_to_weight[40];
 
 /*
  * {de,en}queue flags:
@@ -2450,8 +2462,8 @@ extern void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags);
 extern const_debug unsigned int sysctl_sched_nr_migrate;
 extern const_debug unsigned int sysctl_sched_migration_cost;
 
-#ifdef CONFIG_SCHED_DEBUG
 extern unsigned int sysctl_sched_latency;
+#ifdef CONFIG_SCHED_DEBUG
 extern unsigned int sysctl_sched_min_granularity;
 extern unsigned int sysctl_sched_idle_min_granularity;
 extern unsigned int sysctl_sched_wakeup_granularity;
-- 
2.17.1

[PATCH v5 5/7] sched/fair: Add sched group latency support

[Vincent Guittot]

Task can set its latency priority with sched_setattr(), which is then used
to set the latency offset of its sched_entity, but sched group entities
still have the default latency offset value.

Add a latency.nice field in cpu cgroup controller to set the latency
priority of the group similarly to sched_setattr(). The latency priority
is then used to set the offset of the sched_entities of the group.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 Documentation/admin-guide/cgroup-v2.rst |  8 ++++
 kernel/sched/core.c                     | 53 +++++++++++++++++++++++++
 kernel/sched/fair.c                     | 33 +++++++++++++++
 kernel/sched/sched.h                    |  4 ++
 4 files changed, 98 insertions(+)

diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index be4a77baf784..d8ae7e411f9c 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1095,6 +1095,14 @@ All time durations are in microseconds.
         values similar to the sched_setattr(2). This maximum utilization
         value is used to clamp the task specific maximum utilization clamp.
 
+  cpu.latency.nice
+	A read-write single value file which exists on non-root
+	cgroups.  The default is "0".
+
+	The nice value is in the range [-20, 19].
+
+	This interface file allows reading and setting latency using the
+	same values used by sched_setattr(2).
 
 
 Memory
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 00fa2da12506..e8a1105bc87d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -10890,6 +10890,48 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,
 {
 	return sched_group_set_idle(css_tg(css), idle);
 }
+
+static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
+				    struct cftype *cft)
+{
+	int last_delta = INT_MAX;
+	int prio, delta;
+	s64 weight;
+
+	weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX;
+	weight = div_s64(weight, sysctl_sched_latency);
+
+	/* Find the closest nice value to the current weight */
+	for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) {
+		delta = abs(sched_latency_to_weight[prio] - weight);
+		if (delta >= last_delta)
+			break;
+		last_delta = delta;
+	}
+
+	return LATENCY_TO_NICE(prio-1);
+}
+
+static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
+				     struct cftype *cft, s64 nice)
+{
+	s64 latency_offset;
+	long weight;
+	int idx;
+
+	if (nice < MIN_LATENCY_NICE || nice > MAX_LATENCY_NICE)
+		return -ERANGE;
+
+	idx = NICE_TO_LATENCY(nice);
+	idx = array_index_nospec(idx, LATENCY_NICE_WIDTH);
+	weight = sched_latency_to_weight[idx];
+
+	latency_offset = sysctl_sched_latency * weight;
+	latency_offset = div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX);
+
+	return sched_group_set_latency(css_tg(css), latency_offset);
+}
+
 #endif
 
 static struct cftype cpu_legacy_files[] = {
@@ -10904,6 +10946,11 @@ static struct cftype cpu_legacy_files[] = {
 		.read_s64 = cpu_idle_read_s64,
 		.write_s64 = cpu_idle_write_s64,
 	},
+	{
+		.name = "latency.nice",
+		.read_s64 = cpu_latency_nice_read_s64,
+		.write_s64 = cpu_latency_nice_write_s64,
+	},
 #endif
 #ifdef CONFIG_CFS_BANDWIDTH
 	{
@@ -11121,6 +11168,12 @@ static struct cftype cpu_files[] = {
 		.read_s64 = cpu_idle_read_s64,
 		.write_s64 = cpu_idle_write_s64,
 	},
+	{
+		.name = "latency.nice",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.read_s64 = cpu_latency_nice_read_s64,
+		.write_s64 = cpu_latency_nice_write_s64,
+	},
 #endif
 #ifdef CONFIG_CFS_BANDWIDTH
 	{
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c3f857630dcf..74e42d19c1ce 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11768,6 +11768,7 @@ int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
 		goto err;
 
 	tg->shares = NICE_0_LOAD;
+	tg->latency_offset = 0;
 
 	init_cfs_bandwidth(tg_cfs_bandwidth(tg));
 
@@ -11866,6 +11867,9 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
 	}
 
 	se->my_q = cfs_rq;
+
+	se->latency_offset = tg->latency_offset;
+
 	/* guarantee group entities always have weight */
 	update_load_set(&se->load, NICE_0_LOAD);
 	se->parent = parent;
@@ -11996,6 +12000,35 @@ int sched_group_set_idle(struct task_group *tg, long idle)
 	return 0;
 }
 
+int sched_group_set_latency(struct task_group *tg, s64 latency)
+{
+	int i;
+
+	if (tg == &root_task_group)
+		return -EINVAL;
+
+	if (abs(latency) > sysctl_sched_latency)
+		return -EINVAL;
+
+	mutex_lock(&shares_mutex);
+
+	if (tg->latency_offset == latency) {
+		mutex_unlock(&shares_mutex);
+		return 0;
+	}
+
+	tg->latency_offset = latency;
+
+	for_each_possible_cpu(i) {
+		struct sched_entity *se = tg->se[i];
+
+		WRITE_ONCE(se->latency_offset, latency);
+	}
+
+	mutex_unlock(&shares_mutex);
+	return 0;
+}
+
 #else /* CONFIG_FAIR_GROUP_SCHED */
 
 void free_fair_sched_group(struct task_group *tg) { }
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 80c4d2f5827f..a15fb955092c 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -407,6 +407,8 @@ struct task_group {
 
 	/* A positive value indicates that this is a SCHED_IDLE group. */
 	int			idle;
+	/* latency constraint of the group. */
+	int			latency_offset;
 
 #ifdef	CONFIG_SMP
 	/*
@@ -517,6 +519,8 @@ extern int sched_group_set_shares(struct task_group *tg, unsigned long shares);
 
 extern int sched_group_set_idle(struct task_group *tg, long idle);
 
+extern int sched_group_set_latency(struct task_group *tg, s64 latency);
+
 #ifdef CONFIG_SMP
 extern void set_task_rq_fair(struct sched_entity *se,
 			     struct cfs_rq *prev, struct cfs_rq *next);
-- 
2.17.1

[PATCH v5 6/7] sched/core: Support latency priority with sched core

[Vincent Guittot]

Take into account wakeup_latency_gran() when ordering the cfs threads.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 kernel/sched/fair.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 74e42d19c1ce..e524e892d118 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11443,6 +11443,10 @@ bool cfs_prio_less(struct task_struct *a, struct task_struct *b, bool in_fi)
 	delta = (s64)(sea->vruntime - seb->vruntime) +
 		(s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi);
 
+	/* Take into account latency prio */
+	delta -= wakeup_latency_gran(sea, seb);
+
+
 	return delta > 0;
 }
 #else
-- 
2.17.1

[PATCH v5 7/7] sched/fair: Add latency list

[Vincent Guittot]

Add a rb tree for latency sensitive entities so we can schedule the most
sensitive one first even when it failed to preempt current at wakeup or
when it got quickly preempted by another entity of higher priority.

In order to keep fairness, the latency is used once at wakeup to get a
minimum slice and not during the following scheduling slice to prevent
long running entity to got more running time than allocated to his nice
priority.

The rb tree nebales to cover the last corner case where latency
sensitive entity can't got schedule quickly after the wakeup.

hackbench -l 10000 -g $group &
cyclictest --policy other -D 5 -q -n
        latency 0           latency -20
group   min  avg    max     min  avg    max
0       17    19     29      17   18     30
1       65   306   7149      64   83    208
4       50   395  15731      56   80    271
8       56   781  41548      54   80    301
16      60  1392  87237      59   86    490

group = 0 means that hackbench is not running.

Both avg and max are significantly improved with nice latency -20. If we
add the histogram parameters to get details of latency, we have :

hackbench -l 10000 -g 16 &
cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
              latency 0    latency -20
Min Latencies:    60           61
Avg Latencies:  1077           86
Max Latencies: 87311          444
50% latencies:    92           85
75% latencies:   554           90
85% latencies:  1019           93
90% latencies:  1346           96
95% latencies:  5400          100
99% latencies: 19044          110

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
---
 include/linux/sched.h |  2 +
 kernel/sched/fair.c   | 96 +++++++++++++++++++++++++++++++++++++++++--
 kernel/sched/sched.h  |  1 +
 3 files changed, 96 insertions(+), 3 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index a74cad08e91e..0b92674e3664 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -547,6 +547,8 @@ struct sched_entity {
 	/* For load-balancing: */
 	struct load_weight		load;
 	struct rb_node			run_node;
+	struct rb_node			latency_node;
+	unsigned int			on_latency;
 	struct list_head		group_node;
 	unsigned int			on_rq;
 
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e524e892d118..1a72f34136d8 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
 
 	return __node_2_se(last);
 }
+#endif
 
+/**************************************************************
+ * Scheduling class tree data structure manipulation methods:
+ * for latency
+ */
+
+static inline bool latency_before(struct sched_entity *a,
+				struct sched_entity *b)
+{
+	return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
+}
+
+#define __latency_node_2_se(node) \
+	rb_entry((node), struct sched_entity, latency_node)
+
+static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
+{
+	return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
+}
+
+/*
+ * Enqueue an entity into the latency rb-tree:
+ */
+static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
+{
+
+	/* Only latency sensitive entity can be added to the list */
+	if (se->latency_offset >= 0)
+		return;
+
+	if (se->on_latency)
+		return;
+
+	/*
+	 * An execution time less than sysctl_sched_min_granularity means that
+	 * the entity has been preempted by a higher sched class or an entity
+	 * with higher latency constraint.
+	 * Put it back in the list so it gets a chance to run 1st during the
+	 * next slice.
+	 */
+	if (!(flags & ENQUEUE_WAKEUP)) {
+		u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
+
+		if (delta_exec >= sysctl_sched_min_granularity)
+			return;
+	}
+
+	rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
+	se->on_latency = 1;
+}
+
+static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
+{
+	if (se->on_latency) {
+		rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
+		se->on_latency = 0;
+	}
+}
+
+static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
+{
+	struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
+
+	if (!left)
+		return NULL;
+
+	return __latency_node_2_se(left);
+}
+
+#ifdef CONFIG_SCHED_DEBUG
 /**************************************************************
  * Scheduling class statistics methods:
  */
@@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	check_schedstat_required();
 	update_stats_enqueue_fair(cfs_rq, se, flags);
 	check_spread(cfs_rq, se);
-	if (!curr)
+	if (!curr) {
 		__enqueue_entity(cfs_rq, se);
+		__enqueue_latency(cfs_rq, se, flags);
+	}
 	se->on_rq = 1;
 
 	if (cfs_rq->nr_running == 1) {
@@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
 	clear_buddies(cfs_rq, se);
 
-	if (se != cfs_rq->curr)
+	if (se != cfs_rq->curr) {
 		__dequeue_entity(cfs_rq, se);
+		__dequeue_latency(cfs_rq, se);
+	}
 	se->on_rq = 0;
 	account_entity_dequeue(cfs_rq, se);
 
@@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 		 */
 		update_stats_wait_end_fair(cfs_rq, se);
 		__dequeue_entity(cfs_rq, se);
+		__dequeue_latency(cfs_rq, se);
 		update_load_avg(cfs_rq, se, UPDATE_TG);
 	}
 
@@ -4669,7 +4744,7 @@ static struct sched_entity *
 pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 {
 	struct sched_entity *left = __pick_first_entity(cfs_rq);
-	struct sched_entity *se;
+	struct sched_entity *latency, *se;
 
 	/*
 	 * If curr is set we have to see if its left of the leftmost entity
@@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 		se = cfs_rq->last;
 	}
 
+	/* Check for latency sensitive entity waiting for running */
+	latency = __pick_first_latency(cfs_rq);
+	if (latency && (latency != se) &&
+	    wakeup_preempt_entity(latency, se) < 1)
+		se = latency;
+
 	return se;
 }
 
@@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
 		update_stats_wait_start_fair(cfs_rq, prev);
 		/* Put 'current' back into the tree. */
 		__enqueue_entity(cfs_rq, prev);
+		__enqueue_latency(cfs_rq, prev, 0);
 		/* in !on_rq case, update occurred at dequeue */
 		update_load_avg(cfs_rq, prev, 0);
 	}
@@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 void init_cfs_rq(struct cfs_rq *cfs_rq)
 {
 	cfs_rq->tasks_timeline = RB_ROOT_CACHED;
+	cfs_rq->latency_timeline = RB_ROOT_CACHED;
 	u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
 #ifdef CONFIG_SMP
 	raw_spin_lock_init(&cfs_rq->removed.lock);
@@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency)
 
 	for_each_possible_cpu(i) {
 		struct sched_entity *se = tg->se[i];
+		struct rq *rq = cpu_rq(i);
+		struct rq_flags rf;
+
+		rq_lock_irqsave(rq, &rf);
 
+		__dequeue_latency(se->cfs_rq, se);
 		WRITE_ONCE(se->latency_offset, latency);
+
+		rq_unlock_irqrestore(rq, &rf);
 	}
 
 	mutex_unlock(&shares_mutex);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a15fb955092c..76bca172585c 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -599,6 +599,7 @@ struct cfs_rq {
 #endif
 
 	struct rb_root_cached	tasks_timeline;
+	struct rb_root_cached	latency_timeline;
 
 	/*
 	 * 'curr' points to currently running entity on this cfs_rq.
-- 
2.17.1

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Youssef Esmat]

Hi Vincent,


On Sun, Sep 25, 2022 at 9:39 AM Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> Add a rb tree for latency sensitive entities so we can schedule the most
> sensitive one first even when it failed to preempt current at wakeup or
> when it got quickly preempted by another entity of higher priority.
>
> In order to keep fairness, the latency is used once at wakeup to get a
> minimum slice and not during the following scheduling slice to prevent
> long running entity to got more running time than allocated to his nice
> priority.
>
> The rb tree nebales to cover the last corner case where latency
> sensitive entity can't got schedule quickly after the wakeup.
>
> hackbench -l 10000 -g $group &
> cyclictest --policy other -D 5 -q -n
>         latency 0           latency -20
> group   min  avg    max     min  avg    max
> 0       17    19     29      17   18     30
> 1       65   306   7149      64   83    208
> 4       50   395  15731      56   80    271
> 8       56   781  41548      54   80    301
> 16      60  1392  87237      59   86    490
>
> group = 0 means that hackbench is not running.
>
> Both avg and max are significantly improved with nice latency -20. If we
> add the histogram parameters to get details of latency, we have :
>
> hackbench -l 10000 -g 16 &
> cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
>               latency 0    latency -20
> Min Latencies:    60           61
> Avg Latencies:  1077           86
> Max Latencies: 87311          444
> 50% latencies:    92           85
> 75% latencies:   554           90
> 85% latencies:  1019           93
> 90% latencies:  1346           96
> 95% latencies:  5400          100
> 99% latencies: 19044          110
>
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> ---

The ability to boost the latency sensitivity of a task seems very
interesting. I have been playing around with these changes and have
some observations.

I tried 2 bursty tasks affinitized to the same CPU. The tasks sleep
for 1ms and run for 10ms in a loop. I first tried it without adjusting
the latency_nice value and took perf sched traces:

latency_test:7040     |   2447.137 ms |        8 | avg:   6.546 ms |
max:  10.674 ms | max start:   353.809487 s | max end:   353.820161 s
latency_test:7028     |   2454.777 ms |        7 | avg:   4.494 ms |
max:  10.609 ms | max start:   354.804386 s | max end:   354.814995 s

Everything looked as expected, for a 5s run they had similar runtime
and latency.

I then adjusted one task to have a latency_nice of -20 (pid 8614
below) and took another set of traces:

latency_test:8618     |   1845.534 ms |      131 | avg:   9.764 ms |
max:  10.686 ms | max start:  1405.737905 s | max end:  1405.748592 s
latency_test:8614     |   3033.635 ms |       16 | avg:   3.559 ms |
max:  10.467 ms | max start:  1407.594751 s | max end:  1407.605218 s

The task with -20 latency_nice had significantly more runtime. The
average latency was improved but the max roughly stayed the same. As
expected the one with latency_nice value of 0 experienced more
switches, but so did the one with latency_nice of -20.

Also tried running the same test but instead of using latency nice I
adjusted the nice value as a comparison. In that case one task had a
nice of -5 and the other was 0.

nice_test:25219       |   1216.839 ms |      242 | avg:  10.295 ms |
max:  11.927 ms | max start:  5877.881279 s | max end:  5877.893206 s
nice_test:25235       |   3711.788 ms |        6 | avg:   1.026 ms |
max:   6.143 ms | max start:  5875.603741 s | max end:  5875.609883 s

As expected the one with a nice value of -5 had more runtime but also
had better latency numbers than in the previous case of using
latency_nice.

I also tried a similar test with 3 bursty tasks instead of two. In
this case all tasks had a latency_nice of 0:

latency_test:11467    |   1641.131 ms |      161 | avg:  17.489 ms |
max:  21.011 ms | max start:  1542.656275 s | max end:  1542.677286 s
latency_test:11463    |   1644.809 ms |      161 | avg:  11.994 ms |
max:  25.012 ms | max start:  1545.657776 s | max end:  1545.682788 s
latency_test:11478    |   1643.211 ms |      160 | avg:  11.465 ms |
max:  21.012 ms | max start:  1546.159026 s | max end:  1546.180038 s

Next I tried two tasks with a latency_nice of 0 and a third one had a
latency_nice of -20 (pid 11763 below):

latency_test:11763    |   1645.482 ms |      159 | avg:  19.634 ms |
max:  31.016 ms | max start:  1623.834862 s | max end:  1623.865877 s
latency_test:11750    |   1644.276 ms |      259 | avg:   9.985 ms |
max:  21.012 ms | max start:  1623.953921 s | max end:  1623.974933 s
latency_test:11747    |   1642.745 ms |      262 | avg:   9.079 ms |
max:  25.013 ms | max start:  1620.980435 s | max end:  1621.005447 s

In this case it seemed like the runtime was not affected by the
latency_nice value, but strangely the task with the latency nice of
-20 had a worse average and max latency than the other two. The
context switch times are also increased from the previous case.

Have we considered an approach where the task that is marked as
latency sensitive gets a boosted nice value when it sleeps and is
either scaled down exponentially as it runs, or immediately reset to
its default when it runs for one tick?

Thanks,
Youssef


>  include/linux/sched.h |  2 +
>  kernel/sched/fair.c   | 96 +++++++++++++++++++++++++++++++++++++++++--
>  kernel/sched/sched.h  |  1 +
>  3 files changed, 96 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index a74cad08e91e..0b92674e3664 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -547,6 +547,8 @@ struct sched_entity {
>         /* For load-balancing: */
>         struct load_weight              load;
>         struct rb_node                  run_node;
> +       struct rb_node                  latency_node;
> +       unsigned int                    on_latency;
>         struct list_head                group_node;
>         unsigned int                    on_rq;
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index e524e892d118..1a72f34136d8 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
>
>         return __node_2_se(last);
>  }
> +#endif
>
> +/**************************************************************
> + * Scheduling class tree data structure manipulation methods:
> + * for latency
> + */
> +
> +static inline bool latency_before(struct sched_entity *a,
> +                               struct sched_entity *b)
> +{
> +       return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> +}
> +
> +#define __latency_node_2_se(node) \
> +       rb_entry((node), struct sched_entity, latency_node)
> +
> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> +{
> +       return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> +}
> +
> +/*
> + * Enqueue an entity into the latency rb-tree:
> + */
> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> +{
> +
> +       /* Only latency sensitive entity can be added to the list */
> +       if (se->latency_offset >= 0)
> +               return;
> +
> +       if (se->on_latency)
> +               return;
> +
> +       /*
> +        * An execution time less than sysctl_sched_min_granularity means that
> +        * the entity has been preempted by a higher sched class or an entity
> +        * with higher latency constraint.
> +        * Put it back in the list so it gets a chance to run 1st during the
> +        * next slice.
> +        */
> +       if (!(flags & ENQUEUE_WAKEUP)) {
> +               u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> +
> +               if (delta_exec >= sysctl_sched_min_granularity)
> +                       return;
> +       }
> +
> +       rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> +       se->on_latency = 1;
> +}
> +
> +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> +{
> +       if (se->on_latency) {
> +               rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> +               se->on_latency = 0;
> +       }
> +}
> +
> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> +{
> +       struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> +
> +       if (!left)
> +               return NULL;
> +
> +       return __latency_node_2_se(left);
> +}
> +
> +#ifdef CONFIG_SCHED_DEBUG
>  /**************************************************************
>   * Scheduling class statistics methods:
>   */
> @@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>         check_schedstat_required();
>         update_stats_enqueue_fair(cfs_rq, se, flags);
>         check_spread(cfs_rq, se);
> -       if (!curr)
> +       if (!curr) {
>                 __enqueue_entity(cfs_rq, se);
> +               __enqueue_latency(cfs_rq, se, flags);
> +       }
>         se->on_rq = 1;
>
>         if (cfs_rq->nr_running == 1) {
> @@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>
>         clear_buddies(cfs_rq, se);
>
> -       if (se != cfs_rq->curr)
> +       if (se != cfs_rq->curr) {
>                 __dequeue_entity(cfs_rq, se);
> +               __dequeue_latency(cfs_rq, se);
> +       }
>         se->on_rq = 0;
>         account_entity_dequeue(cfs_rq, se);
>
> @@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>                  */
>                 update_stats_wait_end_fair(cfs_rq, se);
>                 __dequeue_entity(cfs_rq, se);
> +               __dequeue_latency(cfs_rq, se);
>                 update_load_avg(cfs_rq, se, UPDATE_TG);
>         }
>
> @@ -4669,7 +4744,7 @@ static struct sched_entity *
>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>  {
>         struct sched_entity *left = __pick_first_entity(cfs_rq);
> -       struct sched_entity *se;
> +       struct sched_entity *latency, *se;
>
>         /*
>          * If curr is set we have to see if its left of the leftmost entity
> @@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>                 se = cfs_rq->last;
>         }
>
> +       /* Check for latency sensitive entity waiting for running */
> +       latency = __pick_first_latency(cfs_rq);
> +       if (latency && (latency != se) &&
> +           wakeup_preempt_entity(latency, se) < 1)
> +               se = latency;
> +
>         return se;
>  }
>
> @@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
>                 update_stats_wait_start_fair(cfs_rq, prev);
>                 /* Put 'current' back into the tree. */
>                 __enqueue_entity(cfs_rq, prev);
> +               __enqueue_latency(cfs_rq, prev, 0);
>                 /* in !on_rq case, update occurred at dequeue */
>                 update_load_avg(cfs_rq, prev, 0);
>         }
> @@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>  void init_cfs_rq(struct cfs_rq *cfs_rq)
>  {
>         cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> +       cfs_rq->latency_timeline = RB_ROOT_CACHED;
>         u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
>  #ifdef CONFIG_SMP
>         raw_spin_lock_init(&cfs_rq->removed.lock);
> @@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency)
>
>         for_each_possible_cpu(i) {
>                 struct sched_entity *se = tg->se[i];
> +               struct rq *rq = cpu_rq(i);
> +               struct rq_flags rf;
> +
> +               rq_lock_irqsave(rq, &rf);
>
> +               __dequeue_latency(se->cfs_rq, se);
>                 WRITE_ONCE(se->latency_offset, latency);
> +
> +               rq_unlock_irqrestore(rq, &rf);
>         }
>
>         mutex_unlock(&shares_mutex);
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index a15fb955092c..76bca172585c 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -599,6 +599,7 @@ struct cfs_rq {
>  #endif
>
>         struct rb_root_cached   tasks_timeline;
> +       struct rb_root_cached   latency_timeline;
>
>         /*
>          * 'curr' points to currently running entity on this cfs_rq.
> --
> 2.17.1
>
>
>

RE: [PATCH v5 7/7] sched/fair: Add latency list

[David Laight]

...
> Have we considered an approach where the task that is marked as
> latency sensitive gets a boosted nice value when it sleeps and is
> either scaled down exponentially as it runs, or immediately reset to
> its default when it runs for one tick?

Or use the RT scheduler for anything that requires low latency.
Isn't that what it is for?

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

RE: [PATCH v5 7/7] sched/fair: Add latency list

[Steven Rostedt]

On October 8, 2022 5:14:54 PM EDT, David Laight <David.Laight@ACULAB.COM> wrote:
>...
>> Have we considered an approach where the task that is marked as
>> latency sensitive gets a boosted nice value when it sleeps and is
>> either scaled down exponentially as it runs, or immediately reset to
>> its default when it runs for one tick?
>
>Or use the RT scheduler for anything that requires low latency.
>Isn't that what it is for?
>

The RT scheduler is for tasks that require strick scheduling requirements over all else, including performance.

But we are seeing latency issues for tasks that don't have such requirements but this latency effects its quality.

-- Steve

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity and top posting.

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Vincent Guittot]

Hi Youssef,

On Sat, 8 Oct 2022 at 03:05, Youssef Esmat <youssefesmat@chromium.org> wrote:
>
> Hi Vincent,
>
>
> On Sun, Sep 25, 2022 at 9:39 AM Vincent Guittot
> <vincent.guittot@linaro.org> wrote:
> >
> > Add a rb tree for latency sensitive entities so we can schedule the most
> > sensitive one first even when it failed to preempt current at wakeup or
> > when it got quickly preempted by another entity of higher priority.
> >
> > In order to keep fairness, the latency is used once at wakeup to get a
> > minimum slice and not during the following scheduling slice to prevent
> > long running entity to got more running time than allocated to his nice
> > priority.
> >
> > The rb tree nebales to cover the last corner case where latency
> > sensitive entity can't got schedule quickly after the wakeup.
> >
> > hackbench -l 10000 -g $group &
> > cyclictest --policy other -D 5 -q -n
> >         latency 0           latency -20
> > group   min  avg    max     min  avg    max
> > 0       17    19     29      17   18     30
> > 1       65   306   7149      64   83    208
> > 4       50   395  15731      56   80    271
> > 8       56   781  41548      54   80    301
> > 16      60  1392  87237      59   86    490
> >
> > group = 0 means that hackbench is not running.
> >
> > Both avg and max are significantly improved with nice latency -20. If we
> > add the histogram parameters to get details of latency, we have :
> >
> > hackbench -l 10000 -g 16 &
> > cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
> >               latency 0    latency -20
> > Min Latencies:    60           61
> > Avg Latencies:  1077           86
> > Max Latencies: 87311          444
> > 50% latencies:    92           85
> > 75% latencies:   554           90
> > 85% latencies:  1019           93
> > 90% latencies:  1346           96
> > 95% latencies:  5400          100
> > 99% latencies: 19044          110
> >
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > ---
>
> The ability to boost the latency sensitivity of a task seems very
> interesting. I have been playing around with these changes and have
> some observations.
>
> I tried 2 bursty tasks affinitized to the same CPU. The tasks sleep
> for 1ms and run for 10ms in a loop. I first tried it without adjusting
> the latency_nice value and took perf sched traces:

The CPU is overloaded almost all the time as it can't run the 2 tasks
(2*10ms every 11ms)

>
> latency_test:7040     |   2447.137 ms |        8 | avg:   6.546 ms |
> max:  10.674 ms | max start:   353.809487 s | max end:   353.820161 s
> latency_test:7028     |   2454.777 ms |        7 | avg:   4.494 ms |
> max:  10.609 ms | max start:   354.804386 s | max end:   354.814995 s
>
> Everything looked as expected, for a 5s run they had similar runtime
> and latency.
>
> I then adjusted one task to have a latency_nice of -20 (pid 8614
> below) and took another set of traces:
>
> latency_test:8618     |   1845.534 ms |      131 | avg:   9.764 ms |
> max:  10.686 ms | max start:  1405.737905 s | max end:  1405.748592 s
> latency_test:8614     |   3033.635 ms |       16 | avg:   3.559 ms |
> max:  10.467 ms | max start:  1407.594751 s | max end:  1407.605218 s
>
> The task with -20 latency_nice had significantly more runtime. The
> average latency was improved but the max roughly stayed the same. As
> expected the one with latency_nice value of 0 experienced more
> switches, but so did the one with latency_nice of -20.

Your results look unexpected because the vruntime of the tasks is not
modified. So I can imagine that the thread with the low latency runs
first up to the offset at the beg of the test but then they should
switch regularly. I have tried a similar test with a modified rt-app
and the result seems ok. I have a small difference but not the
difference that you see.

Could you share more details about your setup ? I'm going to try to
reproduce your sequence

>
> Also tried running the same test but instead of using latency nice I
> adjusted the nice value as a comparison. In that case one task had a
> nice of -5 and the other was 0.
>
> nice_test:25219       |   1216.839 ms |      242 | avg:  10.295 ms |
> max:  11.927 ms | max start:  5877.881279 s | max end:  5877.893206 s
> nice_test:25235       |   3711.788 ms |        6 | avg:   1.026 ms |
> max:   6.143 ms | max start:  5875.603741 s | max end:  5875.609883 s
>
> As expected the one with a nice value of -5 had more runtime but also
> had better latency numbers than in the previous case of using
> latency_nice.
>
> I also tried a similar test with 3 bursty tasks instead of two. In
> this case all tasks had a latency_nice of 0:
>
> latency_test:11467    |   1641.131 ms |      161 | avg:  17.489 ms |
> max:  21.011 ms | max start:  1542.656275 s | max end:  1542.677286 s
> latency_test:11463    |   1644.809 ms |      161 | avg:  11.994 ms |
> max:  25.012 ms | max start:  1545.657776 s | max end:  1545.682788 s
> latency_test:11478    |   1643.211 ms |      160 | avg:  11.465 ms |
> max:  21.012 ms | max start:  1546.159026 s | max end:  1546.180038 s
>
> Next I tried two tasks with a latency_nice of 0 and a third one had a
> latency_nice of -20 (pid 11763 below):
>
> latency_test:11763    |   1645.482 ms |      159 | avg:  19.634 ms |
> max:  31.016 ms | max start:  1623.834862 s | max end:  1623.865877 s
> latency_test:11750    |   1644.276 ms |      259 | avg:   9.985 ms |
> max:  21.012 ms | max start:  1623.953921 s | max end:  1623.974933 s
> latency_test:11747    |   1642.745 ms |      262 | avg:   9.079 ms |
> max:  25.013 ms | max start:  1620.980435 s | max end:  1621.005447 s
>
> In this case it seemed like the runtime was not affected by the
> latency_nice value, but strangely the task with the latency nice of
> -20 had a worse average and max latency than the other two. The
> context switch times are also increased from the previous case.
>
> Have we considered an approach where the task that is marked as
> latency sensitive gets a boosted nice value when it sleeps and is
> either scaled down exponentially as it runs, or immediately reset to
> its default when it runs for one tick?
>
> Thanks,
> Youssef
>
>
> >  include/linux/sched.h |  2 +
> >  kernel/sched/fair.c   | 96 +++++++++++++++++++++++++++++++++++++++++--
> >  kernel/sched/sched.h  |  1 +
> >  3 files changed, 96 insertions(+), 3 deletions(-)
> >
> > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > index a74cad08e91e..0b92674e3664 100644
> > --- a/include/linux/sched.h
> > +++ b/include/linux/sched.h
> > @@ -547,6 +547,8 @@ struct sched_entity {
> >         /* For load-balancing: */
> >         struct load_weight              load;
> >         struct rb_node                  run_node;
> > +       struct rb_node                  latency_node;
> > +       unsigned int                    on_latency;
> >         struct list_head                group_node;
> >         unsigned int                    on_rq;
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index e524e892d118..1a72f34136d8 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> >
> >         return __node_2_se(last);
> >  }
> > +#endif
> >
> > +/**************************************************************
> > + * Scheduling class tree data structure manipulation methods:
> > + * for latency
> > + */
> > +
> > +static inline bool latency_before(struct sched_entity *a,
> > +                               struct sched_entity *b)
> > +{
> > +       return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> > +}
> > +
> > +#define __latency_node_2_se(node) \
> > +       rb_entry((node), struct sched_entity, latency_node)
> > +
> > +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> > +{
> > +       return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> > +}
> > +
> > +/*
> > + * Enqueue an entity into the latency rb-tree:
> > + */
> > +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > +{
> > +
> > +       /* Only latency sensitive entity can be added to the list */
> > +       if (se->latency_offset >= 0)
> > +               return;
> > +
> > +       if (se->on_latency)
> > +               return;
> > +
> > +       /*
> > +        * An execution time less than sysctl_sched_min_granularity means that
> > +        * the entity has been preempted by a higher sched class or an entity
> > +        * with higher latency constraint.
> > +        * Put it back in the list so it gets a chance to run 1st during the
> > +        * next slice.
> > +        */
> > +       if (!(flags & ENQUEUE_WAKEUP)) {
> > +               u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> > +
> > +               if (delta_exec >= sysctl_sched_min_granularity)
> > +                       return;
> > +       }
> > +
> > +       rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> > +       se->on_latency = 1;
> > +}
> > +
> > +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > +{
> > +       if (se->on_latency) {
> > +               rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> > +               se->on_latency = 0;
> > +       }
> > +}
> > +
> > +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> > +{
> > +       struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> > +
> > +       if (!left)
> > +               return NULL;
> > +
> > +       return __latency_node_2_se(left);
> > +}
> > +
> > +#ifdef CONFIG_SCHED_DEBUG
> >  /**************************************************************
> >   * Scheduling class statistics methods:
> >   */
> > @@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >         check_schedstat_required();
> >         update_stats_enqueue_fair(cfs_rq, se, flags);
> >         check_spread(cfs_rq, se);
> > -       if (!curr)
> > +       if (!curr) {
> >                 __enqueue_entity(cfs_rq, se);
> > +               __enqueue_latency(cfs_rq, se, flags);
> > +       }
> >         se->on_rq = 1;
> >
> >         if (cfs_rq->nr_running == 1) {
> > @@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >
> >         clear_buddies(cfs_rq, se);
> >
> > -       if (se != cfs_rq->curr)
> > +       if (se != cfs_rq->curr) {
> >                 __dequeue_entity(cfs_rq, se);
> > +               __dequeue_latency(cfs_rq, se);
> > +       }
> >         se->on_rq = 0;
> >         account_entity_dequeue(cfs_rq, se);
> >
> > @@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >                  */
> >                 update_stats_wait_end_fair(cfs_rq, se);
> >                 __dequeue_entity(cfs_rq, se);
> > +               __dequeue_latency(cfs_rq, se);
> >                 update_load_avg(cfs_rq, se, UPDATE_TG);
> >         }
> >
> > @@ -4669,7 +4744,7 @@ static struct sched_entity *
> >  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >  {
> >         struct sched_entity *left = __pick_first_entity(cfs_rq);
> > -       struct sched_entity *se;
> > +       struct sched_entity *latency, *se;
> >
> >         /*
> >          * If curr is set we have to see if its left of the leftmost entity
> > @@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >                 se = cfs_rq->last;
> >         }
> >
> > +       /* Check for latency sensitive entity waiting for running */
> > +       latency = __pick_first_latency(cfs_rq);
> > +       if (latency && (latency != se) &&
> > +           wakeup_preempt_entity(latency, se) < 1)
> > +               se = latency;
> > +
> >         return se;
> >  }
> >
> > @@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> >                 update_stats_wait_start_fair(cfs_rq, prev);
> >                 /* Put 'current' back into the tree. */
> >                 __enqueue_entity(cfs_rq, prev);
> > +               __enqueue_latency(cfs_rq, prev, 0);
> >                 /* in !on_rq case, update occurred at dequeue */
> >                 update_load_avg(cfs_rq, prev, 0);
> >         }
> > @@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> >  void init_cfs_rq(struct cfs_rq *cfs_rq)
> >  {
> >         cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> > +       cfs_rq->latency_timeline = RB_ROOT_CACHED;
> >         u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> >  #ifdef CONFIG_SMP
> >         raw_spin_lock_init(&cfs_rq->removed.lock);
> > @@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency)
> >
> >         for_each_possible_cpu(i) {
> >                 struct sched_entity *se = tg->se[i];
> > +               struct rq *rq = cpu_rq(i);
> > +               struct rq_flags rf;
> > +
> > +               rq_lock_irqsave(rq, &rf);
> >
> > +               __dequeue_latency(se->cfs_rq, se);
> >                 WRITE_ONCE(se->latency_offset, latency);
> > +
> > +               rq_unlock_irqrestore(rq, &rf);
> >         }
> >
> >         mutex_unlock(&shares_mutex);
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index a15fb955092c..76bca172585c 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -599,6 +599,7 @@ struct cfs_rq {
> >  #endif
> >
> >         struct rb_root_cached   tasks_timeline;
> > +       struct rb_root_cached   latency_timeline;
> >
> >         /*
> >          * 'curr' points to currently running entity on this cfs_rq.
> > --
> > 2.17.1
> >
> >
> >

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Youssef Esmat]

Hi Vincent,

On Tue, Oct 11, 2022 at 12:10 PM Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> Hi Youssef,
>
> On Sat, 8 Oct 2022 at 03:05, Youssef Esmat <youssefesmat@chromium.org> wrote:
> >
> > Hi Vincent,
> >
> >
> > On Sun, Sep 25, 2022 at 9:39 AM Vincent Guittot
> > <vincent.guittot@linaro.org> wrote:
> > >
> > > Add a rb tree for latency sensitive entities so we can schedule the most
> > > sensitive one first even when it failed to preempt current at wakeup or
> > > when it got quickly preempted by another entity of higher priority.
> > >
> > > In order to keep fairness, the latency is used once at wakeup to get a
> > > minimum slice and not during the following scheduling slice to prevent
> > > long running entity to got more running time than allocated to his nice
> > > priority.
> > >
> > > The rb tree nebales to cover the last corner case where latency
> > > sensitive entity can't got schedule quickly after the wakeup.
> > >
> > > hackbench -l 10000 -g $group &
> > > cyclictest --policy other -D 5 -q -n
> > >         latency 0           latency -20
> > > group   min  avg    max     min  avg    max
> > > 0       17    19     29      17   18     30
> > > 1       65   306   7149      64   83    208
> > > 4       50   395  15731      56   80    271
> > > 8       56   781  41548      54   80    301
> > > 16      60  1392  87237      59   86    490
> > >
> > > group = 0 means that hackbench is not running.
> > >
> > > Both avg and max are significantly improved with nice latency -20. If we
> > > add the histogram parameters to get details of latency, we have :
> > >
> > > hackbench -l 10000 -g 16 &
> > > cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
> > >               latency 0    latency -20
> > > Min Latencies:    60           61
> > > Avg Latencies:  1077           86
> > > Max Latencies: 87311          444
> > > 50% latencies:    92           85
> > > 75% latencies:   554           90
> > > 85% latencies:  1019           93
> > > 90% latencies:  1346           96
> > > 95% latencies:  5400          100
> > > 99% latencies: 19044          110
> > >
> > > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > > ---
> >
> > The ability to boost the latency sensitivity of a task seems very
> > interesting. I have been playing around with these changes and have
> > some observations.
> >
> > I tried 2 bursty tasks affinitized to the same CPU. The tasks sleep
> > for 1ms and run for 10ms in a loop. I first tried it without adjusting
> > the latency_nice value and took perf sched traces:
>
> The CPU is overloaded almost all the time as it can't run the 2 tasks
> (2*10ms every 11ms)
>
> >
> > latency_test:7040     |   2447.137 ms |        8 | avg:   6.546 ms |
> > max:  10.674 ms | max start:   353.809487 s | max end:   353.820161 s
> > latency_test:7028     |   2454.777 ms |        7 | avg:   4.494 ms |
> > max:  10.609 ms | max start:   354.804386 s | max end:   354.814995 s
> >
> > Everything looked as expected, for a 5s run they had similar runtime
> > and latency.
> >
> > I then adjusted one task to have a latency_nice of -20 (pid 8614
> > below) and took another set of traces:
> >
> > latency_test:8618     |   1845.534 ms |      131 | avg:   9.764 ms |
> > max:  10.686 ms | max start:  1405.737905 s | max end:  1405.748592 s
> > latency_test:8614     |   3033.635 ms |       16 | avg:   3.559 ms |
> > max:  10.467 ms | max start:  1407.594751 s | max end:  1407.605218 s
> >
> > The task with -20 latency_nice had significantly more runtime. The
> > average latency was improved but the max roughly stayed the same. As
> > expected the one with latency_nice value of 0 experienced more
> > switches, but so did the one with latency_nice of -20.
>
> Your results look unexpected because the vruntime of the tasks is not
> modified. So I can imagine that the thread with the low latency runs
> first up to the offset at the beg of the test but then they should
> switch regularly. I have tried a similar test with a modified rt-app
> and the result seems ok. I have a small difference but not the
> difference that you see.
>
> Could you share more details about your setup ? I'm going to try to
> reproduce your sequence

I was using an intel core i7 with this frequency details:
CPU MHz:                         4200.000
CPU max MHz:                  4800.0000
CPU min MHz:                   400.0000

This is a snippet of the test I was using:

struct sched_attr attr;
memset(&attr, 0, sizeof(struct sched_attr));
attr.size = sizeof(struct sched_attr);
attr.sched_latency_nice = nice_latency;
attr.sched_flags = SCHED_FLAG_LATENCY_NICE;

// set nice latency value
int res = syscall(__NR_sched_setattr, 0, &attr, 0);

while(1){
  // wake up every ms
  usleep(1000);
  for(int i = 0; i < 40000000; i++){}
}

>
> >
> > Also tried running the same test but instead of using latency nice I
> > adjusted the nice value as a comparison. In that case one task had a
> > nice of -5 and the other was 0.
> >
> > nice_test:25219       |   1216.839 ms |      242 | avg:  10.295 ms |
> > max:  11.927 ms | max start:  5877.881279 s | max end:  5877.893206 s
> > nice_test:25235       |   3711.788 ms |        6 | avg:   1.026 ms |
> > max:   6.143 ms | max start:  5875.603741 s | max end:  5875.609883 s
> >
> > As expected the one with a nice value of -5 had more runtime but also
> > had better latency numbers than in the previous case of using
> > latency_nice.
> >
> > I also tried a similar test with 3 bursty tasks instead of two. In
> > this case all tasks had a latency_nice of 0:
> >
> > latency_test:11467    |   1641.131 ms |      161 | avg:  17.489 ms |
> > max:  21.011 ms | max start:  1542.656275 s | max end:  1542.677286 s
> > latency_test:11463    |   1644.809 ms |      161 | avg:  11.994 ms |
> > max:  25.012 ms | max start:  1545.657776 s | max end:  1545.682788 s
> > latency_test:11478    |   1643.211 ms |      160 | avg:  11.465 ms |
> > max:  21.012 ms | max start:  1546.159026 s | max end:  1546.180038 s
> >
> > Next I tried two tasks with a latency_nice of 0 and a third one had a
> > latency_nice of -20 (pid 11763 below):
> >
> > latency_test:11763    |   1645.482 ms |      159 | avg:  19.634 ms |
> > max:  31.016 ms | max start:  1623.834862 s | max end:  1623.865877 s
> > latency_test:11750    |   1644.276 ms |      259 | avg:   9.985 ms |
> > max:  21.012 ms | max start:  1623.953921 s | max end:  1623.974933 s
> > latency_test:11747    |   1642.745 ms |      262 | avg:   9.079 ms |
> > max:  25.013 ms | max start:  1620.980435 s | max end:  1621.005447 s
> >
> > In this case it seemed like the runtime was not affected by the
> > latency_nice value, but strangely the task with the latency nice of
> > -20 had a worse average and max latency than the other two. The
> > context switch times are also increased from the previous case.
> >
> > Have we considered an approach where the task that is marked as
> > latency sensitive gets a boosted nice value when it sleeps and is
> > either scaled down exponentially as it runs, or immediately reset to
> > its default when it runs for one tick?
> >
> > Thanks,
> > Youssef
> >
> >
> > >  include/linux/sched.h |  2 +
> > >  kernel/sched/fair.c   | 96 +++++++++++++++++++++++++++++++++++++++++--
> > >  kernel/sched/sched.h  |  1 +
> > >  3 files changed, 96 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > > index a74cad08e91e..0b92674e3664 100644
> > > --- a/include/linux/sched.h
> > > +++ b/include/linux/sched.h
> > > @@ -547,6 +547,8 @@ struct sched_entity {
> > >         /* For load-balancing: */
> > >         struct load_weight              load;
> > >         struct rb_node                  run_node;
> > > +       struct rb_node                  latency_node;
> > > +       unsigned int                    on_latency;
> > >         struct list_head                group_node;
> > >         unsigned int                    on_rq;
> > >
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index e524e892d118..1a72f34136d8 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> > >
> > >         return __node_2_se(last);
> > >  }
> > > +#endif
> > >
> > > +/**************************************************************
> > > + * Scheduling class tree data structure manipulation methods:
> > > + * for latency
> > > + */
> > > +
> > > +static inline bool latency_before(struct sched_entity *a,
> > > +                               struct sched_entity *b)
> > > +{
> > > +       return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> > > +}
> > > +
> > > +#define __latency_node_2_se(node) \
> > > +       rb_entry((node), struct sched_entity, latency_node)
> > > +
> > > +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> > > +{
> > > +       return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> > > +}
> > > +
> > > +/*
> > > + * Enqueue an entity into the latency rb-tree:
> > > + */
> > > +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > +{
> > > +
> > > +       /* Only latency sensitive entity can be added to the list */
> > > +       if (se->latency_offset >= 0)
> > > +               return;
> > > +
> > > +       if (se->on_latency)
> > > +               return;
> > > +
> > > +       /*
> > > +        * An execution time less than sysctl_sched_min_granularity means that
> > > +        * the entity has been preempted by a higher sched class or an entity
> > > +        * with higher latency constraint.
> > > +        * Put it back in the list so it gets a chance to run 1st during the
> > > +        * next slice.
> > > +        */
> > > +       if (!(flags & ENQUEUE_WAKEUP)) {
> > > +               u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> > > +
> > > +               if (delta_exec >= sysctl_sched_min_granularity)
> > > +                       return;
> > > +       }
> > > +
> > > +       rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> > > +       se->on_latency = 1;
> > > +}
> > > +
> > > +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > +{
> > > +       if (se->on_latency) {
> > > +               rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> > > +               se->on_latency = 0;
> > > +       }
> > > +}
> > > +
> > > +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> > > +{
> > > +       struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> > > +
> > > +       if (!left)
> > > +               return NULL;
> > > +
> > > +       return __latency_node_2_se(left);
> > > +}
> > > +
> > > +#ifdef CONFIG_SCHED_DEBUG
> > >  /**************************************************************
> > >   * Scheduling class statistics methods:
> > >   */
> > > @@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > >         check_schedstat_required();
> > >         update_stats_enqueue_fair(cfs_rq, se, flags);
> > >         check_spread(cfs_rq, se);
> > > -       if (!curr)
> > > +       if (!curr) {
> > >                 __enqueue_entity(cfs_rq, se);
> > > +               __enqueue_latency(cfs_rq, se, flags);
> > > +       }
> > >         se->on_rq = 1;
> > >
> > >         if (cfs_rq->nr_running == 1) {
> > > @@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > >
> > >         clear_buddies(cfs_rq, se);
> > >
> > > -       if (se != cfs_rq->curr)
> > > +       if (se != cfs_rq->curr) {
> > >                 __dequeue_entity(cfs_rq, se);
> > > +               __dequeue_latency(cfs_rq, se);
> > > +       }
> > >         se->on_rq = 0;
> > >         account_entity_dequeue(cfs_rq, se);
> > >
> > > @@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > >                  */
> > >                 update_stats_wait_end_fair(cfs_rq, se);
> > >                 __dequeue_entity(cfs_rq, se);
> > > +               __dequeue_latency(cfs_rq, se);
> > >                 update_load_avg(cfs_rq, se, UPDATE_TG);
> > >         }
> > >
> > > @@ -4669,7 +4744,7 @@ static struct sched_entity *
> > >  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > >  {
> > >         struct sched_entity *left = __pick_first_entity(cfs_rq);
> > > -       struct sched_entity *se;
> > > +       struct sched_entity *latency, *se;
> > >
> > >         /*
> > >          * If curr is set we have to see if its left of the leftmost entity
> > > @@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > >                 se = cfs_rq->last;
> > >         }
> > >
> > > +       /* Check for latency sensitive entity waiting for running */
> > > +       latency = __pick_first_latency(cfs_rq);
> > > +       if (latency && (latency != se) &&
> > > +           wakeup_preempt_entity(latency, se) < 1)
> > > +               se = latency;
> > > +
> > >         return se;
> > >  }
> > >
> > > @@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> > >                 update_stats_wait_start_fair(cfs_rq, prev);
> > >                 /* Put 'current' back into the tree. */
> > >                 __enqueue_entity(cfs_rq, prev);
> > > +               __enqueue_latency(cfs_rq, prev, 0);
> > >                 /* in !on_rq case, update occurred at dequeue */
> > >                 update_load_avg(cfs_rq, prev, 0);
> > >         }
> > > @@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> > >  void init_cfs_rq(struct cfs_rq *cfs_rq)
> > >  {
> > >         cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> > > +       cfs_rq->latency_timeline = RB_ROOT_CACHED;
> > >         u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> > >  #ifdef CONFIG_SMP
> > >         raw_spin_lock_init(&cfs_rq->removed.lock);
> > > @@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency)
> > >
> > >         for_each_possible_cpu(i) {
> > >                 struct sched_entity *se = tg->se[i];
> > > +               struct rq *rq = cpu_rq(i);
> > > +               struct rq_flags rf;
> > > +
> > > +               rq_lock_irqsave(rq, &rf);
> > >
> > > +               __dequeue_latency(se->cfs_rq, se);
> > >                 WRITE_ONCE(se->latency_offset, latency);
> > > +
> > > +               rq_unlock_irqrestore(rq, &rf);
> > >         }
> > >
> > >         mutex_unlock(&shares_mutex);
> > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > > index a15fb955092c..76bca172585c 100644
> > > --- a/kernel/sched/sched.h
> > > +++ b/kernel/sched/sched.h
> > > @@ -599,6 +599,7 @@ struct cfs_rq {
> > >  #endif
> > >
> > >         struct rb_root_cached   tasks_timeline;
> > > +       struct rb_root_cached   latency_timeline;
> > >
> > >         /*
> > >          * 'curr' points to currently running entity on this cfs_rq.
> > > --
> > > 2.17.1
> > >
> > >
> > >

Re: [PATCH v5 5/7] sched/fair: Add sched group latency support

[Qais Yousef]

On 09/25/22 16:39, Vincent Guittot wrote:
> Task can set its latency priority with sched_setattr(), which is then used
> to set the latency offset of its sched_entity, but sched group entities
> still have the default latency offset value.
> 
> Add a latency.nice field in cpu cgroup controller to set the latency
> priority of the group similarly to sched_setattr(). The latency priority
> is then used to set the offset of the sched_entities of the group.
> 
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> ---
>  Documentation/admin-guide/cgroup-v2.rst |  8 ++++
>  kernel/sched/core.c                     | 53 +++++++++++++++++++++++++
>  kernel/sched/fair.c                     | 33 +++++++++++++++
>  kernel/sched/sched.h                    |  4 ++
>  4 files changed, 98 insertions(+)
> 
> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> index be4a77baf784..d8ae7e411f9c 100644
> --- a/Documentation/admin-guide/cgroup-v2.rst
> +++ b/Documentation/admin-guide/cgroup-v2.rst
> @@ -1095,6 +1095,14 @@ All time durations are in microseconds.
>          values similar to the sched_setattr(2). This maximum utilization
>          value is used to clamp the task specific maximum utilization clamp.
>  
> +  cpu.latency.nice
> +	A read-write single value file which exists on non-root
> +	cgroups.  The default is "0".
> +
> +	The nice value is in the range [-20, 19].
> +
> +	This interface file allows reading and setting latency using the
> +	same values used by sched_setattr(2).

I still don't understand how tasks will inherit the latency_nice value from
cgroups they're attached to.

For example, in EAS path we operate at task level only. If the task's
p->latency_nice = 0, but it belongs to a task group tg->latency_nice = -19;
what should the task's latency_nice be in this case? If it's in a hierarchy,
how would the effective value be calculated?


Thanks

--
Qais Yousef

Re: [PATCH v5 0/7] Add latency priority for CFS class

[K Prateek Nayak]

Hello Vincent,

Sharing results from testing on dual socket Zen3 system (2 x 64C/128T)

tl;dr

o I don't see any regression when workloads are running with
  DEFAULT_LATENCY_NICE
o I can reproduce similar results as one reported in Patch 4 for
  hackbench with latency nice 19 and hackbench and cyclictest
  with various combination of latency nice values.
o I can see improvements to tail latency for schbench with hackbench
  running in the background.
o There is an unexpected non-linear behavior observed for couple of
  cases that I cannot explain yet. (Marked with "^" in detailed results)
  I have not yet gotten to the bottom of it but if I've missed
  something, please do let me know.

Detailed results are shared below:

On 9/25/2022 8:09 PM, Vincent Guittot wrote:
> This patchset restarts the work about adding a latency priority to describe
> the latency tolerance of cfs tasks.
> 
> The patches [1-3] have been done by Parth:
> https://lore.kernel.org/lkml/20200228090755.22829-1-parth@linux.ibm.com/
> 
> I have just rebased and moved the set of latency priority outside the
> priority update. I have removed the reviewed tag because the patches
> are 2 years old.
> 
> This aims to be a generic interface and the following patches is one use
> of it to improve the scheduling latency of cfs tasks.
> 
> The patch [4] uses latency nice priority to define a latency offset
> and then decide if a cfs task can or should preempt the current
> running task. The patch gives some tests results with cyclictests and
> hackbench to highlight the benefit of latency priority for short
> interactive task or long intensive tasks.
> 
> Patch [5] adds the support of latency nice priority to task group by
> adding a cpu.latency.nice field. The range is [-20:19] as for setting task
> latency priority.
> 
> Patch [6] makes sched_core taking into account the latency offset.
> 
> Patch [7] adds a rb tree to cover some corner cases where the latency
> sensitive task (priority < 0) is preempted by high priority task (RT/DL)
> or fails to preempt them. This patch ensures that tasks will have at least
> a slice of sched_min_granularity in priority at wakeup. The patch gives
> results to show the benefit in addition to patch 4.
> 
> I have also backported the patchset on a dragonboard RB3 with an android
> mainline kernel based on v5.18 for a quick test. I have used the
> TouchLatency app which is part of AOSP and described to be a very good
> test to highlight jitter and jank frame sources of a system [1].
> In addition to the app, I have added some short running tasks waking-up
> regularly (to use the 8 cpus for 4 ms every 37777us) to stress the system
> without overloading it (and disabling EAS). The 1st results shows that the
> patchset helps to reduce the missed deadline frames from 5% to less than
> 0.1% when the cpu.latency.nice of task group are set.
> 
> I have also tested the patchset with the modified version of the alsa
> latency test that has been shared by Tim. The test quickly xruns with
> default latency nice priority 0 but is able to run without underuns with
> a latency -20 and hackbench running simultaneously.
> 
> 
> [1] https://source.android.com/docs/core/debug/eval_perf#touchlatency

Following are the results from running standard benchmarks on a
dual socket Zen3 (2 x 64C/128T) machine configured in different
NPS modes.

NPS Modes are used to logically divide single socket into
multiple NUMA region.
Following is the NUMA configuration for each NPS mode on the system:

NPS1: Each socket is a NUMA node.
    Total 2 NUMA nodes in the dual socket machine.

    Node 0: 0-63,   128-191
    Node 1: 64-127, 192-255

NPS2: Each socket is further logically divided into 2 NUMA regions.
    Total 4 NUMA nodes exist over 2 socket.
   
    Node 0: 0-31,   128-159
    Node 1: 32-63,  160-191
    Node 2: 64-95,  192-223
    Node 3: 96-127, 223-255

NPS4: Each socket is logically divided into 4 NUMA regions.
    Total 8 NUMA nodes exist over 2 socket.
   
    Node 0: 0-15,    128-143
    Node 1: 16-31,   144-159
    Node 2: 32-47,   160-175
    Node 3: 48-63,   176-191
    Node 4: 64-79,   192-207
    Node 5: 80-95,   208-223
    Node 6: 96-111,  223-231
    Node 7: 112-127, 232-255

Benchmark Results:

Kernel versions:
- tip:          5.19.0 tip sched/core
- latency_nice: 5.19.0 tip sched/core + this series

When we started testing, the tip was at:
commit 7e9518baed4c ("sched/fair: Move call to list_last_entry() in detach_tasks")

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ hackbench - DEFAULT_LATENCY_NICE ~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

NPS1

Test:			tip		    latency_nice
 1-groups:	   4.23 (0.00 pct)	   4.06 (4.01 pct)
 2-groups:	   4.93 (0.00 pct)	   4.89 (0.81 pct)
 4-groups:	   5.32 (0.00 pct)	   5.31 (0.18 pct)
 8-groups:	   5.46 (0.00 pct)	   5.54 (-1.46 pct)
16-groups:	   7.31 (0.00 pct)	   7.33 (-0.27 pct)

NPS2

Test:			tip		    latency_nice
 1-groups:	   4.19 (0.00 pct)	   4.12 (1.67 pct)
 2-groups:	   4.77 (0.00 pct)	   4.82 (-1.04 pct)
 4-groups:	   5.15 (0.00 pct)	   5.17 (-0.38 pct)
 8-groups:	   5.47 (0.00 pct)	   5.48 (-0.18 pct)
16-groups:	   6.63 (0.00 pct)	   6.65 (-0.30 pct)

NPS4

Test:			tip		     latency_nice
 1-groups:	   4.23 (0.00 pct)	   4.31 (-1.89 pct)
 2-groups:	   4.78 (0.00 pct)	   4.75 (0.62 pct)
 4-groups:	   5.17 (0.00 pct)	   5.24 (-1.35 pct)
 8-groups:	   5.63 (0.00 pct)	   5.59 (0.71 pct)
16-groups:	   7.88 (0.00 pct)	   7.09 (10.02 pct)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ schbench - DEFAULT_LATENCY_NICE ~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

NPS1

#workers:     tip		   latency_nice
  1:	  22.00 (0.00 pct)	  21.00 (4.54 pct)
  2:	  34.00 (0.00 pct)	  34.00 (0.00 pct)
  4:	  37.00 (0.00 pct)	  40.00 (-8.10 pct)
  8:	  55.00 (0.00 pct)	  49.00 (10.90 pct)
 16:	  69.00 (0.00 pct)	  66.00 (4.34 pct)
 32:	 113.00 (0.00 pct)	 117.00 (-3.53 pct)
 64:	 219.00 (0.00 pct)	 242.00 (-10.50 pct) *
 64:	 219.00 (0.00 pct)	 194.00 (11.41 pct)  [Verification Run]
128:	 506.00 (0.00 pct)	 513.00 (-1.38 pct)
256:	 45440.00 (0.00 pct)	 44992.00 (0.98 pct)
512:	 76672.00 (0.00 pct)	 83328.00 (-8.68 pct)

NPS2

#workers:     tip		    latency_nice
  1:	  31.00 (0.00 pct)	  20.00 (35.48 pct)
  2:	  36.00 (0.00 pct)	  28.00 (22.22 pct)
  4:	  45.00 (0.00 pct)	  37.00 (17.77 pct)
  8:	  47.00 (0.00 pct)	  51.00 (-8.51 pct)
 16:	  66.00 (0.00 pct)	  69.00 (-4.54 pct)
 32:	 114.00 (0.00 pct)	 113.00 (0.87 pct)
 64:	 215.00 (0.00 pct)	 215.00 (0.00 pct)
128:	 495.00 (0.00 pct)	 529.00 (-6.86 pct)  *
128:	 495.00 (0.00 pct)	 416.00 (15.95  pct) [Verification Run]
256:	 48576.00 (0.00 pct)	 46912.00 (3.42 pct)
512:	 79232.00 (0.00 pct)	 82560.00 (-4.20 pct)

NPS4

#workers:     tip		    latency_nice
  1:	  30.00 (0.00 pct)	  34.00 (-13.33 pct)
  2:	  34.00 (0.00 pct)	  42.00 (-23.52 pct)
  4:	  41.00 (0.00 pct)	  42.00 (-2.43 pct)
  8:	  60.00 (0.00 pct)	  55.00 (8.33 pct)
 16:	  68.00 (0.00 pct)	  69.00 (-1.47 pct)
 32:	 116.00 (0.00 pct)	 115.00 (0.86 pct)
 64:	 224.00 (0.00 pct)	 223.00 (0.44 pct)
128:	 495.00 (0.00 pct)	 677.00 (-36.76 pct) *
128:	 495.00 (0.00 pct)       388.00 (21.61 pct)  [Verification Run]
256:	 45888.00 (0.00 pct)	 44608.00 (2.78 pct)
512:	 78464.00 (0.00 pct)	 81536.00 (-3.91 pct)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ tbench - DEFAULT_LATENCY_NICE ~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

NPS1

Clients:      tip		   latency_nice
    1	 550.66 (0.00 pct)	 546.63 (-0.73 pct)
    2	 1009.69 (0.00 pct)	 1016.40 (0.66 pct)
    4	 1795.32 (0.00 pct)	 1773.95 (-1.19 pct)
    8	 2971.16 (0.00 pct)	 2930.26 (-1.37 pct)
   16	 4627.98 (0.00 pct)	 4727.82 (2.15 pct)
   32	 8065.15 (0.00 pct)	 9019.11 (11.82 pct)
   64	 14994.32 (0.00 pct)	 15100.22 (0.70 pct)
  128	 5175.73 (0.00 pct)	 18223.69 (252.09 pct) *
  128    20029.53 (0.00 pct)     20517.17 (2.43 pct)   [Verification Run]
  256	 48763.57 (0.00 pct)	 44463.63 (-8.81 pct)
  512	 43780.78 (0.00 pct)	 44170.21 (0.88 pct)
 1024	 40341.84 (0.00 pct)	 40883.10 (1.34 pct)

NPS2

Clients:      tip		    latency_nice
    1	 551.06 (0.00 pct)	 547.43 (-0.65 pct)
    2	 1000.76 (0.00 pct)	 1014.83 (1.40 pct)
    4	 1737.02 (0.00 pct)	 1742.30 (0.30 pct)
    8	 2992.31 (0.00 pct)	 2951.59 (-1.36 pct)
   16	 4579.29 (0.00 pct)	 4558.05 (-0.46 pct)
   32	 9120.73 (0.00 pct)	 8122.06 (-10.94 pct) *
   32    8814.62 (0.00 pct)      8965.54 (1.71 pct)   [Verification Run]
   64	 14918.58 (0.00 pct)	 14890.93 (-0.18 pct)
  128	 20830.61 (0.00 pct)	 20410.48 (-2.01 pct)
  256	 47708.18 (0.00 pct)	 45312.84 (-5.02 pct) *
  256    44941.88 (0.00 pct)     44555.92 (-0.85 pct) [Verification Run]
  512	 43721.79 (0.00 pct)	 43653.43 (-0.15 pct)
 1024	 40920.49 (0.00 pct)	 41162.17 (0.59 pct)

NPS4

Clients:      tip		    latency_nice
    1	 549.22 (0.00 pct)	 539.81 (-1.71 pct)
    2	 1000.08 (0.00 pct)	 1010.12 (1.00 pct)
    4	 1794.78 (0.00 pct)	 1736.06 (-3.27 pct)
    8	 3008.50 (0.00 pct)	 2952.68 (-1.85 pct)
   16	 4804.71 (0.00 pct)	 4454.17 (-7.29 pct)  *
   16    4391.10 (0.00 pct)      4497.43 (2.42 pct)   [Verification Run]
   32	 9156.57 (0.00 pct)	 8820.05 (-3.67 pct)
   64	 14901.45 (0.00 pct)	 14786.25 (-0.77 pct)
  128	 20771.20 (0.00 pct)	 19955.11 (-3.92 pct)
  256	 47033.88 (0.00 pct)	 44937.51 (-4.45 pct)
  512	 43429.01 (0.00 pct)	 42638.81 (-1.81 pct)
 1024	 39271.27 (0.00 pct)	 40044.17 (1.96 pct)


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ stream - DEFAULT_LATENCY_NICE ~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

NPS1

10 Runs:

Test:		tip	                latency_nice
 Copy:	 336311.52 (0.00 pct)	 326015.98 (-3.06 pct)
Scale:	 212955.82 (0.00 pct)	 208667.27 (-2.01 pct)
  Add:	 251518.23 (0.00 pct)	 237286.20 (-5.65 pct)
Triad:	 262077.88 (0.00 pct)	 258949.80 (-1.19 pct)

100 Runs:

Test:		tip			latency_nice
 Copy:	 339533.83 (0.00 pct)	 335126.73 (-1.29 pct)
Scale:	 194736.72 (0.00 pct)	 221151.24 (13.56 pct)
  Add:	 218294.54 (0.00 pct)	 251427.43 (15.17 pct)
Triad:	 262371.40 (0.00 pct)	 260100.85 (-0.86 pct)

NPS2

10 Runs:

Test:		tip			latency_nice
 Copy:   335277.15 (0.00 pct)    339614.38 (1.29 pct)
Scale:   220990.24 (0.00 pct)    221052.78 (0.02 pct)
  Add:   264156.13 (0.00 pct)    263684.19 (-0.17 pct)
Triad:   268707.53 (0.00 pct)    272610.96 (1.45 pct)

100 Runs:

Test:           tip                     latency_nice
 Copy:   334913.73 (0.00 pct)    339001.88 (1.22 pct)
Scale:   230522.47 (0.00 pct)    229848.86 (-0.29 pct)
  Add:   264567.28 (0.00 pct)    264288.34 (-0.10 pct)
Triad:   272974.23 (0.00 pct)    272045.17 (-0.34 pct)

NPS4

10 Runs:

Test:	        tip			latency_nice
 Copy:   299432.31 (0.00 pct)    307649.18 (2.74 pct)
Scale:   217998.17 (0.00 pct)    205763.70 (-5.61 pct)
  Add:   234305.46 (0.00 pct)    226381.75 (-3.38 pct)
Triad:   244369.15 (0.00 pct)    254225.30 (4.03 pct)

100 Runs:

Test:		tip			latency_nice
 Copy:   344421.25 (0.00 pct)    322189.81 (-6.45 pct)
Scale:   237998.44 (0.00 pct)    227709.58 (-4.32 pct)
  Add:   257501.82 (0.00 pct)    244009.58 (-5.23 pct)
Triad:   267686.50 (0.00 pct)    251840.25 (-5.91 pct)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ Test cases for Latency Nice ~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Note: Latency Nice might be referred to as LN in the data below. Latency Nice
value was set using a wrapper script for all the workload threads during the
testing.
All the test results reported below are for NPS1 configuration.

o Hackbench Pipes (100000 loops, threads)

Test:                  tip                 Latency Nice: -20        Latency Nice: 0         Latency Nice: 19
 1-groups:         4.23 (0.00 pct)          4.39 (-3.78 pct)        3.99 (5.67 pct)         3.88 (8.27 pct)
 2-groups:         4.93 (0.00 pct)          4.91 (0.40 pct)         4.69 (4.86 pct)         4.59 (6.89 pct)
 4-groups:         5.32 (0.00 pct)          5.37 (-0.93 pct)        5.19 (2.44 pct)         5.05 (5.07 pct)
 8-groups:         5.46 (0.00 pct)          5.90 (-8.05 pct)        5.34 (2.19 pct)         5.17 (5.31 pct)
16-groups:         7.31 (0.00 pct)          7.99 (-9.30 pct)        6.96 (4.78 pct)         6.51 (10.94 pct)

o Only Hackbench with different Latency Nice Values

> Loops: 100000

- Pipe (Process)

  Test:            Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
 1-groups:         3.77 (0.00 pct)         4.23 (-12.20 pct)       3.83 (-1.59 pct)
 2-groups:         4.39 (0.00 pct)         4.73 (-7.74 pct)        4.31 (1.82 pct)
 4-groups:         4.80 (0.00 pct)         5.07 (-5.62 pct)        4.68 (2.50 pct)
 8-groups:         4.95 (0.00 pct)         5.68 (-14.74 pct)       4.76 (3.83 pct)
16-groups:         6.47 (0.00 pct)         7.87 (-21.63 pct)       6.08 (6.02 pct)

- Socket (Thread)

Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
 1-groups:         6.08 (0.00 pct)         5.99 (1.48 pct)         6.08 (0.00 pct)
 2-groups:         6.15 (0.00 pct)         6.25 (-1.62 pct)        6.14 (0.16 pct)
 4-groups:         6.39 (0.00 pct)         6.42 (-0.46 pct)        6.44 (-0.78 pct)
 8-groups:         8.51 (0.00 pct)         9.01 (-5.87 pct)        8.36 (1.76 pct)
16-groups:        12.48 (0.00 pct)        15.32 (-22.75 pct)      12.72 (-1.92 pct)

- Socket (Process)

Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
 1-groups:         6.44 (0.00 pct)         5.50 (14.59 pct)  ^     6.43 (0.15 pct)
 2-groups:         6.55 (0.00 pct)         5.56 (15.11 pct)  ^     6.36 (2.90 pct)
 4-groups:         6.74 (0.00 pct)         6.19 (8.16 pct)   ^     6.69 (0.74 pct)
 8-groups:         8.03 (0.00 pct)         8.29 (-3.23 pct)        8.02 (0.12 pct)
16-groups:        12.25 (0.00 pct)        14.11 (-15.18 pct)      12.41 (-1.30 pct)

> Loops: 2160 (Same as in testing)

- Pipe (Thread)

Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
 1-groups:         0.10 (0.00 pct)         0.12 (-20.00 pct)       0.10 (0.00 pct)
 2-groups:         0.12 (0.00 pct)         0.15 (-25.00 pct)       0.11 (8.33 pct)
 4-groups:         0.14 (0.00 pct)         0.18 (-28.57 pct)       0.15 (-7.14 pct)
 8-groups:         0.17 (0.00 pct)         0.24 (-41.17 pct)       0.17 (0.00 pct)
16-groups:         0.26 (0.00 pct)         0.33 (-26.92 pct)       0.21 (19.23 pct)

- Pipe (Process)

Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
 1-groups:         0.10 (0.00 pct)         0.12 (-20.00 pct)       0.10 (0.00 pct)
 2-groups:         0.12 (0.00 pct)         0.16 (-33.33 pct)       0.12 (0.00 pct)
 4-groups:         0.14 (0.00 pct)         0.17 (-21.42 pct)       0.13 (7.14 pct)
 8-groups:         0.16 (0.00 pct)         0.24 (-50.00 pct)       0.16 (0.00 pct)
16-groups:         0.23 (0.00 pct)         0.33 (-43.47 pct)       0.19 (17.39 pct)

- Socket (Thread)

Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
 1-groups:         0.19 (0.00 pct)         0.18 (5.26 pct)         0.18 (5.26 pct)
 2-groups:         0.21 (0.00 pct)         0.21 (0.00 pct)         0.20 (4.76 pct)
 4-groups:         0.22 (0.00 pct)         0.25 (-13.63 pct)       0.22 (0.00 pct)
 8-groups:         0.27 (0.00 pct)         0.36 (-33.33 pct)       0.27 (0.00 pct)
16-groups:         0.42 (0.00 pct)         0.55 (-30.95 pct)       0.40 (4.76 pct)

- Socket (Process)

Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
 1-groups:         0.17 (0.00 pct)         0.17 (0.00 pct)         0.17 (0.00 pct)
 2-groups:         0.19 (0.00 pct)         0.20 (-5.26 pct)        0.19 (0.00 pct)
 4-groups:         0.20 (0.00 pct)         0.22 (-10.00 pct)       0.20 (0.00 pct)
 8-groups:         0.25 (0.00 pct)         0.32 (-28.00 pct)       0.25 (0.00 pct)
16-groups:         0.40 (0.00 pct)         0.51 (-27.50 pct)       0.39 (2.50 pct)

o Hackbench and Cyclictest in NPS1 configuration

perf bench sched messaging -p -t -l 100000 -g 16&
cyclictest --policy other -D 5 -q -n -H 20000 

-----------------------------------------------------------------------------------------------------------------
|Hackbench     |      Cyclictest LN = 19        |         Cyclictest LN = 0       |      Cyclictest LN = -20    |
|LN            |--------------------------------|---------------------------------|-----------------------------|
|v             |   Min  |   Avg   |  Max        |     Min  |   Avg   |  Max       |     Min  |   Avg   |  Max   |
|--------------|--------|---------|-------------|----------|---------|------------|----------|---------|--------|
|0             |  54.00 |  117.00 | 3021.67     |    53.67 |  65.33  | 133.00     |    53.67 |  65.00  | 201.33 |  ^
|19            |  50.00 |  100.67 | 3099.33     |    41.00 |  64.33  | 1014.33    |    54.00 |  63.67  | 213.33 |
|-20           |  53.00 |  169.00 | 11661.67    |    53.67 |  217.33 | 14313.67   |    46.00 |  61.33  | 236.00 |  ^
-----------------------------------------------------------------------------------------------------------------

o Hackbench and schbench in NPS1 configuration

perf bench sched messaging -p -t -l 1000000 -g 16&
schebcnh -m 1 -t 64 -s 30s

------------------------------------------------------------------------------------------------------------
|Hackbench     |   schbench LN = 19         |        schbench LN = 0         |       schbench LN = -20     |
|LN            |----------------------------|--------------------------------|-----------------------------|
|v             |  90th  |  95th  |  99th    |   90th  |  95th   |  99th      |   90th  |   95th   | 99th   |
|--------------|--------|--------|----------|---------|---------|------------|---------|----------|--------|
|0             |  4264  |  6744  |  15664   |   17952 |  32672  |  55488     |   15088 |   25312  | 50112  |
|19            |  288   |  613   |  2332    |   274   |  1015   |  3628      |   374   |   1394   | 4424   |
|-20           |  35904 |  47680 |  79744   |   87168 |  113536 |  176896    |   13008 |   21216  | 42560  |   ^
------------------------------------------------------------------------------------------------------------

o SpecJBB Multi-JVM

---------------------------------------------
| Latency Nice  |         0  |           19 |
---------------------------------------------
| max-jOPS      |      100%  |      109.92% |
| critical-jOPS |      100%  |      153.70% |
---------------------------------------------

In most cases, latency nice delivers what it promises.
Some cases marked with "^" have shown anomalies or non-linear behavior
that is yet to be root caused. If you've seen something similar during
your testing, I would love to know what could lead to such a behavior.

If you would like more details on the benchmarks results reported above
or if there is any specific workload you would like me to test on the
Zen3 machine, please do let me know.

> 
> [..snip..]
> 

--
Thanks and Regards,
Prateek
-- 
--
Thanks and Regards,
Prateek

Re: [PATCH v5 3/7] sched: Allow sched_{get,set}attr to change latency_nice of the task

[K Prateek Nayak]

Hello Vincent,

On 9/25/2022 8:09 PM, Vincent Guittot wrote:
> From: Parth Shah <parth@linux.ibm.com>
> 
> Introduce the latency_nice attribute to sched_attr and provide a
> mechanism to change the value with the use of sched_setattr/sched_getattr
> syscall.
> 
> Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in
> latency_nice of the task on every sched_setattr syscall.
> 
> Signed-off-by: Parth Shah <parth@linux.ibm.com>
> [rebase and add a dedicated __setscheduler_latency ]
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> ---
>  include/uapi/linux/sched.h       |  4 +++-
>  include/uapi/linux/sched/types.h | 19 +++++++++++++++++++
>  kernel/sched/core.c              | 24 ++++++++++++++++++++++++
>  tools/include/uapi/linux/sched.h |  4 +++-
>  4 files changed, 49 insertions(+), 2 deletions(-)
> 
> diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h
> index 3bac0a8ceab2..b2e932c25be6 100644
> --- a/include/uapi/linux/sched.h
> +++ b/include/uapi/linux/sched.h
> @@ -132,6 +132,7 @@ struct clone_args {
>  #define SCHED_FLAG_KEEP_PARAMS		0x10
>  #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
>  #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
> +#define SCHED_FLAG_LATENCY_NICE		0x80

[1]

>  
>  #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
>  				 SCHED_FLAG_KEEP_PARAMS)
> @@ -143,6 +144,7 @@ struct clone_args {
>  			 SCHED_FLAG_RECLAIM		| \
>  			 SCHED_FLAG_DL_OVERRUN		| \
>  			 SCHED_FLAG_KEEP_ALL		| \
> -			 SCHED_FLAG_UTIL_CLAMP)
> +			 SCHED_FLAG_UTIL_CLAMP		| \
> +			 SCHED_FLAG_LATENCY_NICE)
>  
>  #endif /* _UAPI_LINUX_SCHED_H */
> diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/types.h
> index f2c4589d4dbf..db1e8199e8c8 100644
> --- a/include/uapi/linux/sched/types.h
> +++ b/include/uapi/linux/sched/types.h
> @@ -10,6 +10,7 @@ struct sched_param {
>  
>  #define SCHED_ATTR_SIZE_VER0	48	/* sizeof first published struct */
>  #define SCHED_ATTR_SIZE_VER1	56	/* add: util_{min,max} */
> +#define SCHED_ATTR_SIZE_VER2	60	/* add: latency_nice */
>  
>  /*
>   * Extended scheduling parameters data structure.
> @@ -98,6 +99,22 @@ struct sched_param {
>   * scheduled on a CPU with no more capacity than the specified value.
>   *
>   * A task utilization boundary can be reset by setting the attribute to -1.
> + *
> + * Latency Tolerance Attributes
> + * ===========================
> + *
> + * A subset of sched_attr attributes allows to specify the relative latency
> + * requirements of a task with respect to the other tasks running/queued in the
> + * system.
> + *
> + * @ sched_latency_nice	task's latency_nice value
> + *
> + * The latency_nice of a task can have any value in a range of
> + * [MIN_LATENCY_NICE..MAX_LATENCY_NICE].
> + *
> + * A task with latency_nice with the value of LATENCY_NICE_MIN can be
> + * taken for a task requiring a lower latency as opposed to the task with
> + * higher latency_nice.
>   */
>  struct sched_attr {
>  	__u32 size;
> @@ -120,6 +137,8 @@ struct sched_attr {
>  	__u32 sched_util_min;
>  	__u32 sched_util_max;
>  
> +	/* latency requirement hints */
> +	__s32 sched_latency_nice;
>  };
>  
>  #endif /* _UAPI_LINUX_SCHED_TYPES_H */
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index ada2d05bd894..6a6116ea4c2c 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7318,6 +7318,14 @@ static void __setscheduler_params(struct task_struct *p,
>  	p->rt_priority = attr->sched_priority;
>  	p->normal_prio = normal_prio(p);
>  	set_load_weight(p, true);
> +
> +}
> +
> +static void __setscheduler_latency(struct task_struct *p,
> +		const struct sched_attr *attr)
> +{
> +	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
> +		p->latency_nice = attr->sched_latency_nice;
>  }
>  
>  /*
> @@ -7460,6 +7468,13 @@ static int __sched_setscheduler(struct task_struct *p,
>  			return retval;
>  	}
>  
> +	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
> +		if (attr->sched_latency_nice > MAX_LATENCY_NICE)
> +			return -EINVAL;
> +		if (attr->sched_latency_nice < MIN_LATENCY_NICE)
> +			return -EINVAL;
> +	}
> +
>  	if (pi)
>  		cpuset_read_lock();
>  
> @@ -7494,6 +7509,9 @@ static int __sched_setscheduler(struct task_struct *p,
>  			goto change;
>  		if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
>  			goto change;
> +		if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
> +		    attr->sched_latency_nice != p->latency_nice)
> +			goto change;
>  
>  		p->sched_reset_on_fork = reset_on_fork;
>  		retval = 0;
> @@ -7582,6 +7600,7 @@ static int __sched_setscheduler(struct task_struct *p,
>  		__setscheduler_params(p, attr);
>  		__setscheduler_prio(p, newprio);
>  	}
> +	__setscheduler_latency(p, attr);
>  	__setscheduler_uclamp(p, attr);
>  
>  	if (queued) {
> @@ -7792,6 +7811,9 @@ static int sched_copy_attr(struct sched_attr __user *uattr, struct sched_attr *a
>  	    size < SCHED_ATTR_SIZE_VER1)
>  		return -EINVAL;
>  
> +	if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) &&
> +	    size < SCHED_ATTR_SIZE_VER2)
> +		return -EINVAL;
>  	/*
>  	 * XXX: Do we want to be lenient like existing syscalls; or do we want
>  	 * to be strict and return an error on out-of-bounds values?
> @@ -8029,6 +8051,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
>  	get_params(p, &kattr);
>  	kattr.sched_flags &= SCHED_FLAG_ALL;
>  
> +	kattr.sched_latency_nice = p->latency_nice;
> +
>  #ifdef CONFIG_UCLAMP_TASK
>  	/*
>  	 * This could race with another potential updater, but this is fine
> diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sched.h
> index 3bac0a8ceab2..ecc4884bfe4b 100644
> --- a/tools/include/uapi/linux/sched.h
> +++ b/tools/include/uapi/linux/sched.h
> @@ -132,6 +132,7 @@ struct clone_args {
>  #define SCHED_FLAG_KEEP_PARAMS		0x10
>  #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
>  #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
> +#define SCHED_FLAG_LATENCY_NICE		0X80

Small nit.
Can you change the "x" in "0X80" to lower case here to make it
consistent with [1] and other #define in the same file. When building
tools like perf, I see the following warning:

Warning: Kernel ABI header at 'tools/include/uapi/linux/sched.h' differs from latest version at 'include/uapi/linux/sched.h'
diff -u tools/include/uapi/linux/sched.h include/uapi/linux/sched.h

Following is the output of
diff -u tools/include/uapi/linux/sched.h include/uapi/linux/sched.h

--- tools/include/uapi/linux/sched.h    2022-10-12 14:56:00.925360275 +0000
+++ include/uapi/linux/sched.h  2022-10-12 14:56:00.917360219 +0000
@@ -132,7 +132,7 @@
 #define SCHED_FLAG_KEEP_PARAMS         0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN      0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX      0x40
-#define SCHED_FLAG_LATENCY_NICE                0X80
+#define SCHED_FLAG_LATENCY_NICE                0x80

 #define SCHED_FLAG_KEEP_ALL    (SCHED_FLAG_KEEP_POLICY | \
                                 SCHED_FLAG_KEEP_PARAMS)
--

>  
>  #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
>  				 SCHED_FLAG_KEEP_PARAMS)
> @@ -143,6 +144,7 @@ struct clone_args {
>  			 SCHED_FLAG_RECLAIM		| \
>  			 SCHED_FLAG_DL_OVERRUN		| \
>  			 SCHED_FLAG_KEEP_ALL		| \
> -			 SCHED_FLAG_UTIL_CLAMP)
> +			 SCHED_FLAG_UTIL_CLAMP		| \
> +			 SCHED_FLAG_LATENCY_NICE)
>  
>  #endif /* _UAPI_LINUX_SCHED_H */

--
Thanks and Regards,
Prateek

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Vincent Guittot]

Le mardi 11 oct. 2022 à 18:54:27 (-0500), Youssef Esmat a écrit :
> Hi Vincent,
> 
> On Tue, Oct 11, 2022 at 12:10 PM Vincent Guittot
> <vincent.guittot@linaro.org> wrote:

...

> > > >               latency 0    latency -20
> > > > Min Latencies:    60           61
> > > > Avg Latencies:  1077           86
> > > > Max Latencies: 87311          444
> > > > 50% latencies:    92           85
> > > > 75% latencies:   554           90
> > > > 85% latencies:  1019           93
> > > > 90% latencies:  1346           96
> > > > 95% latencies:  5400          100
> > > > 99% latencies: 19044          110
> > > >
> > > > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > > > ---
> > >
> > > The ability to boost the latency sensitivity of a task seems very
> > > interesting. I have been playing around with these changes and have
> > > some observations.
> > >
> > > I tried 2 bursty tasks affinitized to the same CPU. The tasks sleep
> > > for 1ms and run for 10ms in a loop. I first tried it without adjusting
> > > the latency_nice value and took perf sched traces:
> >
> > The CPU is overloaded almost all the time as it can't run the 2 tasks
> > (2*10ms every 11ms)
> >
> > >
> > > latency_test:7040     |   2447.137 ms |        8 | avg:   6.546 ms |
> > > max:  10.674 ms | max start:   353.809487 s | max end:   353.820161 s
> > > latency_test:7028     |   2454.777 ms |        7 | avg:   4.494 ms |
> > > max:  10.609 ms | max start:   354.804386 s | max end:   354.814995 s
> > >
> > > Everything looked as expected, for a 5s run they had similar runtime
> > > and latency.
> > >
> > > I then adjusted one task to have a latency_nice of -20 (pid 8614
> > > below) and took another set of traces:
> > >
> > > latency_test:8618     |   1845.534 ms |      131 | avg:   9.764 ms |
> > > max:  10.686 ms | max start:  1405.737905 s | max end:  1405.748592 s
> > > latency_test:8614     |   3033.635 ms |       16 | avg:   3.559 ms |
> > > max:  10.467 ms | max start:  1407.594751 s | max end:  1407.605218 s
> > >
> > > The task with -20 latency_nice had significantly more runtime. The
> > > average latency was improved but the max roughly stayed the same. As
> > > expected the one with latency_nice value of 0 experienced more
> > > switches, but so did the one with latency_nice of -20.
> >
> > Your results look unexpected because the vruntime of the tasks is not
> > modified. So I can imagine that the thread with the low latency runs
> > first up to the offset at the beg of the test but then they should
> > switch regularly. I have tried a similar test with a modified rt-app
> > and the result seems ok. I have a small difference but not the
> > difference that you see.
> >
> > Could you share more details about your setup ? I'm going to try to
> > reproduce your sequence
> 
> I was using an intel core i7 with this frequency details:
> CPU MHz:                         4200.000
> CPU max MHz:                  4800.0000
> CPU min MHz:                   400.0000
> 
> This is a snippet of the test I was using:
> 
> struct sched_attr attr;
> memset(&attr, 0, sizeof(struct sched_attr));
> attr.size = sizeof(struct sched_attr);
> attr.sched_latency_nice = nice_latency;
> attr.sched_flags = SCHED_FLAG_LATENCY_NICE;
> 
> // set nice latency value
> int res = syscall(__NR_sched_setattr, 0, &attr, 0);
> 
> while(1){
>   // wake up every ms
>   usleep(1000);
>   for(int i = 0; i < 40000000; i++){}
> }

Between v2 and v3, the sched_feat(GENTLE_FAIR_SLEEPERS) has diseappeared when
I moved the computation of latency_offset at the setting of the latency prio
instead of runtime. As a result, the latency nice task can preempt up to
(sysctl_sched_latency - sysctl_sched_wakeup_granularity) but other threads are
cap to half of sysctl_sched_latency at wakeup.

Could you try the patch below ?

---
 kernel/sched/core.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 8b44685ae247..68f9a83d7089 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1287,11 +1287,15 @@ static void set_load_weight(struct task_struct *p, bool update_load)
 static void set_latency_offset(struct task_struct *p)
 {
 	long weight = sched_latency_to_weight[p->latency_prio];
+	unsigned long period = sysctl_sched_latency;
 	s64 offset;
 
-	offset = sysctl_sched_latency * weight;
+	if (sched_feat(GENTLE_FAIR_SLEEPERS))
+		period >>= 1;
+	offset = period * weight;
 	offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX);
 	p->se.latency_offset = (long)offset;
+	trace_printk("set_latency_offset pid %d offset %ld", p->pid, offset);
 }
 
 #ifdef CONFIG_UCLAMP_TASK
@@ -10894,12 +10898,17 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,
 static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
 				    struct cftype *cft)
 {
+	unsigned long period = sysctl_sched_latency;
 	int last_delta = INT_MAX;
 	int prio, delta;
 	s64 weight;
 
+	if (sched_feat(GENTLE_FAIR_SLEEPERS))
+		period >>= 1;
+
 	weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX;
-	weight = div_s64(weight, sysctl_sched_latency);
+	period = sysctl_sched_latency;
+	weight = div_s64(weight, period);
 
 	/* Find the closest nice value to the current weight */
 	for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) {
@@ -10915,6 +10924,7 @@ static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
 static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
 				     struct cftype *cft, s64 nice)
 {
+	unsigned long period;
 	s64 latency_offset;
 	long weight;
 	int idx;
@@ -10926,7 +10936,10 @@ static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
 	idx = array_index_nospec(idx, LATENCY_NICE_WIDTH);
 	weight = sched_latency_to_weight[idx];
 
-	latency_offset = sysctl_sched_latency * weight;
+	period = sysctl_sched_latency;
+	if (sched_feat(GENTLE_FAIR_SLEEPERS))
+		period >>= 1;
+	latency_offset = period * weight;
 	latency_offset = div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX);
 
 	return sched_group_set_latency(css_tg(css), latency_offset);
-- 
2.17.1

> 
> >
> > >
> > > Also tried running the same test but instead of using latency nice I
> > > adjusted the nice value as a comparison. In that case one task had a
> > > nice of -5 and the other was 0.
> > >
> > > nice_test:25219       |   1216.839 ms |      242 | avg:  10.295 ms |
> > > max:  11.927 ms | max start:  5877.881279 s | max end:  5877.893206 s
> > > nice_test:25235       |   3711.788 ms |        6 | avg:   1.026 ms |
> > > max:   6.143 ms | max start:  5875.603741 s | max end:  5875.609883 s
> > >
> > > As expected the one with a nice value of -5 had more runtime but also
> > > had better latency numbers than in the previous case of using
> > > latency_nice.
> > >
> > > I also tried a similar test with 3 bursty tasks instead of two. In
> > > this case all tasks had a latency_nice of 0:
> > >
> > > latency_test:11467    |   1641.131 ms |      161 | avg:  17.489 ms |
> > > max:  21.011 ms | max start:  1542.656275 s | max end:  1542.677286 s
> > > latency_test:11463    |   1644.809 ms |      161 | avg:  11.994 ms |
> > > max:  25.012 ms | max start:  1545.657776 s | max end:  1545.682788 s
> > > latency_test:11478    |   1643.211 ms |      160 | avg:  11.465 ms |
> > > max:  21.012 ms | max start:  1546.159026 s | max end:  1546.180038 s
> > >
> > > Next I tried two tasks with a latency_nice of 0 and a third one had a
> > > latency_nice of -20 (pid 11763 below):
> > >
> > > latency_test:11763    |   1645.482 ms |      159 | avg:  19.634 ms |
> > > max:  31.016 ms | max start:  1623.834862 s | max end:  1623.865877 s
> > > latency_test:11750    |   1644.276 ms |      259 | avg:   9.985 ms |
> > > max:  21.012 ms | max start:  1623.953921 s | max end:  1623.974933 s
> > > latency_test:11747    |   1642.745 ms |      262 | avg:   9.079 ms |
> > > max:  25.013 ms | max start:  1620.980435 s | max end:  1621.005447 s
> > >
> > > In this case it seemed like the runtime was not affected by the
> > > latency_nice value, but strangely the task with the latency nice of
> > > -20 had a worse average and max latency than the other two. The
> > > context switch times are also increased from the previous case.
> > >
> > > Have we considered an approach where the task that is marked as
> > > latency sensitive gets a boosted nice value when it sleeps and is
> > > either scaled down exponentially as it runs, or immediately reset to
> > > its default when it runs for one tick?
> > >
> > > Thanks,
> > > Youssef
> > >
> > >
> > > >  include/linux/sched.h |  2 +
> > > >  kernel/sched/fair.c   | 96 +++++++++++++++++++++++++++++++++++++++++--
> > > >  kernel/sched/sched.h  |  1 +
> > > >  3 files changed, 96 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > > > index a74cad08e91e..0b92674e3664 100644
> > > > --- a/include/linux/sched.h
> > > > +++ b/include/linux/sched.h
> > > > @@ -547,6 +547,8 @@ struct sched_entity {
> > > >         /* For load-balancing: */
> > > >         struct load_weight              load;
> > > >         struct rb_node                  run_node;
> > > > +       struct rb_node                  latency_node;
> > > > +       unsigned int                    on_latency;
> > > >         struct list_head                group_node;
> > > >         unsigned int                    on_rq;
> > > >
> > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > > index e524e892d118..1a72f34136d8 100644
> > > > --- a/kernel/sched/fair.c
> > > > +++ b/kernel/sched/fair.c
> > > > @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> > > >
> > > >         return __node_2_se(last);
> > > >  }
> > > > +#endif
> > > >
> > > > +/**************************************************************
> > > > + * Scheduling class tree data structure manipulation methods:
> > > > + * for latency
> > > > + */
> > > > +
> > > > +static inline bool latency_before(struct sched_entity *a,
> > > > +                               struct sched_entity *b)
> > > > +{
> > > > +       return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> > > > +}
> > > > +
> > > > +#define __latency_node_2_se(node) \
> > > > +       rb_entry((node), struct sched_entity, latency_node)
> > > > +
> > > > +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> > > > +{
> > > > +       return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> > > > +}
> > > > +
> > > > +/*
> > > > + * Enqueue an entity into the latency rb-tree:
> > > > + */
> > > > +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > +{
> > > > +
> > > > +       /* Only latency sensitive entity can be added to the list */
> > > > +       if (se->latency_offset >= 0)
> > > > +               return;
> > > > +
> > > > +       if (se->on_latency)
> > > > +               return;
> > > > +
> > > > +       /*
> > > > +        * An execution time less than sysctl_sched_min_granularity means that
> > > > +        * the entity has been preempted by a higher sched class or an entity
> > > > +        * with higher latency constraint.
> > > > +        * Put it back in the list so it gets a chance to run 1st during the
> > > > +        * next slice.
> > > > +        */
> > > > +       if (!(flags & ENQUEUE_WAKEUP)) {
> > > > +               u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> > > > +
> > > > +               if (delta_exec >= sysctl_sched_min_granularity)
> > > > +                       return;
> > > > +       }
> > > > +
> > > > +       rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> > > > +       se->on_latency = 1;
> > > > +}
> > > > +
> > > > +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > > +{
> > > > +       if (se->on_latency) {
> > > > +               rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> > > > +               se->on_latency = 0;
> > > > +       }
> > > > +}
> > > > +
> > > > +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> > > > +{
> > > > +       struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> > > > +
> > > > +       if (!left)
> > > > +               return NULL;
> > > > +
> > > > +       return __latency_node_2_se(left);
> > > > +}
> > > > +
> > > > +#ifdef CONFIG_SCHED_DEBUG
> > > >  /**************************************************************
> > > >   * Scheduling class statistics methods:
> > > >   */
> > > > @@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > >         check_schedstat_required();
> > > >         update_stats_enqueue_fair(cfs_rq, se, flags);
> > > >         check_spread(cfs_rq, se);
> > > > -       if (!curr)
> > > > +       if (!curr) {
> > > >                 __enqueue_entity(cfs_rq, se);
> > > > +               __enqueue_latency(cfs_rq, se, flags);
> > > > +       }
> > > >         se->on_rq = 1;
> > > >
> > > >         if (cfs_rq->nr_running == 1) {
> > > > @@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > >
> > > >         clear_buddies(cfs_rq, se);
> > > >
> > > > -       if (se != cfs_rq->curr)
> > > > +       if (se != cfs_rq->curr) {
> > > >                 __dequeue_entity(cfs_rq, se);
> > > > +               __dequeue_latency(cfs_rq, se);
> > > > +       }
> > > >         se->on_rq = 0;
> > > >         account_entity_dequeue(cfs_rq, se);
> > > >
> > > > @@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > >                  */
> > > >                 update_stats_wait_end_fair(cfs_rq, se);
> > > >                 __dequeue_entity(cfs_rq, se);
> > > > +               __dequeue_latency(cfs_rq, se);
> > > >                 update_load_avg(cfs_rq, se, UPDATE_TG);
> > > >         }
> > > >
> > > > @@ -4669,7 +4744,7 @@ static struct sched_entity *
> > > >  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > >  {
> > > >         struct sched_entity *left = __pick_first_entity(cfs_rq);
> > > > -       struct sched_entity *se;
> > > > +       struct sched_entity *latency, *se;
> > > >
> > > >         /*
> > > >          * If curr is set we have to see if its left of the leftmost entity
> > > > @@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > >                 se = cfs_rq->last;
> > > >         }
> > > >
> > > > +       /* Check for latency sensitive entity waiting for running */
> > > > +       latency = __pick_first_latency(cfs_rq);
> > > > +       if (latency && (latency != se) &&
> > > > +           wakeup_preempt_entity(latency, se) < 1)
> > > > +               se = latency;
> > > > +
> > > >         return se;
> > > >  }
> > > >
> > > > @@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> > > >                 update_stats_wait_start_fair(cfs_rq, prev);
> > > >                 /* Put 'current' back into the tree. */
> > > >                 __enqueue_entity(cfs_rq, prev);
> > > > +               __enqueue_latency(cfs_rq, prev, 0);
> > > >                 /* in !on_rq case, update occurred at dequeue */
> > > >                 update_load_avg(cfs_rq, prev, 0);
> > > >         }
> > > > @@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> > > >  void init_cfs_rq(struct cfs_rq *cfs_rq)
> > > >  {
> > > >         cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> > > > +       cfs_rq->latency_timeline = RB_ROOT_CACHED;
> > > >         u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> > > >  #ifdef CONFIG_SMP
> > > >         raw_spin_lock_init(&cfs_rq->removed.lock);
> > > > @@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency)
> > > >
> > > >         for_each_possible_cpu(i) {
> > > >                 struct sched_entity *se = tg->se[i];
> > > > +               struct rq *rq = cpu_rq(i);
> > > > +               struct rq_flags rf;
> > > > +
> > > > +               rq_lock_irqsave(rq, &rf);
> > > >
> > > > +               __dequeue_latency(se->cfs_rq, se);
> > > >                 WRITE_ONCE(se->latency_offset, latency);
> > > > +
> > > > +               rq_unlock_irqrestore(rq, &rf);
> > > >         }
> > > >
> > > >         mutex_unlock(&shares_mutex);
> > > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > > > index a15fb955092c..76bca172585c 100644
> > > > --- a/kernel/sched/sched.h
> > > > +++ b/kernel/sched/sched.h
> > > > @@ -599,6 +599,7 @@ struct cfs_rq {
> > > >  #endif
> > > >
> > > >         struct rb_root_cached   tasks_timeline;
> > > > +       struct rb_root_cached   latency_timeline;
> > > >
> > > >         /*
> > > >          * 'curr' points to currently running entity on this cfs_rq.
> > > > --
> > > > 2.17.1
> > > >
> > > >
> > > >

Re: [PATCH v5 5/7] sched/fair: Add sched group latency support

[Vincent Guittot]

On Wed, 12 Oct 2022 at 16:22, Qais Yousef <qais.yousef@arm.com> wrote:
>
> On 09/25/22 16:39, Vincent Guittot wrote:
> > Task can set its latency priority with sched_setattr(), which is then used
> > to set the latency offset of its sched_entity, but sched group entities
> > still have the default latency offset value.
> >
> > Add a latency.nice field in cpu cgroup controller to set the latency
> > priority of the group similarly to sched_setattr(). The latency priority
> > is then used to set the offset of the sched_entities of the group.
> >
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > ---
> >  Documentation/admin-guide/cgroup-v2.rst |  8 ++++
> >  kernel/sched/core.c                     | 53 +++++++++++++++++++++++++
> >  kernel/sched/fair.c                     | 33 +++++++++++++++
> >  kernel/sched/sched.h                    |  4 ++
> >  4 files changed, 98 insertions(+)
> >
> > diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> > index be4a77baf784..d8ae7e411f9c 100644
> > --- a/Documentation/admin-guide/cgroup-v2.rst
> > +++ b/Documentation/admin-guide/cgroup-v2.rst
> > @@ -1095,6 +1095,14 @@ All time durations are in microseconds.
> >          values similar to the sched_setattr(2). This maximum utilization
> >          value is used to clamp the task specific maximum utilization clamp.
> >
> > +  cpu.latency.nice
> > +     A read-write single value file which exists on non-root
> > +     cgroups.  The default is "0".
> > +
> > +     The nice value is in the range [-20, 19].
> > +
> > +     This interface file allows reading and setting latency using the
> > +     same values used by sched_setattr(2).
>
> I still don't understand how tasks will inherit the latency_nice value from
> cgroups they're attached to.

The behavior is the same as for sched_entity weight. The latency is
applied on the sched_entity of the group

>
> For example, in EAS path we operate at task level only. If the task's
> p->latency_nice = 0, but it belongs to a task group tg->latency_nice = -19;
> what should the task's latency_nice be in this case? If it's in a hierarchy,
> how would the effective value be calculated?
>
>
> Thanks
>
> --
> Qais Yousef

Re: [PATCH v5 3/7] sched: Allow sched_{get,set}attr to change latency_nice of the task

[Vincent Guittot]

On Wed, 12 Oct 2022 at 17:07, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Hello Vincent,
>
> On 9/25/2022 8:09 PM, Vincent Guittot wrote:
> > From: Parth Shah <parth@linux.ibm.com>
> >
> > Introduce the latency_nice attribute to sched_attr and provide a
> > mechanism to change the value with the use of sched_setattr/sched_getattr
> > syscall.
> >
> > Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in
> > latency_nice of the task on every sched_setattr syscall.
> >
> > Signed-off-by: Parth Shah <parth@linux.ibm.com>
> > [rebase and add a dedicated __setscheduler_latency ]
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > ---
> >  include/uapi/linux/sched.h       |  4 +++-
> >  include/uapi/linux/sched/types.h | 19 +++++++++++++++++++
> >  kernel/sched/core.c              | 24 ++++++++++++++++++++++++
> >  tools/include/uapi/linux/sched.h |  4 +++-
> >  4 files changed, 49 insertions(+), 2 deletions(-)
> >
> > diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h
> > index 3bac0a8ceab2..b2e932c25be6 100644
> > --- a/include/uapi/linux/sched.h
> > +++ b/include/uapi/linux/sched.h
> > @@ -132,6 +132,7 @@ struct clone_args {
> >  #define SCHED_FLAG_KEEP_PARAMS               0x10
> >  #define SCHED_FLAG_UTIL_CLAMP_MIN    0x20
> >  #define SCHED_FLAG_UTIL_CLAMP_MAX    0x40
> > +#define SCHED_FLAG_LATENCY_NICE              0x80
>
> [1]
>
> >
> >  #define SCHED_FLAG_KEEP_ALL  (SCHED_FLAG_KEEP_POLICY | \
> >                                SCHED_FLAG_KEEP_PARAMS)
> > @@ -143,6 +144,7 @@ struct clone_args {
> >                        SCHED_FLAG_RECLAIM             | \
> >                        SCHED_FLAG_DL_OVERRUN          | \
> >                        SCHED_FLAG_KEEP_ALL            | \
> > -                      SCHED_FLAG_UTIL_CLAMP)
> > +                      SCHED_FLAG_UTIL_CLAMP          | \
> > +                      SCHED_FLAG_LATENCY_NICE)
> >
> >  #endif /* _UAPI_LINUX_SCHED_H */
> > diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/types.h
> > index f2c4589d4dbf..db1e8199e8c8 100644
> > --- a/include/uapi/linux/sched/types.h
> > +++ b/include/uapi/linux/sched/types.h
> > @@ -10,6 +10,7 @@ struct sched_param {
> >
> >  #define SCHED_ATTR_SIZE_VER0 48      /* sizeof first published struct */
> >  #define SCHED_ATTR_SIZE_VER1 56      /* add: util_{min,max} */
> > +#define SCHED_ATTR_SIZE_VER2 60      /* add: latency_nice */
> >
> >  /*
> >   * Extended scheduling parameters data structure.
> > @@ -98,6 +99,22 @@ struct sched_param {
> >   * scheduled on a CPU with no more capacity than the specified value.
> >   *
> >   * A task utilization boundary can be reset by setting the attribute to -1.
> > + *
> > + * Latency Tolerance Attributes
> > + * ===========================
> > + *
> > + * A subset of sched_attr attributes allows to specify the relative latency
> > + * requirements of a task with respect to the other tasks running/queued in the
> > + * system.
> > + *
> > + * @ sched_latency_nice      task's latency_nice value
> > + *
> > + * The latency_nice of a task can have any value in a range of
> > + * [MIN_LATENCY_NICE..MAX_LATENCY_NICE].
> > + *
> > + * A task with latency_nice with the value of LATENCY_NICE_MIN can be
> > + * taken for a task requiring a lower latency as opposed to the task with
> > + * higher latency_nice.
> >   */
> >  struct sched_attr {
> >       __u32 size;
> > @@ -120,6 +137,8 @@ struct sched_attr {
> >       __u32 sched_util_min;
> >       __u32 sched_util_max;
> >
> > +     /* latency requirement hints */
> > +     __s32 sched_latency_nice;
> >  };
> >
> >  #endif /* _UAPI_LINUX_SCHED_TYPES_H */
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index ada2d05bd894..6a6116ea4c2c 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -7318,6 +7318,14 @@ static void __setscheduler_params(struct task_struct *p,
> >       p->rt_priority = attr->sched_priority;
> >       p->normal_prio = normal_prio(p);
> >       set_load_weight(p, true);
> > +
> > +}
> > +
> > +static void __setscheduler_latency(struct task_struct *p,
> > +             const struct sched_attr *attr)
> > +{
> > +     if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
> > +             p->latency_nice = attr->sched_latency_nice;
> >  }
> >
> >  /*
> > @@ -7460,6 +7468,13 @@ static int __sched_setscheduler(struct task_struct *p,
> >                       return retval;
> >       }
> >
> > +     if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
> > +             if (attr->sched_latency_nice > MAX_LATENCY_NICE)
> > +                     return -EINVAL;
> > +             if (attr->sched_latency_nice < MIN_LATENCY_NICE)
> > +                     return -EINVAL;
> > +     }
> > +
> >       if (pi)
> >               cpuset_read_lock();
> >
> > @@ -7494,6 +7509,9 @@ static int __sched_setscheduler(struct task_struct *p,
> >                       goto change;
> >               if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
> >                       goto change;
> > +             if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
> > +                 attr->sched_latency_nice != p->latency_nice)
> > +                     goto change;
> >
> >               p->sched_reset_on_fork = reset_on_fork;
> >               retval = 0;
> > @@ -7582,6 +7600,7 @@ static int __sched_setscheduler(struct task_struct *p,
> >               __setscheduler_params(p, attr);
> >               __setscheduler_prio(p, newprio);
> >       }
> > +     __setscheduler_latency(p, attr);
> >       __setscheduler_uclamp(p, attr);
> >
> >       if (queued) {
> > @@ -7792,6 +7811,9 @@ static int sched_copy_attr(struct sched_attr __user *uattr, struct sched_attr *a
> >           size < SCHED_ATTR_SIZE_VER1)
> >               return -EINVAL;
> >
> > +     if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) &&
> > +         size < SCHED_ATTR_SIZE_VER2)
> > +             return -EINVAL;
> >       /*
> >        * XXX: Do we want to be lenient like existing syscalls; or do we want
> >        * to be strict and return an error on out-of-bounds values?
> > @@ -8029,6 +8051,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
> >       get_params(p, &kattr);
> >       kattr.sched_flags &= SCHED_FLAG_ALL;
> >
> > +     kattr.sched_latency_nice = p->latency_nice;
> > +
> >  #ifdef CONFIG_UCLAMP_TASK
> >       /*
> >        * This could race with another potential updater, but this is fine
> > diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sched.h
> > index 3bac0a8ceab2..ecc4884bfe4b 100644
> > --- a/tools/include/uapi/linux/sched.h
> > +++ b/tools/include/uapi/linux/sched.h
> > @@ -132,6 +132,7 @@ struct clone_args {
> >  #define SCHED_FLAG_KEEP_PARAMS               0x10
> >  #define SCHED_FLAG_UTIL_CLAMP_MIN    0x20
> >  #define SCHED_FLAG_UTIL_CLAMP_MAX    0x40
> > +#define SCHED_FLAG_LATENCY_NICE              0X80
>
> Small nit.
> Can you change the "x" in "0X80" to lower case here to make it
> consistent with [1] and other #define in the same file. When building
> tools like perf, I see the following warning:

Yes, that's a typo. It should be 0x80

>
> Warning: Kernel ABI header at 'tools/include/uapi/linux/sched.h' differs from latest version at 'include/uapi/linux/sched.h'
> diff -u tools/include/uapi/linux/sched.h include/uapi/linux/sched.h
>
> Following is the output of
> diff -u tools/include/uapi/linux/sched.h include/uapi/linux/sched.h
>
> --- tools/include/uapi/linux/sched.h    2022-10-12 14:56:00.925360275 +0000
> +++ include/uapi/linux/sched.h  2022-10-12 14:56:00.917360219 +0000
> @@ -132,7 +132,7 @@
>  #define SCHED_FLAG_KEEP_PARAMS         0x10
>  #define SCHED_FLAG_UTIL_CLAMP_MIN      0x20
>  #define SCHED_FLAG_UTIL_CLAMP_MAX      0x40
> -#define SCHED_FLAG_LATENCY_NICE                0X80
> +#define SCHED_FLAG_LATENCY_NICE                0x80
>
>  #define SCHED_FLAG_KEEP_ALL    (SCHED_FLAG_KEEP_POLICY | \
>                                  SCHED_FLAG_KEEP_PARAMS)
> --
>
> >
> >  #define SCHED_FLAG_KEEP_ALL  (SCHED_FLAG_KEEP_POLICY | \
> >                                SCHED_FLAG_KEEP_PARAMS)
> > @@ -143,6 +144,7 @@ struct clone_args {
> >                        SCHED_FLAG_RECLAIM             | \
> >                        SCHED_FLAG_DL_OVERRUN          | \
> >                        SCHED_FLAG_KEEP_ALL            | \
> > -                      SCHED_FLAG_UTIL_CLAMP)
> > +                      SCHED_FLAG_UTIL_CLAMP          | \
> > +                      SCHED_FLAG_LATENCY_NICE)
> >
> >  #endif /* _UAPI_LINUX_SCHED_H */
>
> --
> Thanks and Regards,
> Prateek

Re: [PATCH v5 5/7] sched/fair: Add sched group latency support

[Qais Yousef]

On 10/12/22 17:42, Vincent Guittot wrote:
> On Wed, 12 Oct 2022 at 16:22, Qais Yousef <qais.yousef@arm.com> wrote:
> >
> > On 09/25/22 16:39, Vincent Guittot wrote:
> > > Task can set its latency priority with sched_setattr(), which is then used
> > > to set the latency offset of its sched_entity, but sched group entities
> > > still have the default latency offset value.
> > >
> > > Add a latency.nice field in cpu cgroup controller to set the latency
> > > priority of the group similarly to sched_setattr(). The latency priority
> > > is then used to set the offset of the sched_entities of the group.
> > >
> > > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > > ---
> > >  Documentation/admin-guide/cgroup-v2.rst |  8 ++++
> > >  kernel/sched/core.c                     | 53 +++++++++++++++++++++++++
> > >  kernel/sched/fair.c                     | 33 +++++++++++++++
> > >  kernel/sched/sched.h                    |  4 ++
> > >  4 files changed, 98 insertions(+)
> > >
> > > diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> > > index be4a77baf784..d8ae7e411f9c 100644
> > > --- a/Documentation/admin-guide/cgroup-v2.rst
> > > +++ b/Documentation/admin-guide/cgroup-v2.rst
> > > @@ -1095,6 +1095,14 @@ All time durations are in microseconds.
> > >          values similar to the sched_setattr(2). This maximum utilization
> > >          value is used to clamp the task specific maximum utilization clamp.
> > >
> > > +  cpu.latency.nice
> > > +     A read-write single value file which exists on non-root
> > > +     cgroups.  The default is "0".
> > > +
> > > +     The nice value is in the range [-20, 19].
> > > +
> > > +     This interface file allows reading and setting latency using the
> > > +     same values used by sched_setattr(2).
> >
> > I still don't understand how tasks will inherit the latency_nice value from
> > cgroups they're attached to.
> 
> The behavior is the same as for sched_entity weight. The latency is
> applied on the sched_entity of the group

But this is the point I am raising. Not all users behave the same as weight.

In EAS we just look at the effective value of the task (see uclamp for
example). We don't care about the group value except to calculate how it
impacts the task's value.

Or am I missing something here?


Cheers

--
Qais Yousef

Re: [PATCH v5 0/7] Add latency priority for CFS class

[Vincent Guittot]

Hi Prateek,

Thanks for testing the patchset on AMD and the test report below.

On Wed, 12 Oct 2022 at 16:54, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Hello Vincent,
>
> Sharing results from testing on dual socket Zen3 system (2 x 64C/128T)
>
> tl;dr
>
> o I don't see any regression when workloads are running with
>   DEFAULT_LATENCY_NICE
> o I can reproduce similar results as one reported in Patch 4 for
>   hackbench with latency nice 19 and hackbench and cyclictest
>   with various combination of latency nice values.
> o I can see improvements to tail latency for schbench with hackbench
>   running in the background.
> o There is an unexpected non-linear behavior observed for couple of
>   cases that I cannot explain yet. (Marked with "^" in detailed results)
>   I have not yet gotten to the bottom of it but if I've missed
>   something, please do let me know.
>
> Detailed results are shared below:
>
> On 9/25/2022 8:09 PM, Vincent Guittot wrote:
> > This patchset restarts the work about adding a latency priority to describe
> > the latency tolerance of cfs tasks.
> >
> > The patches [1-3] have been done by Parth:
> > https://lore.kernel.org/lkml/20200228090755.22829-1-parth@linux.ibm.com/
> >
> > I have just rebased and moved the set of latency priority outside the
> > priority update. I have removed the reviewed tag because the patches
> > are 2 years old.
> >
> > This aims to be a generic interface and the following patches is one use
> > of it to improve the scheduling latency of cfs tasks.
> >
> > The patch [4] uses latency nice priority to define a latency offset
> > and then decide if a cfs task can or should preempt the current
> > running task. The patch gives some tests results with cyclictests and
> > hackbench to highlight the benefit of latency priority for short
> > interactive task or long intensive tasks.
> >
> > Patch [5] adds the support of latency nice priority to task group by
> > adding a cpu.latency.nice field. The range is [-20:19] as for setting task
> > latency priority.
> >
> > Patch [6] makes sched_core taking into account the latency offset.
> >
> > Patch [7] adds a rb tree to cover some corner cases where the latency
> > sensitive task (priority < 0) is preempted by high priority task (RT/DL)
> > or fails to preempt them. This patch ensures that tasks will have at least
> > a slice of sched_min_granularity in priority at wakeup. The patch gives
> > results to show the benefit in addition to patch 4.
> >
> > I have also backported the patchset on a dragonboard RB3 with an android
> > mainline kernel based on v5.18 for a quick test. I have used the
> > TouchLatency app which is part of AOSP and described to be a very good
> > test to highlight jitter and jank frame sources of a system [1].
> > In addition to the app, I have added some short running tasks waking-up
> > regularly (to use the 8 cpus for 4 ms every 37777us) to stress the system
> > without overloading it (and disabling EAS). The 1st results shows that the
> > patchset helps to reduce the missed deadline frames from 5% to less than
> > 0.1% when the cpu.latency.nice of task group are set.
> >
> > I have also tested the patchset with the modified version of the alsa
> > latency test that has been shared by Tim. The test quickly xruns with
> > default latency nice priority 0 but is able to run without underuns with
> > a latency -20 and hackbench running simultaneously.
> >
> >
> > [1] https://source.android.com/docs/core/debug/eval_perf#touchlatency
>
> Following are the results from running standard benchmarks on a
> dual socket Zen3 (2 x 64C/128T) machine configured in different
> NPS modes.
>
> NPS Modes are used to logically divide single socket into
> multiple NUMA region.
> Following is the NUMA configuration for each NPS mode on the system:
>
> NPS1: Each socket is a NUMA node.
>     Total 2 NUMA nodes in the dual socket machine.
>
>     Node 0: 0-63,   128-191
>     Node 1: 64-127, 192-255
>
> NPS2: Each socket is further logically divided into 2 NUMA regions.
>     Total 4 NUMA nodes exist over 2 socket.
>
>     Node 0: 0-31,   128-159
>     Node 1: 32-63,  160-191
>     Node 2: 64-95,  192-223
>     Node 3: 96-127, 223-255
>
> NPS4: Each socket is logically divided into 4 NUMA regions.
>     Total 8 NUMA nodes exist over 2 socket.
>
>     Node 0: 0-15,    128-143
>     Node 1: 16-31,   144-159
>     Node 2: 32-47,   160-175
>     Node 3: 48-63,   176-191
>     Node 4: 64-79,   192-207
>     Node 5: 80-95,   208-223
>     Node 6: 96-111,  223-231
>     Node 7: 112-127, 232-255
>
> Benchmark Results:
>
> Kernel versions:
> - tip:          5.19.0 tip sched/core
> - latency_nice: 5.19.0 tip sched/core + this series
>
> When we started testing, the tip was at:
> commit 7e9518baed4c ("sched/fair: Move call to list_last_entry() in detach_tasks")
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> ~ hackbench - DEFAULT_LATENCY_NICE ~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> NPS1
>
> Test:                   tip                 latency_nice
>  1-groups:         4.23 (0.00 pct)         4.06 (4.01 pct)
>  2-groups:         4.93 (0.00 pct)         4.89 (0.81 pct)
>  4-groups:         5.32 (0.00 pct)         5.31 (0.18 pct)
>  8-groups:         5.46 (0.00 pct)         5.54 (-1.46 pct)
> 16-groups:         7.31 (0.00 pct)         7.33 (-0.27 pct)
>
> NPS2
>
> Test:                   tip                 latency_nice
>  1-groups:         4.19 (0.00 pct)         4.12 (1.67 pct)
>  2-groups:         4.77 (0.00 pct)         4.82 (-1.04 pct)
>  4-groups:         5.15 (0.00 pct)         5.17 (-0.38 pct)
>  8-groups:         5.47 (0.00 pct)         5.48 (-0.18 pct)
> 16-groups:         6.63 (0.00 pct)         6.65 (-0.30 pct)
>
> NPS4
>
> Test:                   tip                  latency_nice
>  1-groups:         4.23 (0.00 pct)         4.31 (-1.89 pct)
>  2-groups:         4.78 (0.00 pct)         4.75 (0.62 pct)
>  4-groups:         5.17 (0.00 pct)         5.24 (-1.35 pct)
>  8-groups:         5.63 (0.00 pct)         5.59 (0.71 pct)
> 16-groups:         7.88 (0.00 pct)         7.09 (10.02 pct)
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> ~ schbench - DEFAULT_LATENCY_NICE ~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> NPS1
>
> #workers:     tip                  latency_nice
>   1:      22.00 (0.00 pct)        21.00 (4.54 pct)
>   2:      34.00 (0.00 pct)        34.00 (0.00 pct)
>   4:      37.00 (0.00 pct)        40.00 (-8.10 pct)
>   8:      55.00 (0.00 pct)        49.00 (10.90 pct)
>  16:      69.00 (0.00 pct)        66.00 (4.34 pct)
>  32:     113.00 (0.00 pct)       117.00 (-3.53 pct)
>  64:     219.00 (0.00 pct)       242.00 (-10.50 pct) *
>  64:     219.00 (0.00 pct)       194.00 (11.41 pct)  [Verification Run]
> 128:     506.00 (0.00 pct)       513.00 (-1.38 pct)
> 256:     45440.00 (0.00 pct)     44992.00 (0.98 pct)
> 512:     76672.00 (0.00 pct)     83328.00 (-8.68 pct)
>
> NPS2
>
> #workers:     tip                   latency_nice
>   1:      31.00 (0.00 pct)        20.00 (35.48 pct)
>   2:      36.00 (0.00 pct)        28.00 (22.22 pct)
>   4:      45.00 (0.00 pct)        37.00 (17.77 pct)
>   8:      47.00 (0.00 pct)        51.00 (-8.51 pct)
>  16:      66.00 (0.00 pct)        69.00 (-4.54 pct)
>  32:     114.00 (0.00 pct)       113.00 (0.87 pct)
>  64:     215.00 (0.00 pct)       215.00 (0.00 pct)
> 128:     495.00 (0.00 pct)       529.00 (-6.86 pct)  *
> 128:     495.00 (0.00 pct)       416.00 (15.95  pct) [Verification Run]
> 256:     48576.00 (0.00 pct)     46912.00 (3.42 pct)
> 512:     79232.00 (0.00 pct)     82560.00 (-4.20 pct)
>
> NPS4
>
> #workers:     tip                   latency_nice
>   1:      30.00 (0.00 pct)        34.00 (-13.33 pct)
>   2:      34.00 (0.00 pct)        42.00 (-23.52 pct)
>   4:      41.00 (0.00 pct)        42.00 (-2.43 pct)
>   8:      60.00 (0.00 pct)        55.00 (8.33 pct)
>  16:      68.00 (0.00 pct)        69.00 (-1.47 pct)
>  32:     116.00 (0.00 pct)       115.00 (0.86 pct)
>  64:     224.00 (0.00 pct)       223.00 (0.44 pct)
> 128:     495.00 (0.00 pct)       677.00 (-36.76 pct) *
> 128:     495.00 (0.00 pct)       388.00 (21.61 pct)  [Verification Run]
> 256:     45888.00 (0.00 pct)     44608.00 (2.78 pct)
> 512:     78464.00 (0.00 pct)     81536.00 (-3.91 pct)
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> ~ tbench - DEFAULT_LATENCY_NICE ~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> NPS1
>
> Clients:      tip                  latency_nice
>     1    550.66 (0.00 pct)       546.63 (-0.73 pct)
>     2    1009.69 (0.00 pct)      1016.40 (0.66 pct)
>     4    1795.32 (0.00 pct)      1773.95 (-1.19 pct)
>     8    2971.16 (0.00 pct)      2930.26 (-1.37 pct)
>    16    4627.98 (0.00 pct)      4727.82 (2.15 pct)
>    32    8065.15 (0.00 pct)      9019.11 (11.82 pct)
>    64    14994.32 (0.00 pct)     15100.22 (0.70 pct)
>   128    5175.73 (0.00 pct)      18223.69 (252.09 pct) *
>   128    20029.53 (0.00 pct)     20517.17 (2.43 pct)   [Verification Run]
>   256    48763.57 (0.00 pct)     44463.63 (-8.81 pct)
>   512    43780.78 (0.00 pct)     44170.21 (0.88 pct)
>  1024    40341.84 (0.00 pct)     40883.10 (1.34 pct)
>
> NPS2
>
> Clients:      tip                   latency_nice
>     1    551.06 (0.00 pct)       547.43 (-0.65 pct)
>     2    1000.76 (0.00 pct)      1014.83 (1.40 pct)
>     4    1737.02 (0.00 pct)      1742.30 (0.30 pct)
>     8    2992.31 (0.00 pct)      2951.59 (-1.36 pct)
>    16    4579.29 (0.00 pct)      4558.05 (-0.46 pct)
>    32    9120.73 (0.00 pct)      8122.06 (-10.94 pct) *
>    32    8814.62 (0.00 pct)      8965.54 (1.71 pct)   [Verification Run]
>    64    14918.58 (0.00 pct)     14890.93 (-0.18 pct)
>   128    20830.61 (0.00 pct)     20410.48 (-2.01 pct)
>   256    47708.18 (0.00 pct)     45312.84 (-5.02 pct) *
>   256    44941.88 (0.00 pct)     44555.92 (-0.85 pct) [Verification Run]
>   512    43721.79 (0.00 pct)     43653.43 (-0.15 pct)
>  1024    40920.49 (0.00 pct)     41162.17 (0.59 pct)
>
> NPS4
>
> Clients:      tip                   latency_nice
>     1    549.22 (0.00 pct)       539.81 (-1.71 pct)
>     2    1000.08 (0.00 pct)      1010.12 (1.00 pct)
>     4    1794.78 (0.00 pct)      1736.06 (-3.27 pct)
>     8    3008.50 (0.00 pct)      2952.68 (-1.85 pct)
>    16    4804.71 (0.00 pct)      4454.17 (-7.29 pct)  *
>    16    4391.10 (0.00 pct)      4497.43 (2.42 pct)   [Verification Run]
>    32    9156.57 (0.00 pct)      8820.05 (-3.67 pct)
>    64    14901.45 (0.00 pct)     14786.25 (-0.77 pct)
>   128    20771.20 (0.00 pct)     19955.11 (-3.92 pct)
>   256    47033.88 (0.00 pct)     44937.51 (-4.45 pct)
>   512    43429.01 (0.00 pct)     42638.81 (-1.81 pct)
>  1024    39271.27 (0.00 pct)     40044.17 (1.96 pct)
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> ~ stream - DEFAULT_LATENCY_NICE ~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> NPS1
>
> 10 Runs:
>
> Test:           tip                     latency_nice
>  Copy:   336311.52 (0.00 pct)    326015.98 (-3.06 pct)
> Scale:   212955.82 (0.00 pct)    208667.27 (-2.01 pct)
>   Add:   251518.23 (0.00 pct)    237286.20 (-5.65 pct)
> Triad:   262077.88 (0.00 pct)    258949.80 (-1.19 pct)
>
> 100 Runs:
>
> Test:           tip                     latency_nice
>  Copy:   339533.83 (0.00 pct)    335126.73 (-1.29 pct)
> Scale:   194736.72 (0.00 pct)    221151.24 (13.56 pct)
>   Add:   218294.54 (0.00 pct)    251427.43 (15.17 pct)
> Triad:   262371.40 (0.00 pct)    260100.85 (-0.86 pct)
>
> NPS2
>
> 10 Runs:
>
> Test:           tip                     latency_nice
>  Copy:   335277.15 (0.00 pct)    339614.38 (1.29 pct)
> Scale:   220990.24 (0.00 pct)    221052.78 (0.02 pct)
>   Add:   264156.13 (0.00 pct)    263684.19 (-0.17 pct)
> Triad:   268707.53 (0.00 pct)    272610.96 (1.45 pct)
>
> 100 Runs:
>
> Test:           tip                     latency_nice
>  Copy:   334913.73 (0.00 pct)    339001.88 (1.22 pct)
> Scale:   230522.47 (0.00 pct)    229848.86 (-0.29 pct)
>   Add:   264567.28 (0.00 pct)    264288.34 (-0.10 pct)
> Triad:   272974.23 (0.00 pct)    272045.17 (-0.34 pct)
>
> NPS4
>
> 10 Runs:
>
> Test:           tip                     latency_nice
>  Copy:   299432.31 (0.00 pct)    307649.18 (2.74 pct)
> Scale:   217998.17 (0.00 pct)    205763.70 (-5.61 pct)
>   Add:   234305.46 (0.00 pct)    226381.75 (-3.38 pct)
> Triad:   244369.15 (0.00 pct)    254225.30 (4.03 pct)
>
> 100 Runs:
>
> Test:           tip                     latency_nice
>  Copy:   344421.25 (0.00 pct)    322189.81 (-6.45 pct)
> Scale:   237998.44 (0.00 pct)    227709.58 (-4.32 pct)
>   Add:   257501.82 (0.00 pct)    244009.58 (-5.23 pct)
> Triad:   267686.50 (0.00 pct)    251840.25 (-5.91 pct)
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> ~ Test cases for Latency Nice ~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Note: Latency Nice might be referred to as LN in the data below. Latency Nice
> value was set using a wrapper script for all the workload threads during the
> testing.
> All the test results reported below are for NPS1 configuration.
>
> o Hackbench Pipes (100000 loops, threads)
>
> Test:                  tip                 Latency Nice: -20        Latency Nice: 0         Latency Nice: 19
>  1-groups:         4.23 (0.00 pct)          4.39 (-3.78 pct)        3.99 (5.67 pct)         3.88 (8.27 pct)
>  2-groups:         4.93 (0.00 pct)          4.91 (0.40 pct)         4.69 (4.86 pct)         4.59 (6.89 pct)
>  4-groups:         5.32 (0.00 pct)          5.37 (-0.93 pct)        5.19 (2.44 pct)         5.05 (5.07 pct)
>  8-groups:         5.46 (0.00 pct)          5.90 (-8.05 pct)        5.34 (2.19 pct)         5.17 (5.31 pct)
> 16-groups:         7.31 (0.00 pct)          7.99 (-9.30 pct)        6.96 (4.78 pct)         6.51 (10.94 pct)
>
> o Only Hackbench with different Latency Nice Values
>
> > Loops: 100000
>
> - Pipe (Process)
>
>   Test:            Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>  1-groups:         3.77 (0.00 pct)         4.23 (-12.20 pct)       3.83 (-1.59 pct)
>  2-groups:         4.39 (0.00 pct)         4.73 (-7.74 pct)        4.31 (1.82 pct)
>  4-groups:         4.80 (0.00 pct)         5.07 (-5.62 pct)        4.68 (2.50 pct)
>  8-groups:         4.95 (0.00 pct)         5.68 (-14.74 pct)       4.76 (3.83 pct)
> 16-groups:         6.47 (0.00 pct)         7.87 (-21.63 pct)       6.08 (6.02 pct)
>
> - Socket (Thread)
>
> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>  1-groups:         6.08 (0.00 pct)         5.99 (1.48 pct)         6.08 (0.00 pct)
>  2-groups:         6.15 (0.00 pct)         6.25 (-1.62 pct)        6.14 (0.16 pct)
>  4-groups:         6.39 (0.00 pct)         6.42 (-0.46 pct)        6.44 (-0.78 pct)
>  8-groups:         8.51 (0.00 pct)         9.01 (-5.87 pct)        8.36 (1.76 pct)
> 16-groups:        12.48 (0.00 pct)        15.32 (-22.75 pct)      12.72 (-1.92 pct)
>
> - Socket (Process)
>
> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>  1-groups:         6.44 (0.00 pct)         5.50 (14.59 pct)  ^     6.43 (0.15 pct)
>  2-groups:         6.55 (0.00 pct)         5.56 (15.11 pct)  ^     6.36 (2.90 pct)
>  4-groups:         6.74 (0.00 pct)         6.19 (8.16 pct)   ^     6.69 (0.74 pct)
>  8-groups:         8.03 (0.00 pct)         8.29 (-3.23 pct)        8.02 (0.12 pct)
> 16-groups:        12.25 (0.00 pct)        14.11 (-15.18 pct)      12.41 (-1.30 pct)

I don't see any improvement with LN:-20 but only for LN:19

How many iterations do you run ? Could it be that the results vary
between iterations ? For some configuration I have a stddev of 10-20%
for LN:0 and LN:-20

>
> > Loops: 2160 (Same as in testing)
>
> - Pipe (Thread)
>
> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>  1-groups:         0.10 (0.00 pct)         0.12 (-20.00 pct)       0.10 (0.00 pct)
>  2-groups:         0.12 (0.00 pct)         0.15 (-25.00 pct)       0.11 (8.33 pct)
>  4-groups:         0.14 (0.00 pct)         0.18 (-28.57 pct)       0.15 (-7.14 pct)
>  8-groups:         0.17 (0.00 pct)         0.24 (-41.17 pct)       0.17 (0.00 pct)
> 16-groups:         0.26 (0.00 pct)         0.33 (-26.92 pct)       0.21 (19.23 pct)
>
> - Pipe (Process)
>
> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>  1-groups:         0.10 (0.00 pct)         0.12 (-20.00 pct)       0.10 (0.00 pct)
>  2-groups:         0.12 (0.00 pct)         0.16 (-33.33 pct)       0.12 (0.00 pct)
>  4-groups:         0.14 (0.00 pct)         0.17 (-21.42 pct)       0.13 (7.14 pct)
>  8-groups:         0.16 (0.00 pct)         0.24 (-50.00 pct)       0.16 (0.00 pct)
> 16-groups:         0.23 (0.00 pct)         0.33 (-43.47 pct)       0.19 (17.39 pct)
>
> - Socket (Thread)
>
> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>  1-groups:         0.19 (0.00 pct)         0.18 (5.26 pct)         0.18 (5.26 pct)
>  2-groups:         0.21 (0.00 pct)         0.21 (0.00 pct)         0.20 (4.76 pct)
>  4-groups:         0.22 (0.00 pct)         0.25 (-13.63 pct)       0.22 (0.00 pct)
>  8-groups:         0.27 (0.00 pct)         0.36 (-33.33 pct)       0.27 (0.00 pct)
> 16-groups:         0.42 (0.00 pct)         0.55 (-30.95 pct)       0.40 (4.76 pct)
>
> - Socket (Process)
>
> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>  1-groups:         0.17 (0.00 pct)         0.17 (0.00 pct)         0.17 (0.00 pct)
>  2-groups:         0.19 (0.00 pct)         0.20 (-5.26 pct)        0.19 (0.00 pct)
>  4-groups:         0.20 (0.00 pct)         0.22 (-10.00 pct)       0.20 (0.00 pct)
>  8-groups:         0.25 (0.00 pct)         0.32 (-28.00 pct)       0.25 (0.00 pct)
> 16-groups:         0.40 (0.00 pct)         0.51 (-27.50 pct)       0.39 (2.50 pct)
>
> o Hackbench and Cyclictest in NPS1 configuration
>
> perf bench sched messaging -p -t -l 100000 -g 16&
> cyclictest --policy other -D 5 -q -n -H 20000
>
> -----------------------------------------------------------------------------------------------------------------
> |Hackbench     |      Cyclictest LN = 19        |         Cyclictest LN = 0       |      Cyclictest LN = -20    |
> |LN            |--------------------------------|---------------------------------|-----------------------------|
> |v             |   Min  |   Avg   |  Max        |     Min  |   Avg   |  Max       |     Min  |   Avg   |  Max   |
> |--------------|--------|---------|-------------|----------|---------|------------|----------|---------|--------|
> |0             |  54.00 |  117.00 | 3021.67     |    53.67 |  65.33  | 133.00     |    53.67 |  65.00  | 201.33 |  ^
> |19            |  50.00 |  100.67 | 3099.33     |    41.00 |  64.33  | 1014.33    |    54.00 |  63.67  | 213.33 |
> |-20           |  53.00 |  169.00 | 11661.67    |    53.67 |  217.33 | 14313.67   |    46.00 |  61.33  | 236.00 |  ^
> -----------------------------------------------------------------------------------------------------------------

The latency results look good with Cyclictest LN:0 and hackbench LN:0.
133us max latency. This suggests that your system is not overloaded
and cyclictest doesn't really compete with others to run.

>
> o Hackbench and schbench in NPS1 configuration
>
> perf bench sched messaging -p -t -l 1000000 -g 16&
> schebcnh -m 1 -t 64 -s 30s
>
> ------------------------------------------------------------------------------------------------------------
> |Hackbench     |   schbench LN = 19         |        schbench LN = 0         |       schbench LN = -20     |
> |LN            |----------------------------|--------------------------------|-----------------------------|
> |v             |  90th  |  95th  |  99th    |   90th  |  95th   |  99th      |   90th  |   95th   | 99th   |
> |--------------|--------|--------|----------|---------|---------|------------|---------|----------|--------|
> |0             |  4264  |  6744  |  15664   |   17952 |  32672  |  55488     |   15088 |   25312  | 50112  |
> |19            |  288   |  613   |  2332    |   274   |  1015   |  3628      |   374   |   1394   | 4424   |
> |-20           |  35904 |  47680 |  79744   |   87168 |  113536 |  176896    |   13008 |   21216  | 42560  |   ^
> ------------------------------------------------------------------------------------------------------------

For the schbench, your test is 30 seconds long which is longer than
the duration of perf bench sched messaging -p -t -l 1000000 -g 16&

The duration of the latter varies depending of latency nice value so
schbench is disturb more time in some cases
>
> o SpecJBB Multi-JVM
>
> ---------------------------------------------
> | Latency Nice  |         0  |           19 |
> ---------------------------------------------
> | max-jOPS      |      100%  |      109.92% |
> | critical-jOPS |      100%  |      153.70% |
> ---------------------------------------------
>
> In most cases, latency nice delivers what it promises.
> Some cases marked with "^" have shown anomalies or non-linear behavior
> that is yet to be root caused. If you've seen something similar during
> your testing, I would love to know what could lead to such a behavior.

I haven't seen anything like the results that you tagged with ^. As a
side note, the numbers of groups (g1 g4 g8 g1) that I used with
hackbench, have been chosen according to my 8 cores system. Your
system is much larger and hackbench may not overload it with such a
small number of groups. Maybe you could try with g32 g64 g128 g256 ?



>
> If you would like more details on the benchmarks results reported above
> or if there is any specific workload you would like me to test on the
> Zen3 machine, please do let me know.
>
> >
> > [..snip..]
> >
>
> --
> Thanks and Regards,
> Prateek
> --
> --
> Thanks and Regards,
> Prateek

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Youssef Esmat]

On Wed, Oct 12, 2022 at 10:21 AM Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> Le mardi 11 oct. 2022 à 18:54:27 (-0500), Youssef Esmat a écrit :
> > Hi Vincent,
> >
> > On Tue, Oct 11, 2022 at 12:10 PM Vincent Guittot
> > <vincent.guittot@linaro.org> wrote:
>
> ...
>
> > > > >               latency 0    latency -20
> > > > > Min Latencies:    60           61
> > > > > Avg Latencies:  1077           86
> > > > > Max Latencies: 87311          444
> > > > > 50% latencies:    92           85
> > > > > 75% latencies:   554           90
> > > > > 85% latencies:  1019           93
> > > > > 90% latencies:  1346           96
> > > > > 95% latencies:  5400          100
> > > > > 99% latencies: 19044          110
> > > > >
> > > > > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > > > > ---
> > > >
> > > > The ability to boost the latency sensitivity of a task seems very
> > > > interesting. I have been playing around with these changes and have
> > > > some observations.
> > > >
> > > > I tried 2 bursty tasks affinitized to the same CPU. The tasks sleep
> > > > for 1ms and run for 10ms in a loop. I first tried it without adjusting
> > > > the latency_nice value and took perf sched traces:
> > >
> > > The CPU is overloaded almost all the time as it can't run the 2 tasks
> > > (2*10ms every 11ms)
> > >
> > > >
> > > > latency_test:7040     |   2447.137 ms |        8 | avg:   6.546 ms |
> > > > max:  10.674 ms | max start:   353.809487 s | max end:   353.820161 s
> > > > latency_test:7028     |   2454.777 ms |        7 | avg:   4.494 ms |
> > > > max:  10.609 ms | max start:   354.804386 s | max end:   354.814995 s
> > > >
> > > > Everything looked as expected, for a 5s run they had similar runtime
> > > > and latency.
> > > >
> > > > I then adjusted one task to have a latency_nice of -20 (pid 8614
> > > > below) and took another set of traces:
> > > >
> > > > latency_test:8618     |   1845.534 ms |      131 | avg:   9.764 ms |
> > > > max:  10.686 ms | max start:  1405.737905 s | max end:  1405.748592 s
> > > > latency_test:8614     |   3033.635 ms |       16 | avg:   3.559 ms |
> > > > max:  10.467 ms | max start:  1407.594751 s | max end:  1407.605218 s
> > > >
> > > > The task with -20 latency_nice had significantly more runtime. The
> > > > average latency was improved but the max roughly stayed the same. As
> > > > expected the one with latency_nice value of 0 experienced more
> > > > switches, but so did the one with latency_nice of -20.
> > >
> > > Your results look unexpected because the vruntime of the tasks is not
> > > modified. So I can imagine that the thread with the low latency runs
> > > first up to the offset at the beg of the test but then they should
> > > switch regularly. I have tried a similar test with a modified rt-app
> > > and the result seems ok. I have a small difference but not the
> > > difference that you see.
> > >
> > > Could you share more details about your setup ? I'm going to try to
> > > reproduce your sequence
> >
> > I was using an intel core i7 with this frequency details:
> > CPU MHz:                         4200.000
> > CPU max MHz:                  4800.0000
> > CPU min MHz:                   400.0000
> >
> > This is a snippet of the test I was using:
> >
> > struct sched_attr attr;
> > memset(&attr, 0, sizeof(struct sched_attr));
> > attr.size = sizeof(struct sched_attr);
> > attr.sched_latency_nice = nice_latency;
> > attr.sched_flags = SCHED_FLAG_LATENCY_NICE;
> >
> > // set nice latency value
> > int res = syscall(__NR_sched_setattr, 0, &attr, 0);
> >
> > while(1){
> >   // wake up every ms
> >   usleep(1000);
> >   for(int i = 0; i < 40000000; i++){}
> > }
>
> Between v2 and v3, the sched_feat(GENTLE_FAIR_SLEEPERS) has diseappeared when
> I moved the computation of latency_offset at the setting of the latency prio
> instead of runtime. As a result, the latency nice task can preempt up to
> (sysctl_sched_latency - sysctl_sched_wakeup_granularity) but other threads are
> cap to half of sysctl_sched_latency at wakeup.
>
> Could you try the patch below ?

Thanks Vincent. That improved the runtime for the tests above.

Running two bursty tasks (1ms sleep and 10ms running loop) one of
which had a latency_nice of -20 (pid 9209) and the other 0:

latency_test:9205     |   2429.467 ms |       14 | avg:   6.293 ms |
max:  11.340 ms | max start:   209.543482 s | max end:   209.554821 s
latency_test:9209     |   2467.423 ms |       17 | avg:   3.346 ms |
max:  11.475 ms | max start:   208.374517 s | max end:   208.385992 s

The task with -20 latency_nice had an improved average latency, but
the max roughly stayed the same.

Running the same test with 3 bursty tasks, one of which had a
latency_nice of -20 and the other two with latency_nice = 0. The
results for this case roughly stayed the same after the changes:

latency_test:26088    |   1641.458 ms |      158 | avg:  19.613 ms |
max:  33.000 ms | max start:   871.707231 s | max end:   871.740231 s
latency_test:26295    |   1639.766 ms |      238 | avg:  10.259 ms |
max:  24.289 ms | max start:   873.917231 s | max end:   873.941519 s
latency_test:26401    |   1643.580 ms |      241 | avg:  10.200 ms |
max:  22.124 ms | max start:   876.289233 s | max end:   876.311357 s

The task with latency_nice -20 seemed to have the highest average and
max latencies.

I also tried an additional test where we had a task that was cpu bound
(while(1){}) and a task that ran for 1ms and slept for 1ms. The cpu
bound task had the latency_nice value of -20 and the bursty had
latency_nice of 0.

latency_test:17353    |   4557.699 ms |      356 | avg:   1.058 ms |
max:   1.181 ms | max start:  4324.806123 s | max end:  4324.807304 s
latency_test:17452    |    377.804 ms |        1 | avg:   0.000 ms |
max:   0.000 ms | max start:     0.000000 s | max end:     0.000000 s

The cpu bound task (pid 17353 above) had significantly more runtime.
If I reran the test with latency_nice = 0 for both:

latency_test:20748    |   2478.014 ms |     2367 | avg:   1.037 ms |
max:   1.182 ms | max start:  4460.769240 s | max end:  4460.770423 s
latency_test:20972    |   2455.888 ms |        3 | avg:   0.001 ms |
max:   0.002 ms | max start:  4460.628756 s | max end:  4460.628758 s

Would decreasing the latency_offset as the task runs and increasing it
as it sleeps help here?


>
> ---
>  kernel/sched/core.c | 19 ++++++++++++++++---
>  1 file changed, 16 insertions(+), 3 deletions(-)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 8b44685ae247..68f9a83d7089 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -1287,11 +1287,15 @@ static void set_load_weight(struct task_struct *p, bool update_load)
>  static void set_latency_offset(struct task_struct *p)
>  {
>         long weight = sched_latency_to_weight[p->latency_prio];
> +       unsigned long period = sysctl_sched_latency;
>         s64 offset;
>
> -       offset = sysctl_sched_latency * weight;
> +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> +               period >>= 1;
> +       offset = period * weight;
>         offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX);
>         p->se.latency_offset = (long)offset;
> +       trace_printk("set_latency_offset pid %d offset %ld", p->pid, offset);
>  }
>
>  #ifdef CONFIG_UCLAMP_TASK
> @@ -10894,12 +10898,17 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,
>  static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
>                                     struct cftype *cft)
>  {
> +       unsigned long period = sysctl_sched_latency;
>         int last_delta = INT_MAX;
>         int prio, delta;
>         s64 weight;
>
> +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> +               period >>= 1;
> +
>         weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX;
> -       weight = div_s64(weight, sysctl_sched_latency);
> +       period = sysctl_sched_latency;
> +       weight = div_s64(weight, period);
>
>         /* Find the closest nice value to the current weight */
>         for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) {
> @@ -10915,6 +10924,7 @@ static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
>  static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
>                                      struct cftype *cft, s64 nice)
>  {
> +       unsigned long period;
>         s64 latency_offset;
>         long weight;
>         int idx;
> @@ -10926,7 +10936,10 @@ static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
>         idx = array_index_nospec(idx, LATENCY_NICE_WIDTH);
>         weight = sched_latency_to_weight[idx];
>
> -       latency_offset = sysctl_sched_latency * weight;
> +       period = sysctl_sched_latency;
> +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> +               period >>= 1;
> +       latency_offset = period * weight;
>         latency_offset = div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX);
>
>         return sched_group_set_latency(css_tg(css), latency_offset);
> --
> 2.17.1
>
> >
> > >
> > > >
> > > > Also tried running the same test but instead of using latency nice I
> > > > adjusted the nice value as a comparison. In that case one task had a
> > > > nice of -5 and the other was 0.
> > > >
> > > > nice_test:25219       |   1216.839 ms |      242 | avg:  10.295 ms |
> > > > max:  11.927 ms | max start:  5877.881279 s | max end:  5877.893206 s
> > > > nice_test:25235       |   3711.788 ms |        6 | avg:   1.026 ms |
> > > > max:   6.143 ms | max start:  5875.603741 s | max end:  5875.609883 s
> > > >
> > > > As expected the one with a nice value of -5 had more runtime but also
> > > > had better latency numbers than in the previous case of using
> > > > latency_nice.
> > > >
> > > > I also tried a similar test with 3 bursty tasks instead of two. In
> > > > this case all tasks had a latency_nice of 0:
> > > >
> > > > latency_test:11467    |   1641.131 ms |      161 | avg:  17.489 ms |
> > > > max:  21.011 ms | max start:  1542.656275 s | max end:  1542.677286 s
> > > > latency_test:11463    |   1644.809 ms |      161 | avg:  11.994 ms |
> > > > max:  25.012 ms | max start:  1545.657776 s | max end:  1545.682788 s
> > > > latency_test:11478    |   1643.211 ms |      160 | avg:  11.465 ms |
> > > > max:  21.012 ms | max start:  1546.159026 s | max end:  1546.180038 s
> > > >
> > > > Next I tried two tasks with a latency_nice of 0 and a third one had a
> > > > latency_nice of -20 (pid 11763 below):
> > > >
> > > > latency_test:11763    |   1645.482 ms |      159 | avg:  19.634 ms |
> > > > max:  31.016 ms | max start:  1623.834862 s | max end:  1623.865877 s
> > > > latency_test:11750    |   1644.276 ms |      259 | avg:   9.985 ms |
> > > > max:  21.012 ms | max start:  1623.953921 s | max end:  1623.974933 s
> > > > latency_test:11747    |   1642.745 ms |      262 | avg:   9.079 ms |
> > > > max:  25.013 ms | max start:  1620.980435 s | max end:  1621.005447 s
> > > >
> > > > In this case it seemed like the runtime was not affected by the
> > > > latency_nice value, but strangely the task with the latency nice of
> > > > -20 had a worse average and max latency than the other two. The
> > > > context switch times are also increased from the previous case.
> > > >
> > > > Have we considered an approach where the task that is marked as
> > > > latency sensitive gets a boosted nice value when it sleeps and is
> > > > either scaled down exponentially as it runs, or immediately reset to
> > > > its default when it runs for one tick?
> > > >
> > > > Thanks,
> > > > Youssef
> > > >
> > > >
> > > > >  include/linux/sched.h |  2 +
> > > > >  kernel/sched/fair.c   | 96 +++++++++++++++++++++++++++++++++++++++++--
> > > > >  kernel/sched/sched.h  |  1 +
> > > > >  3 files changed, 96 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > > > > index a74cad08e91e..0b92674e3664 100644
> > > > > --- a/include/linux/sched.h
> > > > > +++ b/include/linux/sched.h
> > > > > @@ -547,6 +547,8 @@ struct sched_entity {
> > > > >         /* For load-balancing: */
> > > > >         struct load_weight              load;
> > > > >         struct rb_node                  run_node;
> > > > > +       struct rb_node                  latency_node;
> > > > > +       unsigned int                    on_latency;
> > > > >         struct list_head                group_node;
> > > > >         unsigned int                    on_rq;
> > > > >
> > > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > > > index e524e892d118..1a72f34136d8 100644
> > > > > --- a/kernel/sched/fair.c
> > > > > +++ b/kernel/sched/fair.c
> > > > > @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> > > > >
> > > > >         return __node_2_se(last);
> > > > >  }
> > > > > +#endif
> > > > >
> > > > > +/**************************************************************
> > > > > + * Scheduling class tree data structure manipulation methods:
> > > > > + * for latency
> > > > > + */
> > > > > +
> > > > > +static inline bool latency_before(struct sched_entity *a,
> > > > > +                               struct sched_entity *b)
> > > > > +{
> > > > > +       return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> > > > > +}
> > > > > +
> > > > > +#define __latency_node_2_se(node) \
> > > > > +       rb_entry((node), struct sched_entity, latency_node)
> > > > > +
> > > > > +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> > > > > +{
> > > > > +       return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> > > > > +}
> > > > > +
> > > > > +/*
> > > > > + * Enqueue an entity into the latency rb-tree:
> > > > > + */
> > > > > +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > +{
> > > > > +
> > > > > +       /* Only latency sensitive entity can be added to the list */
> > > > > +       if (se->latency_offset >= 0)
> > > > > +               return;
> > > > > +
> > > > > +       if (se->on_latency)
> > > > > +               return;
> > > > > +
> > > > > +       /*
> > > > > +        * An execution time less than sysctl_sched_min_granularity means that
> > > > > +        * the entity has been preempted by a higher sched class or an entity
> > > > > +        * with higher latency constraint.
> > > > > +        * Put it back in the list so it gets a chance to run 1st during the
> > > > > +        * next slice.
> > > > > +        */
> > > > > +       if (!(flags & ENQUEUE_WAKEUP)) {
> > > > > +               u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> > > > > +
> > > > > +               if (delta_exec >= sysctl_sched_min_granularity)
> > > > > +                       return;
> > > > > +       }
> > > > > +
> > > > > +       rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> > > > > +       se->on_latency = 1;
> > > > > +}
> > > > > +
> > > > > +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > > > +{
> > > > > +       if (se->on_latency) {
> > > > > +               rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> > > > > +               se->on_latency = 0;
> > > > > +       }
> > > > > +}
> > > > > +
> > > > > +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> > > > > +{
> > > > > +       struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> > > > > +
> > > > > +       if (!left)
> > > > > +               return NULL;
> > > > > +
> > > > > +       return __latency_node_2_se(left);
> > > > > +}
> > > > > +
> > > > > +#ifdef CONFIG_SCHED_DEBUG
> > > > >  /**************************************************************
> > > > >   * Scheduling class statistics methods:
> > > > >   */
> > > > > @@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > >         check_schedstat_required();
> > > > >         update_stats_enqueue_fair(cfs_rq, se, flags);
> > > > >         check_spread(cfs_rq, se);
> > > > > -       if (!curr)
> > > > > +       if (!curr) {
> > > > >                 __enqueue_entity(cfs_rq, se);
> > > > > +               __enqueue_latency(cfs_rq, se, flags);
> > > > > +       }
> > > > >         se->on_rq = 1;
> > > > >
> > > > >         if (cfs_rq->nr_running == 1) {
> > > > > @@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > >
> > > > >         clear_buddies(cfs_rq, se);
> > > > >
> > > > > -       if (se != cfs_rq->curr)
> > > > > +       if (se != cfs_rq->curr) {
> > > > >                 __dequeue_entity(cfs_rq, se);
> > > > > +               __dequeue_latency(cfs_rq, se);
> > > > > +       }
> > > > >         se->on_rq = 0;
> > > > >         account_entity_dequeue(cfs_rq, se);
> > > > >
> > > > > @@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > > >                  */
> > > > >                 update_stats_wait_end_fair(cfs_rq, se);
> > > > >                 __dequeue_entity(cfs_rq, se);
> > > > > +               __dequeue_latency(cfs_rq, se);
> > > > >                 update_load_avg(cfs_rq, se, UPDATE_TG);
> > > > >         }
> > > > >
> > > > > @@ -4669,7 +4744,7 @@ static struct sched_entity *
> > > > >  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > > >  {
> > > > >         struct sched_entity *left = __pick_first_entity(cfs_rq);
> > > > > -       struct sched_entity *se;
> > > > > +       struct sched_entity *latency, *se;
> > > > >
> > > > >         /*
> > > > >          * If curr is set we have to see if its left of the leftmost entity
> > > > > @@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > > >                 se = cfs_rq->last;
> > > > >         }
> > > > >
> > > > > +       /* Check for latency sensitive entity waiting for running */
> > > > > +       latency = __pick_first_latency(cfs_rq);
> > > > > +       if (latency && (latency != se) &&
> > > > > +           wakeup_preempt_entity(latency, se) < 1)
> > > > > +               se = latency;
> > > > > +
> > > > >         return se;
> > > > >  }
> > > > >
> > > > > @@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> > > > >                 update_stats_wait_start_fair(cfs_rq, prev);
> > > > >                 /* Put 'current' back into the tree. */
> > > > >                 __enqueue_entity(cfs_rq, prev);
> > > > > +               __enqueue_latency(cfs_rq, prev, 0);
> > > > >                 /* in !on_rq case, update occurred at dequeue */
> > > > >                 update_load_avg(cfs_rq, prev, 0);
> > > > >         }
> > > > > @@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> > > > >  void init_cfs_rq(struct cfs_rq *cfs_rq)
> > > > >  {
> > > > >         cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> > > > > +       cfs_rq->latency_timeline = RB_ROOT_CACHED;
> > > > >         u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> > > > >  #ifdef CONFIG_SMP
> > > > >         raw_spin_lock_init(&cfs_rq->removed.lock);
> > > > > @@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency)
> > > > >
> > > > >         for_each_possible_cpu(i) {
> > > > >                 struct sched_entity *se = tg->se[i];
> > > > > +               struct rq *rq = cpu_rq(i);
> > > > > +               struct rq_flags rf;
> > > > > +
> > > > > +               rq_lock_irqsave(rq, &rf);
> > > > >
> > > > > +               __dequeue_latency(se->cfs_rq, se);
> > > > >                 WRITE_ONCE(se->latency_offset, latency);
> > > > > +
> > > > > +               rq_unlock_irqrestore(rq, &rf);
> > > > >         }
> > > > >
> > > > >         mutex_unlock(&shares_mutex);
> > > > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > > > > index a15fb955092c..76bca172585c 100644
> > > > > --- a/kernel/sched/sched.h
> > > > > +++ b/kernel/sched/sched.h
> > > > > @@ -599,6 +599,7 @@ struct cfs_rq {
> > > > >  #endif
> > > > >
> > > > >         struct rb_root_cached   tasks_timeline;
> > > > > +       struct rb_root_cached   latency_timeline;
> > > > >
> > > > >         /*
> > > > >          * 'curr' points to currently running entity on this cfs_rq.
> > > > > --
> > > > > 2.17.1
> > > > >
> > > > >
> > > > >

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Vincent Guittot]

On Thu, 13 Oct 2022 at 19:19, Youssef Esmat <youssefesmat@chromium.org> wrote:
>
> On Wed, Oct 12, 2022 at 10:21 AM Vincent Guittot
> <vincent.guittot@linaro.org> wrote:
> >
> > Le mardi 11 oct. 2022 à 18:54:27 (-0500), Youssef Esmat a écrit :
> > > Hi Vincent,
> > >
> > > On Tue, Oct 11, 2022 at 12:10 PM Vincent Guittot
> > > <vincent.guittot@linaro.org> wrote:
> >
> > ...
> >
> > > > > >               latency 0    latency -20
> > > > > > Min Latencies:    60           61
> > > > > > Avg Latencies:  1077           86
> > > > > > Max Latencies: 87311          444
> > > > > > 50% latencies:    92           85
> > > > > > 75% latencies:   554           90
> > > > > > 85% latencies:  1019           93
> > > > > > 90% latencies:  1346           96
> > > > > > 95% latencies:  5400          100
> > > > > > 99% latencies: 19044          110
> > > > > >
> > > > > > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > > > > > ---
> > > > >
> > > > > The ability to boost the latency sensitivity of a task seems very
> > > > > interesting. I have been playing around with these changes and have
> > > > > some observations.
> > > > >
> > > > > I tried 2 bursty tasks affinitized to the same CPU. The tasks sleep
> > > > > for 1ms and run for 10ms in a loop. I first tried it without adjusting
> > > > > the latency_nice value and took perf sched traces:
> > > >
> > > > The CPU is overloaded almost all the time as it can't run the 2 tasks
> > > > (2*10ms every 11ms)
> > > >
> > > > >
> > > > > latency_test:7040     |   2447.137 ms |        8 | avg:   6.546 ms |
> > > > > max:  10.674 ms | max start:   353.809487 s | max end:   353.820161 s
> > > > > latency_test:7028     |   2454.777 ms |        7 | avg:   4.494 ms |
> > > > > max:  10.609 ms | max start:   354.804386 s | max end:   354.814995 s
> > > > >
> > > > > Everything looked as expected, for a 5s run they had similar runtime
> > > > > and latency.
> > > > >
> > > > > I then adjusted one task to have a latency_nice of -20 (pid 8614
> > > > > below) and took another set of traces:
> > > > >
> > > > > latency_test:8618     |   1845.534 ms |      131 | avg:   9.764 ms |
> > > > > max:  10.686 ms | max start:  1405.737905 s | max end:  1405.748592 s
> > > > > latency_test:8614     |   3033.635 ms |       16 | avg:   3.559 ms |
> > > > > max:  10.467 ms | max start:  1407.594751 s | max end:  1407.605218 s
> > > > >
> > > > > The task with -20 latency_nice had significantly more runtime. The
> > > > > average latency was improved but the max roughly stayed the same. As
> > > > > expected the one with latency_nice value of 0 experienced more
> > > > > switches, but so did the one with latency_nice of -20.
> > > >
> > > > Your results look unexpected because the vruntime of the tasks is not
> > > > modified. So I can imagine that the thread with the low latency runs
> > > > first up to the offset at the beg of the test but then they should
> > > > switch regularly. I have tried a similar test with a modified rt-app
> > > > and the result seems ok. I have a small difference but not the
> > > > difference that you see.
> > > >
> > > > Could you share more details about your setup ? I'm going to try to
> > > > reproduce your sequence
> > >
> > > I was using an intel core i7 with this frequency details:
> > > CPU MHz:                         4200.000
> > > CPU max MHz:                  4800.0000
> > > CPU min MHz:                   400.0000
> > >
> > > This is a snippet of the test I was using:
> > >
> > > struct sched_attr attr;
> > > memset(&attr, 0, sizeof(struct sched_attr));
> > > attr.size = sizeof(struct sched_attr);
> > > attr.sched_latency_nice = nice_latency;
> > > attr.sched_flags = SCHED_FLAG_LATENCY_NICE;
> > >
> > > // set nice latency value
> > > int res = syscall(__NR_sched_setattr, 0, &attr, 0);
> > >
> > > while(1){
> > >   // wake up every ms
> > >   usleep(1000);
> > >   for(int i = 0; i < 40000000; i++){}
> > > }
> >
> > Between v2 and v3, the sched_feat(GENTLE_FAIR_SLEEPERS) has diseappeared when
> > I moved the computation of latency_offset at the setting of the latency prio
> > instead of runtime. As a result, the latency nice task can preempt up to
> > (sysctl_sched_latency - sysctl_sched_wakeup_granularity) but other threads are
> > cap to half of sysctl_sched_latency at wakeup.
> >
> > Could you try the patch below ?
>
> Thanks Vincent. That improved the runtime for the tests above.
>
> Running two bursty tasks (1ms sleep and 10ms running loop) one of
> which had a latency_nice of -20 (pid 9209) and the other 0:
>
> latency_test:9205     |   2429.467 ms |       14 | avg:   6.293 ms |
> max:  11.340 ms | max start:   209.543482 s | max end:   209.554821 s
> latency_test:9209     |   2467.423 ms |       17 | avg:   3.346 ms |
> max:  11.475 ms | max start:   208.374517 s | max end:   208.385992 s
>
> The task with -20 latency_nice had an improved average latency, but
> the max roughly stayed the same.

This is expected because there is not enough cpu time to run both
tasks in your UC. Task with -20 latency will run 1st whenever it has
not used all its bandwidth but a some point, It will have to let the
other task runs 1st to not get an unfair amount of cpu bandwidth

TL is task with -20 latency
TO is the ther task

TO wakes up and start to run
After 3ms TL wakes  up
TL preempts TO because 3ms delta +12ms latency offset > 4ms
sched_akeup_granularity
TL  runs for 10ms then go to sleep
TL  vruntime is +7ms vs TO
TO starts to run
1ms later TL wakeup
TL preempts TO because delta -6ms delta +12ms latency offset > 4ms
sched_akeup_granularity
TL  runs for 10ms then go to sleep
TL  vruntime is +16ms vs TO
TO starts to run
1ms later TL wakeup
TL can't preempt TO because delta -15ms delta +12ms latency offset <
4ms sched_wakeup_granularity

TL has exhausted its cpu bandwidth and will wait like other cfs task

I will look more deeply on the use cases below

>
> Running the same test with 3 bursty tasks, one of which had a
> latency_nice of -20 and the other two with latency_nice = 0. The
> results for this case roughly stayed the same after the changes:
>
> latency_test:26088    |   1641.458 ms |      158 | avg:  19.613 ms |
> max:  33.000 ms | max start:   871.707231 s | max end:   871.740231 s
> latency_test:26295    |   1639.766 ms |      238 | avg:  10.259 ms |
> max:  24.289 ms | max start:   873.917231 s | max end:   873.941519 s
> latency_test:26401    |   1643.580 ms |      241 | avg:  10.200 ms |
> max:  22.124 ms | max start:   876.289233 s | max end:   876.311357 s
>
> The task with latency_nice -20 seemed to have the highest average and
> max latencies.
>
> I also tried an additional test where we had a task that was cpu bound
> (while(1){}) and a task that ran for 1ms and slept for 1ms. The cpu
> bound task had the latency_nice value of -20 and the bursty had
> latency_nice of 0.
>
> latency_test:17353    |   4557.699 ms |      356 | avg:   1.058 ms |
> max:   1.181 ms | max start:  4324.806123 s | max end:  4324.807304 s
> latency_test:17452    |    377.804 ms |        1 | avg:   0.000 ms |
> max:   0.000 ms | max start:     0.000000 s | max end:     0.000000 s
>
> The cpu bound task (pid 17353 above) had significantly more runtime.
> If I reran the test with latency_nice = 0 for both:
>
> latency_test:20748    |   2478.014 ms |     2367 | avg:   1.037 ms |
> max:   1.182 ms | max start:  4460.769240 s | max end:  4460.770423 s
> latency_test:20972    |   2455.888 ms |        3 | avg:   0.001 ms |
> max:   0.002 ms | max start:  4460.628756 s | max end:  4460.628758 s
>
> Would decreasing the latency_offset as the task runs and increasing it
> as it sleeps help here?
>
>
> >
> > ---
> >  kernel/sched/core.c | 19 ++++++++++++++++---
> >  1 file changed, 16 insertions(+), 3 deletions(-)
> >
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 8b44685ae247..68f9a83d7089 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -1287,11 +1287,15 @@ static void set_load_weight(struct task_struct *p, bool update_load)
> >  static void set_latency_offset(struct task_struct *p)
> >  {
> >         long weight = sched_latency_to_weight[p->latency_prio];
> > +       unsigned long period = sysctl_sched_latency;
> >         s64 offset;
> >
> > -       offset = sysctl_sched_latency * weight;
> > +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > +               period >>= 1;
> > +       offset = period * weight;
> >         offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX);
> >         p->se.latency_offset = (long)offset;
> > +       trace_printk("set_latency_offset pid %d offset %ld", p->pid, offset);
> >  }
> >
> >  #ifdef CONFIG_UCLAMP_TASK
> > @@ -10894,12 +10898,17 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,
> >  static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
> >                                     struct cftype *cft)
> >  {
> > +       unsigned long period = sysctl_sched_latency;
> >         int last_delta = INT_MAX;
> >         int prio, delta;
> >         s64 weight;
> >
> > +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > +               period >>= 1;
> > +
> >         weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX;
> > -       weight = div_s64(weight, sysctl_sched_latency);
> > +       period = sysctl_sched_latency;
> > +       weight = div_s64(weight, period);
> >
> >         /* Find the closest nice value to the current weight */
> >         for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) {
> > @@ -10915,6 +10924,7 @@ static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
> >  static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
> >                                      struct cftype *cft, s64 nice)
> >  {
> > +       unsigned long period;
> >         s64 latency_offset;
> >         long weight;
> >         int idx;
> > @@ -10926,7 +10936,10 @@ static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
> >         idx = array_index_nospec(idx, LATENCY_NICE_WIDTH);
> >         weight = sched_latency_to_weight[idx];
> >
> > -       latency_offset = sysctl_sched_latency * weight;
> > +       period = sysctl_sched_latency;
> > +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > +               period >>= 1;
> > +       latency_offset = period * weight;
> >         latency_offset = div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX);
> >
> >         return sched_group_set_latency(css_tg(css), latency_offset);
> > --
> > 2.17.1
> >
> > >
> > > >
> > > > >
> > > > > Also tried running the same test but instead of using latency nice I
> > > > > adjusted the nice value as a comparison. In that case one task had a
> > > > > nice of -5 and the other was 0.
> > > > >
> > > > > nice_test:25219       |   1216.839 ms |      242 | avg:  10.295 ms |
> > > > > max:  11.927 ms | max start:  5877.881279 s | max end:  5877.893206 s
> > > > > nice_test:25235       |   3711.788 ms |        6 | avg:   1.026 ms |
> > > > > max:   6.143 ms | max start:  5875.603741 s | max end:  5875.609883 s
> > > > >
> > > > > As expected the one with a nice value of -5 had more runtime but also
> > > > > had better latency numbers than in the previous case of using
> > > > > latency_nice.
> > > > >
> > > > > I also tried a similar test with 3 bursty tasks instead of two. In
> > > > > this case all tasks had a latency_nice of 0:
> > > > >
> > > > > latency_test:11467    |   1641.131 ms |      161 | avg:  17.489 ms |
> > > > > max:  21.011 ms | max start:  1542.656275 s | max end:  1542.677286 s
> > > > > latency_test:11463    |   1644.809 ms |      161 | avg:  11.994 ms |
> > > > > max:  25.012 ms | max start:  1545.657776 s | max end:  1545.682788 s
> > > > > latency_test:11478    |   1643.211 ms |      160 | avg:  11.465 ms |
> > > > > max:  21.012 ms | max start:  1546.159026 s | max end:  1546.180038 s
> > > > >
> > > > > Next I tried two tasks with a latency_nice of 0 and a third one had a
> > > > > latency_nice of -20 (pid 11763 below):
> > > > >
> > > > > latency_test:11763    |   1645.482 ms |      159 | avg:  19.634 ms |
> > > > > max:  31.016 ms | max start:  1623.834862 s | max end:  1623.865877 s
> > > > > latency_test:11750    |   1644.276 ms |      259 | avg:   9.985 ms |
> > > > > max:  21.012 ms | max start:  1623.953921 s | max end:  1623.974933 s
> > > > > latency_test:11747    |   1642.745 ms |      262 | avg:   9.079 ms |
> > > > > max:  25.013 ms | max start:  1620.980435 s | max end:  1621.005447 s
> > > > >
> > > > > In this case it seemed like the runtime was not affected by the
> > > > > latency_nice value, but strangely the task with the latency nice of
> > > > > -20 had a worse average and max latency than the other two. The
> > > > > context switch times are also increased from the previous case.
> > > > >
> > > > > Have we considered an approach where the task that is marked as
> > > > > latency sensitive gets a boosted nice value when it sleeps and is
> > > > > either scaled down exponentially as it runs, or immediately reset to
> > > > > its default when it runs for one tick?
> > > > >
> > > > > Thanks,
> > > > > Youssef
> > > > >
> > > > >
> > > > > >  include/linux/sched.h |  2 +
> > > > > >  kernel/sched/fair.c   | 96 +++++++++++++++++++++++++++++++++++++++++--
> > > > > >  kernel/sched/sched.h  |  1 +
> > > > > >  3 files changed, 96 insertions(+), 3 deletions(-)
> > > > > >
> > > > > > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > > > > > index a74cad08e91e..0b92674e3664 100644
> > > > > > --- a/include/linux/sched.h
> > > > > > +++ b/include/linux/sched.h
> > > > > > @@ -547,6 +547,8 @@ struct sched_entity {
> > > > > >         /* For load-balancing: */
> > > > > >         struct load_weight              load;
> > > > > >         struct rb_node                  run_node;
> > > > > > +       struct rb_node                  latency_node;
> > > > > > +       unsigned int                    on_latency;
> > > > > >         struct list_head                group_node;
> > > > > >         unsigned int                    on_rq;
> > > > > >
> > > > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > > > > index e524e892d118..1a72f34136d8 100644
> > > > > > --- a/kernel/sched/fair.c
> > > > > > +++ b/kernel/sched/fair.c
> > > > > > @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> > > > > >
> > > > > >         return __node_2_se(last);
> > > > > >  }
> > > > > > +#endif
> > > > > >
> > > > > > +/**************************************************************
> > > > > > + * Scheduling class tree data structure manipulation methods:
> > > > > > + * for latency
> > > > > > + */
> > > > > > +
> > > > > > +static inline bool latency_before(struct sched_entity *a,
> > > > > > +                               struct sched_entity *b)
> > > > > > +{
> > > > > > +       return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> > > > > > +}
> > > > > > +
> > > > > > +#define __latency_node_2_se(node) \
> > > > > > +       rb_entry((node), struct sched_entity, latency_node)
> > > > > > +
> > > > > > +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> > > > > > +{
> > > > > > +       return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> > > > > > +}
> > > > > > +
> > > > > > +/*
> > > > > > + * Enqueue an entity into the latency rb-tree:
> > > > > > + */
> > > > > > +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > > +{
> > > > > > +
> > > > > > +       /* Only latency sensitive entity can be added to the list */
> > > > > > +       if (se->latency_offset >= 0)
> > > > > > +               return;
> > > > > > +
> > > > > > +       if (se->on_latency)
> > > > > > +               return;
> > > > > > +
> > > > > > +       /*
> > > > > > +        * An execution time less than sysctl_sched_min_granularity means that
> > > > > > +        * the entity has been preempted by a higher sched class or an entity
> > > > > > +        * with higher latency constraint.
> > > > > > +        * Put it back in the list so it gets a chance to run 1st during the
> > > > > > +        * next slice.
> > > > > > +        */
> > > > > > +       if (!(flags & ENQUEUE_WAKEUP)) {
> > > > > > +               u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> > > > > > +
> > > > > > +               if (delta_exec >= sysctl_sched_min_granularity)
> > > > > > +                       return;
> > > > > > +       }
> > > > > > +
> > > > > > +       rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> > > > > > +       se->on_latency = 1;
> > > > > > +}
> > > > > > +
> > > > > > +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > > > > +{
> > > > > > +       if (se->on_latency) {
> > > > > > +               rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> > > > > > +               se->on_latency = 0;
> > > > > > +       }
> > > > > > +}
> > > > > > +
> > > > > > +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> > > > > > +{
> > > > > > +       struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> > > > > > +
> > > > > > +       if (!left)
> > > > > > +               return NULL;
> > > > > > +
> > > > > > +       return __latency_node_2_se(left);
> > > > > > +}
> > > > > > +
> > > > > > +#ifdef CONFIG_SCHED_DEBUG
> > > > > >  /**************************************************************
> > > > > >   * Scheduling class statistics methods:
> > > > > >   */
> > > > > > @@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > >         check_schedstat_required();
> > > > > >         update_stats_enqueue_fair(cfs_rq, se, flags);
> > > > > >         check_spread(cfs_rq, se);
> > > > > > -       if (!curr)
> > > > > > +       if (!curr) {
> > > > > >                 __enqueue_entity(cfs_rq, se);
> > > > > > +               __enqueue_latency(cfs_rq, se, flags);
> > > > > > +       }
> > > > > >         se->on_rq = 1;
> > > > > >
> > > > > >         if (cfs_rq->nr_running == 1) {
> > > > > > @@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > >
> > > > > >         clear_buddies(cfs_rq, se);
> > > > > >
> > > > > > -       if (se != cfs_rq->curr)
> > > > > > +       if (se != cfs_rq->curr) {
> > > > > >                 __dequeue_entity(cfs_rq, se);
> > > > > > +               __dequeue_latency(cfs_rq, se);
> > > > > > +       }
> > > > > >         se->on_rq = 0;
> > > > > >         account_entity_dequeue(cfs_rq, se);
> > > > > >
> > > > > > @@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > > > >                  */
> > > > > >                 update_stats_wait_end_fair(cfs_rq, se);
> > > > > >                 __dequeue_entity(cfs_rq, se);
> > > > > > +               __dequeue_latency(cfs_rq, se);
> > > > > >                 update_load_avg(cfs_rq, se, UPDATE_TG);
> > > > > >         }
> > > > > >
> > > > > > @@ -4669,7 +4744,7 @@ static struct sched_entity *
> > > > > >  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > > > >  {
> > > > > >         struct sched_entity *left = __pick_first_entity(cfs_rq);
> > > > > > -       struct sched_entity *se;
> > > > > > +       struct sched_entity *latency, *se;
> > > > > >
> > > > > >         /*
> > > > > >          * If curr is set we have to see if its left of the leftmost entity
> > > > > > @@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > > > >                 se = cfs_rq->last;
> > > > > >         }
> > > > > >
> > > > > > +       /* Check for latency sensitive entity waiting for running */
> > > > > > +       latency = __pick_first_latency(cfs_rq);
> > > > > > +       if (latency && (latency != se) &&
> > > > > > +           wakeup_preempt_entity(latency, se) < 1)
> > > > > > +               se = latency;
> > > > > > +
> > > > > >         return se;
> > > > > >  }
> > > > > >
> > > > > > @@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> > > > > >                 update_stats_wait_start_fair(cfs_rq, prev);
> > > > > >                 /* Put 'current' back into the tree. */
> > > > > >                 __enqueue_entity(cfs_rq, prev);
> > > > > > +               __enqueue_latency(cfs_rq, prev, 0);
> > > > > >                 /* in !on_rq case, update occurred at dequeue */
> > > > > >                 update_load_avg(cfs_rq, prev, 0);
> > > > > >         }
> > > > > > @@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> > > > > >  void init_cfs_rq(struct cfs_rq *cfs_rq)
> > > > > >  {
> > > > > >         cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> > > > > > +       cfs_rq->latency_timeline = RB_ROOT_CACHED;
> > > > > >         u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> > > > > >  #ifdef CONFIG_SMP
> > > > > >         raw_spin_lock_init(&cfs_rq->removed.lock);
> > > > > > @@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency)
> > > > > >
> > > > > >         for_each_possible_cpu(i) {
> > > > > >                 struct sched_entity *se = tg->se[i];
> > > > > > +               struct rq *rq = cpu_rq(i);
> > > > > > +               struct rq_flags rf;
> > > > > > +
> > > > > > +               rq_lock_irqsave(rq, &rf);
> > > > > >
> > > > > > +               __dequeue_latency(se->cfs_rq, se);
> > > > > >                 WRITE_ONCE(se->latency_offset, latency);
> > > > > > +
> > > > > > +               rq_unlock_irqrestore(rq, &rf);
> > > > > >         }
> > > > > >
> > > > > >         mutex_unlock(&shares_mutex);
> > > > > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > > > > > index a15fb955092c..76bca172585c 100644
> > > > > > --- a/kernel/sched/sched.h
> > > > > > +++ b/kernel/sched/sched.h
> > > > > > @@ -599,6 +599,7 @@ struct cfs_rq {
> > > > > >  #endif
> > > > > >
> > > > > >         struct rb_root_cached   tasks_timeline;
> > > > > > +       struct rb_root_cached   latency_timeline;
> > > > > >
> > > > > >         /*
> > > > > >          * 'curr' points to currently running entity on this cfs_rq.
> > > > > > --
> > > > > > 2.17.1
> > > > > >
> > > > > >
> > > > > >

Re: [PATCH v5 0/7] Add latency priority for CFS class

[K Prateek Nayak]

Hello Vincent,

Thank you for taking a look at the report.

On 10/13/2022 8:54 PM, Vincent Guittot wrote:
> Hi Prateek,
> 
> Thanks for testing the patchset on AMD and the test report below.
> 
> On Wed, 12 Oct 2022 at 16:54, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>>
>> [..snip..]
>>
>> - Socket (Process)
>>
>> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>>  1-groups:         6.44 (0.00 pct)         5.50 (14.59 pct)  ^     6.43 (0.15 pct)
>>  2-groups:         6.55 (0.00 pct)         5.56 (15.11 pct)  ^     6.36 (2.90 pct)
>>  4-groups:         6.74 (0.00 pct)         6.19 (8.16 pct)   ^     6.69 (0.74 pct)
>>  8-groups:         8.03 (0.00 pct)         8.29 (-3.23 pct)        8.02 (0.12 pct)
>> 16-groups:        12.25 (0.00 pct)        14.11 (-15.18 pct)      12.41 (-1.30 pct)
> 
> I don't see any improvement with LN:-20 but only for LN:19
> 
> How many iterations do you run ? Could it be that the results vary
> between iterations ? For some configuration I have a stddev of 10-20%
> for LN:0 and LN:-20
> 

Yes I do see a lot of run to run variation for the above runs:

For 1-group:

LN:           :    0             -20              19
Min           : 6.26            4.97            6.28
Max           : 6.54            6.71            6.55
Median        : 6.45            5.28            6.43
AMean         : 6.44            5.50            6.43
GMean         : 6.44            5.47            6.43
HMean         : 6.44            5.44            6.43
AMean Stddev  : 0.08            0.60            0.08
AMean CoefVar : 1.18 pct       10.89 pct        1.28 pct

For 2-group:

LN:           :    0             -20              19
Min           : 5.80            5.38            5.28
Max           : 6.80            6.70            6.32
Median        : 6.66            6.53            5.48
AMean         : 6.55            6.36            5.56
GMean         : 6.55            6.35            5.55
HMean         : 6.54            6.33            5.54
AMean Stddev  : 0.29            0.41            0.33
AMean CoefVar : 4.38 pct        6.48 pct        5.99 pct

I've rerun this data point and following are the results:

- Socket (Process) (Loop: 100000)

Test:                   LN:0                    LN:-20                  LN:19
 1-groups:         6.81 (0.00 pct)         6.62 (2.79 pct)         6.62 (2.79 pct)
 2-groups:         6.76 (0.00 pct)         6.69 (1.03 pct)         6.65 (1.62 pct)
 4-groups:         6.62 (0.00 pct)         6.65 (-0.45 pct)        6.63 (-0.15 pct)
 8-groups:         7.84 (0.00 pct)         7.81 (0.38 pct)         7.78 (0.76 pct)
16-groups:        12.87 (0.00 pct)        12.40 (3.65 pct)        12.35 (4.04 pct)

Results are more stable in these runs but runs with LN: -20,
have comparatively larger Stddev compared to LN: 0 and LN: 19

>>
>>> Loops: 2160 (Same as in testing)
>>
>> - Pipe (Thread)
>>
>> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>>  1-groups:         0.10 (0.00 pct)         0.12 (-20.00 pct)       0.10 (0.00 pct)
>>  2-groups:         0.12 (0.00 pct)         0.15 (-25.00 pct)       0.11 (8.33 pct)
>>  4-groups:         0.14 (0.00 pct)         0.18 (-28.57 pct)       0.15 (-7.14 pct)
>>  8-groups:         0.17 (0.00 pct)         0.24 (-41.17 pct)       0.17 (0.00 pct)
>> 16-groups:         0.26 (0.00 pct)         0.33 (-26.92 pct)       0.21 (19.23 pct)
>>
>> - Pipe (Process)
>>
>> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>>  1-groups:         0.10 (0.00 pct)         0.12 (-20.00 pct)       0.10 (0.00 pct)
>>  2-groups:         0.12 (0.00 pct)         0.16 (-33.33 pct)       0.12 (0.00 pct)
>>  4-groups:         0.14 (0.00 pct)         0.17 (-21.42 pct)       0.13 (7.14 pct)
>>  8-groups:         0.16 (0.00 pct)         0.24 (-50.00 pct)       0.16 (0.00 pct)
>> 16-groups:         0.23 (0.00 pct)         0.33 (-43.47 pct)       0.19 (17.39 pct)
>>
>> - Socket (Thread)
>>
>> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>>  1-groups:         0.19 (0.00 pct)         0.18 (5.26 pct)         0.18 (5.26 pct)
>>  2-groups:         0.21 (0.00 pct)         0.21 (0.00 pct)         0.20 (4.76 pct)
>>  4-groups:         0.22 (0.00 pct)         0.25 (-13.63 pct)       0.22 (0.00 pct)
>>  8-groups:         0.27 (0.00 pct)         0.36 (-33.33 pct)       0.27 (0.00 pct)
>> 16-groups:         0.42 (0.00 pct)         0.55 (-30.95 pct)       0.40 (4.76 pct)
>>
>> - Socket (Process)
>>
>> Test:              Latency Nice: 0         Latency Nice: -20       Latency Nice: 19
>>  1-groups:         0.17 (0.00 pct)         0.17 (0.00 pct)         0.17 (0.00 pct)
>>  2-groups:         0.19 (0.00 pct)         0.20 (-5.26 pct)        0.19 (0.00 pct)
>>  4-groups:         0.20 (0.00 pct)         0.22 (-10.00 pct)       0.20 (0.00 pct)
>>  8-groups:         0.25 (0.00 pct)         0.32 (-28.00 pct)       0.25 (0.00 pct)
>> 16-groups:         0.40 (0.00 pct)         0.51 (-27.50 pct)       0.39 (2.50 pct)
>>
>> o Hackbench and Cyclictest in NPS1 configuration
>>
>> perf bench sched messaging -p -t -l 100000 -g 16&
>> cyclictest --policy other -D 5 -q -n -H 20000
>>
>> -----------------------------------------------------------------------------------------------------------------
>> |Hackbench     |      Cyclictest LN = 19        |         Cyclictest LN = 0       |      Cyclictest LN = -20    |
>> |LN            |--------------------------------|---------------------------------|-----------------------------|
>> |v             |   Min  |   Avg   |  Max        |     Min  |   Avg   |  Max       |     Min  |   Avg   |  Max   |
>> |--------------|--------|---------|-------------|----------|---------|------------|----------|---------|--------|
>> |0             |  54.00 |  117.00 | 3021.67     |    53.67 |  65.33  | 133.00     |    53.67 |  65.00  | 201.33 |  ^
>> |19            |  50.00 |  100.67 | 3099.33     |    41.00 |  64.33  | 1014.33    |    54.00 |  63.67  | 213.33 |
>> |-20           |  53.00 |  169.00 | 11661.67    |    53.67 |  217.33 | 14313.67   |    46.00 |  61.33  | 236.00 |  ^
>> -----------------------------------------------------------------------------------------------------------------
> 
> The latency results look good with Cyclictest LN:0 and hackbench LN:0.
> 133us max latency. This suggests that your system is not overloaded
> and cyclictest doesn't really compete with others to run.

I'll get data while running hanckbench with larger number
of groups. I'll look out for larger latency in LN: (0, 0)
case to check for CPU contention.

> 
>>
>> o Hackbench and schbench in NPS1 configuration
>>
>> perf bench sched messaging -p -t -l 1000000 -g 16&
>> schebcnh -m 1 -t 64 -s 30s
>>
>> ------------------------------------------------------------------------------------------------------------
>> |Hackbench     |   schbench LN = 19         |        schbench LN = 0         |       schbench LN = -20     |
>> |LN            |----------------------------|--------------------------------|-----------------------------|
>> |v             |  90th  |  95th  |  99th    |   90th  |  95th   |  99th      |   90th  |   95th   | 99th   |
>> |--------------|--------|--------|----------|---------|---------|------------|---------|----------|--------|
>> |0             |  4264  |  6744  |  15664   |   17952 |  32672  |  55488     |   15088 |   25312  | 50112  |
>> |19            |  288   |  613   |  2332    |   274   |  1015   |  3628      |   374   |   1394   | 4424   |
>> |-20           |  35904 |  47680 |  79744   |   87168 |  113536 |  176896    |   13008 |   21216  | 42560  |   ^
>> ------------------------------------------------------------------------------------------------------------
> 
> For the schbench, your test is 30 seconds long which is longer than
> the duration of perf bench sched messaging -p -t -l 1000000 -g 16&

With loop size 1 million, I see the schbench complete before
hackbench in all the cases. I'll rerun this with larger group
size too to get more data and make sure hackbench runs longer
than schbench in all cases.

> 
> The duration of the latter varies depending of latency nice value so
> schbench is disturb more time in some cases
>>
>> o SpecJBB Multi-JVM
>>
>> ---------------------------------------------
>> | Latency Nice  |         0  |           19 |
>> ---------------------------------------------
>> | max-jOPS      |      100%  |      109.92% |
>> | critical-jOPS |      100%  |      153.70% |
>> ---------------------------------------------
>>
>> In most cases, latency nice delivers what it promises.
>> Some cases marked with "^" have shown anomalies or non-linear behavior
>> that is yet to be root caused. If you've seen something similar during
>> your testing, I would love to know what could lead to such a behavior.
> 
> I haven't seen anything like the results that you tagged with ^. As a
> side note, the numbers of groups (g1 g4 g8 g1) that I used with
> hackbench, have been chosen according to my 8 cores system. Your
> system is much larger and hackbench may not overload it with such a
> small number of groups. Maybe you could try with g32 g64 g128 g256 ?
> 

I agree. I'll get the data for cyclictest and schbench with hackbench
running larger number of groups alongside.

> 
>>
>> If you would like more details on the benchmarks results reported above
>> or if there is any specific workload you would like me to test on the
>> Zen3 machine, please do let me know.
>>
>>>
>>> [..snip..]
>>> 
--
Thanks and Regards,
Prateek

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Vincent Guittot]

Le vendredi 14 oct. 2022 à 17:22:35 (+0200), Vincent Guittot a écrit :
> On Thu, 13 Oct 2022 at 19:19, Youssef Esmat <youssefesmat@chromium.org> wrote:
> >
> > On Wed, Oct 12, 2022 at 10:21 AM Vincent Guittot
> > <vincent.guittot@linaro.org> wrote:
> > >
> > > Le mardi 11 oct. 2022 à 18:54:27 (-0500), Youssef Esmat a écrit :
> > > > Hi Vincent,
> > > >
> > > > On Tue, Oct 11, 2022 at 12:10 PM Vincent Guittot
> > > > <vincent.guittot@linaro.org> wrote:
> > >
> > > ...
> > >
> > > > > > >               latency 0    latency -20
> > > > > > > Min Latencies:    60           61
> > > > > > > Avg Latencies:  1077           86
> > > > > > > Max Latencies: 87311          444
> > > > > > > 50% latencies:    92           85
> > > > > > > 75% latencies:   554           90
> > > > > > > 85% latencies:  1019           93
> > > > > > > 90% latencies:  1346           96
> > > > > > > 95% latencies:  5400          100
> > > > > > > 99% latencies: 19044          110
> > > > > > >
> > > > > > > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > > > > > > ---
> > > > > >
> > > > > > The ability to boost the latency sensitivity of a task seems very
> > > > > > interesting. I have been playing around with these changes and have
> > > > > > some observations.
> > > > > >
> > > > > > I tried 2 bursty tasks affinitized to the same CPU. The tasks sleep
> > > > > > for 1ms and run for 10ms in a loop. I first tried it without adjusting
> > > > > > the latency_nice value and took perf sched traces:
> > > > >
> > > > > The CPU is overloaded almost all the time as it can't run the 2 tasks
> > > > > (2*10ms every 11ms)
> > > > >
> > > > > >
> > > > > > latency_test:7040     |   2447.137 ms |        8 | avg:   6.546 ms |
> > > > > > max:  10.674 ms | max start:   353.809487 s | max end:   353.820161 s
> > > > > > latency_test:7028     |   2454.777 ms |        7 | avg:   4.494 ms |
> > > > > > max:  10.609 ms | max start:   354.804386 s | max end:   354.814995 s
> > > > > >
> > > > > > Everything looked as expected, for a 5s run they had similar runtime
> > > > > > and latency.
> > > > > >
> > > > > > I then adjusted one task to have a latency_nice of -20 (pid 8614
> > > > > > below) and took another set of traces:
> > > > > >
> > > > > > latency_test:8618     |   1845.534 ms |      131 | avg:   9.764 ms |
> > > > > > max:  10.686 ms | max start:  1405.737905 s | max end:  1405.748592 s
> > > > > > latency_test:8614     |   3033.635 ms |       16 | avg:   3.559 ms |
> > > > > > max:  10.467 ms | max start:  1407.594751 s | max end:  1407.605218 s
> > > > > >
> > > > > > The task with -20 latency_nice had significantly more runtime. The
> > > > > > average latency was improved but the max roughly stayed the same. As
> > > > > > expected the one with latency_nice value of 0 experienced more
> > > > > > switches, but so did the one with latency_nice of -20.
> > > > >
> > > > > Your results look unexpected because the vruntime of the tasks is not
> > > > > modified. So I can imagine that the thread with the low latency runs
> > > > > first up to the offset at the beg of the test but then they should
> > > > > switch regularly. I have tried a similar test with a modified rt-app
> > > > > and the result seems ok. I have a small difference but not the
> > > > > difference that you see.
> > > > >
> > > > > Could you share more details about your setup ? I'm going to try to
> > > > > reproduce your sequence
> > > >
> > > > I was using an intel core i7 with this frequency details:
> > > > CPU MHz:                         4200.000
> > > > CPU max MHz:                  4800.0000
> > > > CPU min MHz:                   400.0000
> > > >
> > > > This is a snippet of the test I was using:
> > > >
> > > > struct sched_attr attr;
> > > > memset(&attr, 0, sizeof(struct sched_attr));
> > > > attr.size = sizeof(struct sched_attr);
> > > > attr.sched_latency_nice = nice_latency;
> > > > attr.sched_flags = SCHED_FLAG_LATENCY_NICE;
> > > >
> > > > // set nice latency value
> > > > int res = syscall(__NR_sched_setattr, 0, &attr, 0);
> > > >
> > > > while(1){
> > > >   // wake up every ms
> > > >   usleep(1000);
> > > >   for(int i = 0; i < 40000000; i++){}
> > > > }
> > >
> > > Between v2 and v3, the sched_feat(GENTLE_FAIR_SLEEPERS) has diseappeared when
> > > I moved the computation of latency_offset at the setting of the latency prio
> > > instead of runtime. As a result, the latency nice task can preempt up to
> > > (sysctl_sched_latency - sysctl_sched_wakeup_granularity) but other threads are
> > > cap to half of sysctl_sched_latency at wakeup.
> > >
> > > Could you try the patch below ?
> >
> > Thanks Vincent. That improved the runtime for the tests above.
> >
> > Running two bursty tasks (1ms sleep and 10ms running loop) one of
> > which had a latency_nice of -20 (pid 9209) and the other 0:
> >
> > latency_test:9205     |   2429.467 ms |       14 | avg:   6.293 ms |
> > max:  11.340 ms | max start:   209.543482 s | max end:   209.554821 s
> > latency_test:9209     |   2467.423 ms |       17 | avg:   3.346 ms |
> > max:  11.475 ms | max start:   208.374517 s | max end:   208.385992 s
> >
> > The task with -20 latency_nice had an improved average latency, but
> > the max roughly stayed the same.
> 
> This is expected because there is not enough cpu time to run both
> tasks in your UC. Task with -20 latency will run 1st whenever it has
> not used all its bandwidth but a some point, It will have to let the
> other task runs 1st to not get an unfair amount of cpu bandwidth
> 
> TL is task with -20 latency
> TO is the ther task
> 
> TO wakes up and start to run
> After 3ms TL wakes  up
> TL preempts TO because 3ms delta +12ms latency offset > 4ms
> sched_akeup_granularity
> TL  runs for 10ms then go to sleep
> TL  vruntime is +7ms vs TO
> TO starts to run
> 1ms later TL wakeup
> TL preempts TO because delta -6ms delta +12ms latency offset > 4ms
> sched_akeup_granularity
> TL  runs for 10ms then go to sleep
> TL  vruntime is +16ms vs TO
> TO starts to run
> 1ms later TL wakeup
> TL can't preempt TO because delta -15ms delta +12ms latency offset <
> 4ms sched_wakeup_granularity
> 
> TL has exhausted its cpu bandwidth and will wait like other cfs task
> 
> I will look more deeply on the use cases below

I have finally got time to look at the use cases. See my comment below

> 
> >
> > Running the same test with 3 bursty tasks, one of which had a
> > latency_nice of -20 and the other two with latency_nice = 0. The
> > results for this case roughly stayed the same after the changes:
> >
> > latency_test:26088    |   1641.458 ms |      158 | avg:  19.613 ms |
> > max:  33.000 ms | max start:   871.707231 s | max end:   871.740231 s
> > latency_test:26295    |   1639.766 ms |      238 | avg:  10.259 ms |
> > max:  24.289 ms | max start:   873.917231 s | max end:   873.941519 s
> > latency_test:26401    |   1643.580 ms |      241 | avg:  10.200 ms |
> > max:  22.124 ms | max start:   876.289233 s | max end:   876.311357 s
> >
> > The task with latency_nice -20 seemed to have the highest average and
> > max latencies.
> >
> > I also tried an additional test where we had a task that was cpu bound
> > (while(1){}) and a task that ran for 1ms and slept for 1ms. The cpu
> > bound task had the latency_nice value of -20 and the bursty had
> > latency_nice of 0.
> >
> > latency_test:17353    |   4557.699 ms |      356 | avg:   1.058 ms |
> > max:   1.181 ms | max start:  4324.806123 s | max end:  4324.807304 s
> > latency_test:17452    |    377.804 ms |        1 | avg:   0.000 ms |
> > max:   0.000 ms | max start:     0.000000 s | max end:     0.000000 s
> >
> > The cpu bound task (pid 17353 above) had significantly more runtime.
> > If I reran the test with latency_nice = 0 for both:
> >
> > latency_test:20748    |   2478.014 ms |     2367 | avg:   1.037 ms |
> > max:   1.182 ms | max start:  4460.769240 s | max end:  4460.770423 s
> > latency_test:20972    |   2455.888 ms |        3 | avg:   0.001 ms |
> > max:   0.002 ms | max start:  4460.628756 s | max end:  4460.628758 s
> >
> > Would decreasing the latency_offset as the task runs and increasing it
> > as it sleeps help here?

In fact, the problem above is not only linked to latency nice and you can
face similar problem with nice priority. As an example, if the 2 tasks have
a nice priority of 5 (without latency nice prio), the task that runs 1ms then
sleep 1 ms, will get only a 1/8 of the bandwidth whereas it should get 50%.

The problem comes from the fact that a task that wakes up with a
vruntime == min_vruntime - sched_latency and doesn't preempt current, will
lose some bandwidth because its vruntime is already capped to the max oldest
vruntime and the new waiting time will be lost at next wakeup because
vrutime is capped at min_vruntime - sched_latency

The patch below fixes the problem for both:

---
 kernel/sched/fair.c  | 54 +++++++++++++++++++++++---------------------
 kernel/sched/sched.h | 19 ++++++++++++++++
 2 files changed, 47 insertions(+), 26 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d80fb154bb80..2c4a95633192 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4415,33 +4415,17 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
 {
 	u64 vruntime = cfs_rq->min_vruntime;

-	/*
-	 * The 'current' period is already promised to the current tasks,
-	 * however the extra weight of the new task will slow them down a
-	 * little, place the new task so that it fits in the slot that
-	 * stays open at the end.
-	 */
-	if (initial && sched_feat(START_DEBIT))
-		vruntime += sched_vslice(cfs_rq, se);
-
-	/* sleeps up to a single latency don't count. */
-	if (!initial) {
-		unsigned long thresh;
-
-		if (se_is_idle(se))
-			thresh = sysctl_sched_min_granularity;
-		else
-			thresh = sysctl_sched_latency;
-
+	if (!initial)
+		/* sleeps up to a single latency don't count. */
+		vruntime -= get_sched_latency(se_is_idle(se));
+	else if (sched_feat(START_DEBIT))
 		/*
-		 * Halve their sleep time's effect, to allow
-		 * for a gentler effect of sleepers:
+		 * The 'current' period is already promised to the current tasks,
+		 * however the extra weight of the new task will slow them down a
+		 * little, place the new task so that it fits in the slot that
+		 * stays open at the end.
 		 */
-		if (sched_feat(GENTLE_FAIR_SLEEPERS))
-			thresh >>= 1;
-
-		vruntime -= thresh;
-	}
+		vruntime += sched_vslice(cfs_rq, se);

 	/* ensure we never gain time by being placed backwards. */
 	se->vruntime = max_vruntime(se->vruntime, vruntime);
@@ -7318,7 +7302,25 @@ static unsigned long wakeup_gran(struct sched_entity *se)
 static int
 wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
 {
-	s64 gran, vdiff = curr->vruntime - se->vruntime;
+	struct cfs_rq *cfs_rq = cfs_rq_of(curr);
+	s64 min_vdiff, period, gran, vdiff;
+
+	min_vdiff = cfs_rq->min_vruntime - se->vruntime;
+
+	/*
+	 * At wake up, the vuntime of a task is updated to not be more older
+	 * than a sched_latency period behind the min_vruntime. This prevents
+	 * long sleeping task to get unlimited credit at wakeup.
+	 * Such waking up task should then preempt current to use it CPU
+	 * bandwidth.
+	 */
+	period = get_sched_latency(false);
+	period -= sysctl_sched_min_granularity;
+	if (min_vdiff >= period)
+		return 1;
+
+	/* Take into account latency priority */
+	vdiff = curr->vruntime - se->vruntime;

 	/* Take into account latency priority */
 	vdiff -= wakeup_latency_gran(curr, se);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 76bca172585c..d1534a96f476 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2483,6 +2483,25 @@ extern unsigned int sysctl_numa_balancing_scan_period_max;
 extern unsigned int sysctl_numa_balancing_scan_size;
 #endif

+static inline unsigned long  get_sched_latency(bool idle)
+{
+	unsigned long thresh;
+
+	if (idle)
+		thresh = sysctl_sched_min_granularity;
+	else
+		thresh = sysctl_sched_latency;
+
+	/*
+	 * Halve their sleep time's effect, to allow
+	 * for a gentler effect of sleepers:
+	 */
+	if (sched_feat(GENTLE_FAIR_SLEEPERS))
+		thresh >>= 1;
+
+	return thresh;
+}
+
 #ifdef CONFIG_SCHED_HRTICK

 /*
--
2.17.1


> >
> >
> > >
> > > ---
> > >  kernel/sched/core.c | 19 ++++++++++++++++---
> > >  1 file changed, 16 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > > index 8b44685ae247..68f9a83d7089 100644
> > > --- a/kernel/sched/core.c
> > > +++ b/kernel/sched/core.c
> > > @@ -1287,11 +1287,15 @@ static void set_load_weight(struct task_struct *p, bool update_load)
> > >  static void set_latency_offset(struct task_struct *p)
> > >  {
> > >         long weight = sched_latency_to_weight[p->latency_prio];
> > > +       unsigned long period = sysctl_sched_latency;
> > >         s64 offset;
> > >
> > > -       offset = sysctl_sched_latency * weight;
> > > +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > > +               period >>= 1;
> > > +       offset = period * weight;
> > >         offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX);
> > >         p->se.latency_offset = (long)offset;
> > > +       trace_printk("set_latency_offset pid %d offset %ld", p->pid, offset);
> > >  }
> > >
> > >  #ifdef CONFIG_UCLAMP_TASK
> > > @@ -10894,12 +10898,17 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,
> > >  static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
> > >                                     struct cftype *cft)
> > >  {
> > > +       unsigned long period = sysctl_sched_latency;
> > >         int last_delta = INT_MAX;
> > >         int prio, delta;
> > >         s64 weight;
> > >
> > > +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > > +               period >>= 1;
> > > +
> > >         weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX;
> > > -       weight = div_s64(weight, sysctl_sched_latency);
> > > +       period = sysctl_sched_latency;
> > > +       weight = div_s64(weight, period);
> > >
> > >         /* Find the closest nice value to the current weight */
> > >         for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) {
> > > @@ -10915,6 +10924,7 @@ static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
> > >  static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
> > >                                      struct cftype *cft, s64 nice)
> > >  {
> > > +       unsigned long period;
> > >         s64 latency_offset;
> > >         long weight;
> > >         int idx;
> > > @@ -10926,7 +10936,10 @@ static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
> > >         idx = array_index_nospec(idx, LATENCY_NICE_WIDTH);
> > >         weight = sched_latency_to_weight[idx];
> > >
> > > -       latency_offset = sysctl_sched_latency * weight;
> > > +       period = sysctl_sched_latency;
> > > +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > > +               period >>= 1;
> > > +       latency_offset = period * weight;
> > >         latency_offset = div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX);
> > >
> > >         return sched_group_set_latency(css_tg(css), latency_offset);
> > > --
> > > 2.17.1
> > >
> > > >
> > > > >
> > > > > >
> > > > > > Also tried running the same test but instead of using latency nice I
> > > > > > adjusted the nice value as a comparison. In that case one task had a
> > > > > > nice of -5 and the other was 0.
> > > > > >
> > > > > > nice_test:25219       |   1216.839 ms |      242 | avg:  10.295 ms |
> > > > > > max:  11.927 ms | max start:  5877.881279 s | max end:  5877.893206 s
> > > > > > nice_test:25235       |   3711.788 ms |        6 | avg:   1.026 ms |
> > > > > > max:   6.143 ms | max start:  5875.603741 s | max end:  5875.609883 s
> > > > > >
> > > > > > As expected the one with a nice value of -5 had more runtime but also
> > > > > > had better latency numbers than in the previous case of using
> > > > > > latency_nice.
> > > > > >
> > > > > > I also tried a similar test with 3 bursty tasks instead of two. In
> > > > > > this case all tasks had a latency_nice of 0:
> > > > > >
> > > > > > latency_test:11467    |   1641.131 ms |      161 | avg:  17.489 ms |
> > > > > > max:  21.011 ms | max start:  1542.656275 s | max end:  1542.677286 s
> > > > > > latency_test:11463    |   1644.809 ms |      161 | avg:  11.994 ms |
> > > > > > max:  25.012 ms | max start:  1545.657776 s | max end:  1545.682788 s
> > > > > > latency_test:11478    |   1643.211 ms |      160 | avg:  11.465 ms |
> > > > > > max:  21.012 ms | max start:  1546.159026 s | max end:  1546.180038 s
> > > > > >
> > > > > > Next I tried two tasks with a latency_nice of 0 and a third one had a
> > > > > > latency_nice of -20 (pid 11763 below):
> > > > > >
> > > > > > latency_test:11763    |   1645.482 ms |      159 | avg:  19.634 ms |
> > > > > > max:  31.016 ms | max start:  1623.834862 s | max end:  1623.865877 s
> > > > > > latency_test:11750    |   1644.276 ms |      259 | avg:   9.985 ms |
> > > > > > max:  21.012 ms | max start:  1623.953921 s | max end:  1623.974933 s
> > > > > > latency_test:11747    |   1642.745 ms |      262 | avg:   9.079 ms |
> > > > > > max:  25.013 ms | max start:  1620.980435 s | max end:  1621.005447 s
> > > > > >
> > > > > > In this case it seemed like the runtime was not affected by the
> > > > > > latency_nice value, but strangely the task with the latency nice of
> > > > > > -20 had a worse average and max latency than the other two. The
> > > > > > context switch times are also increased from the previous case.
> > > > > >
> > > > > > Have we considered an approach where the task that is marked as
> > > > > > latency sensitive gets a boosted nice value when it sleeps and is
> > > > > > either scaled down exponentially as it runs, or immediately reset to
> > > > > > its default when it runs for one tick?
> > > > > >
> > > > > > Thanks,
> > > > > > Youssef
> > > > > >
> > > > > >
> > > > > > >  include/linux/sched.h |  2 +
> > > > > > >  kernel/sched/fair.c   | 96 +++++++++++++++++++++++++++++++++++++++++--
> > > > > > >  kernel/sched/sched.h  |  1 +
> > > > > > >  3 files changed, 96 insertions(+), 3 deletions(-)
> > > > > > >
> > > > > > > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > > > > > > index a74cad08e91e..0b92674e3664 100644
> > > > > > > --- a/include/linux/sched.h
> > > > > > > +++ b/include/linux/sched.h
> > > > > > > @@ -547,6 +547,8 @@ struct sched_entity {
> > > > > > >         /* For load-balancing: */
> > > > > > >         struct load_weight              load;
> > > > > > >         struct rb_node                  run_node;
> > > > > > > +       struct rb_node                  latency_node;
> > > > > > > +       unsigned int                    on_latency;
> > > > > > >         struct list_head                group_node;
> > > > > > >         unsigned int                    on_rq;
> > > > > > >
> > > > > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > > > > > index e524e892d118..1a72f34136d8 100644
> > > > > > > --- a/kernel/sched/fair.c
> > > > > > > +++ b/kernel/sched/fair.c
> > > > > > > @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> > > > > > >
> > > > > > >         return __node_2_se(last);
> > > > > > >  }
> > > > > > > +#endif
> > > > > > >
> > > > > > > +/**************************************************************
> > > > > > > + * Scheduling class tree data structure manipulation methods:
> > > > > > > + * for latency
> > > > > > > + */
> > > > > > > +
> > > > > > > +static inline bool latency_before(struct sched_entity *a,
> > > > > > > +                               struct sched_entity *b)
> > > > > > > +{
> > > > > > > +       return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +#define __latency_node_2_se(node) \
> > > > > > > +       rb_entry((node), struct sched_entity, latency_node)
> > > > > > > +
> > > > > > > +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> > > > > > > +{
> > > > > > > +       return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> > > > > > > +}
> > > > > > > +
> > > > > > > +/*
> > > > > > > + * Enqueue an entity into the latency rb-tree:
> > > > > > > + */
> > > > > > > +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > > > +{
> > > > > > > +
> > > > > > > +       /* Only latency sensitive entity can be added to the list */
> > > > > > > +       if (se->latency_offset >= 0)
> > > > > > > +               return;
> > > > > > > +
> > > > > > > +       if (se->on_latency)
> > > > > > > +               return;
> > > > > > > +
> > > > > > > +       /*
> > > > > > > +        * An execution time less than sysctl_sched_min_granularity means that
> > > > > > > +        * the entity has been preempted by a higher sched class or an entity
> > > > > > > +        * with higher latency constraint.
> > > > > > > +        * Put it back in the list so it gets a chance to run 1st during the
> > > > > > > +        * next slice.
> > > > > > > +        */
> > > > > > > +       if (!(flags & ENQUEUE_WAKEUP)) {
> > > > > > > +               u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> > > > > > > +
> > > > > > > +               if (delta_exec >= sysctl_sched_min_granularity)
> > > > > > > +                       return;
> > > > > > > +       }
> > > > > > > +
> > > > > > > +       rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> > > > > > > +       se->on_latency = 1;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > > > > > +{
> > > > > > > +       if (se->on_latency) {
> > > > > > > +               rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> > > > > > > +               se->on_latency = 0;
> > > > > > > +       }
> > > > > > > +}
> > > > > > > +
> > > > > > > +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> > > > > > > +{
> > > > > > > +       struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> > > > > > > +
> > > > > > > +       if (!left)
> > > > > > > +               return NULL;
> > > > > > > +
> > > > > > > +       return __latency_node_2_se(left);
> > > > > > > +}
> > > > > > > +
> > > > > > > +#ifdef CONFIG_SCHED_DEBUG
> > > > > > >  /**************************************************************
> > > > > > >   * Scheduling class statistics methods:
> > > > > > >   */
> > > > > > > @@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > > >         check_schedstat_required();
> > > > > > >         update_stats_enqueue_fair(cfs_rq, se, flags);
> > > > > > >         check_spread(cfs_rq, se);
> > > > > > > -       if (!curr)
> > > > > > > +       if (!curr) {
> > > > > > >                 __enqueue_entity(cfs_rq, se);
> > > > > > > +               __enqueue_latency(cfs_rq, se, flags);
> > > > > > > +       }
> > > > > > >         se->on_rq = 1;
> > > > > > >
> > > > > > >         if (cfs_rq->nr_running == 1) {
> > > > > > > @@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > > >
> > > > > > >         clear_buddies(cfs_rq, se);
> > > > > > >
> > > > > > > -       if (se != cfs_rq->curr)
> > > > > > > +       if (se != cfs_rq->curr) {
> > > > > > >                 __dequeue_entity(cfs_rq, se);
> > > > > > > +               __dequeue_latency(cfs_rq, se);
> > > > > > > +       }
> > > > > > >         se->on_rq = 0;
> > > > > > >         account_entity_dequeue(cfs_rq, se);
> > > > > > >
> > > > > > > @@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > > > > >                  */
> > > > > > >                 update_stats_wait_end_fair(cfs_rq, se);
> > > > > > >                 __dequeue_entity(cfs_rq, se);
> > > > > > > +               __dequeue_latency(cfs_rq, se);
> > > > > > >                 update_load_avg(cfs_rq, se, UPDATE_TG);
> > > > > > >         }
> > > > > > >
> > > > > > > @@ -4669,7 +4744,7 @@ static struct sched_entity *
> > > > > > >  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > > > > >  {
> > > > > > >         struct sched_entity *left = __pick_first_entity(cfs_rq);
> > > > > > > -       struct sched_entity *se;
> > > > > > > +       struct sched_entity *latency, *se;
> > > > > > >
> > > > > > >         /*
> > > > > > >          * If curr is set we have to see if its left of the leftmost entity
> > > > > > > @@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > > > > >                 se = cfs_rq->last;
> > > > > > >         }
> > > > > > >
> > > > > > > +       /* Check for latency sensitive entity waiting for running */
> > > > > > > +       latency = __pick_first_latency(cfs_rq);
> > > > > > > +       if (latency && (latency != se) &&
> > > > > > > +           wakeup_preempt_entity(latency, se) < 1)
> > > > > > > +               se = latency;
> > > > > > > +
> > > > > > >         return se;
> > > > > > >  }
> > > > > > >
> > > > > > > @@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> > > > > > >                 update_stats_wait_start_fair(cfs_rq, prev);
> > > > > > >                 /* Put 'current' back into the tree. */
> > > > > > >                 __enqueue_entity(cfs_rq, prev);
> > > > > > > +               __enqueue_latency(cfs_rq, prev, 0);
> > > > > > >                 /* in !on_rq case, update occurred at dequeue */
> > > > > > >                 update_load_avg(cfs_rq, prev, 0);
> > > > > > >         }
> > > > > > > @@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> > > > > > >  void init_cfs_rq(struct cfs_rq *cfs_rq)
> > > > > > >  {
> > > > > > >         cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> > > > > > > +       cfs_rq->latency_timeline = RB_ROOT_CACHED;
> > > > > > >         u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> > > > > > >  #ifdef CONFIG_SMP
> > > > > > >         raw_spin_lock_init(&cfs_rq->removed.lock);
> > > > > > > @@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency)
> > > > > > >
> > > > > > >         for_each_possible_cpu(i) {
> > > > > > >                 struct sched_entity *se = tg->se[i];
> > > > > > > +               struct rq *rq = cpu_rq(i);
> > > > > > > +               struct rq_flags rf;
> > > > > > > +
> > > > > > > +               rq_lock_irqsave(rq, &rf);
> > > > > > >
> > > > > > > +               __dequeue_latency(se->cfs_rq, se);
> > > > > > >                 WRITE_ONCE(se->latency_offset, latency);
> > > > > > > +
> > > > > > > +               rq_unlock_irqrestore(rq, &rf);
> > > > > > >         }
> > > > > > >
> > > > > > >         mutex_unlock(&shares_mutex);
> > > > > > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > > > > > > index a15fb955092c..76bca172585c 100644
> > > > > > > --- a/kernel/sched/sched.h
> > > > > > > +++ b/kernel/sched/sched.h
> > > > > > > @@ -599,6 +599,7 @@ struct cfs_rq {
> > > > > > >  #endif
> > > > > > >
> > > > > > >         struct rb_root_cached   tasks_timeline;
> > > > > > > +       struct rb_root_cached   latency_timeline;
> > > > > > >
> > > > > > >         /*
> > > > > > >          * 'curr' points to currently running entity on this cfs_rq.
> > > > > > > --
> > > > > > > 2.17.1
> > > > > > >
> > > > > > >
> > > > > > >

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Vincent Guittot]

On Wed, 19 Oct 2022 at 18:53, Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> Le vendredi 14 oct. 2022 à 17:22:35 (+0200), Vincent Guittot a écrit :
> > On Thu, 13 Oct 2022 at 19:19, Youssef Esmat <youssefesmat@chromium.org> wrote:
> > >
> > > On Wed, Oct 12, 2022 at 10:21 AM Vincent Guittot
> > > <vincent.guittot@linaro.org> wrote:
> > > >
> > > > Le mardi 11 oct. 2022 à 18:54:27 (-0500), Youssef Esmat a écrit :
> > > > > Hi Vincent,
> > > > >
> > > > > On Tue, Oct 11, 2022 at 12:10 PM Vincent Guittot
> > > > > <vincent.guittot@linaro.org> wrote:
> > > >
> > > > ...
> > > >
> > > > > > > >               latency 0    latency -20
> > > > > > > > Min Latencies:    60           61
> > > > > > > > Avg Latencies:  1077           86
> > > > > > > > Max Latencies: 87311          444
> > > > > > > > 50% latencies:    92           85
> > > > > > > > 75% latencies:   554           90
> > > > > > > > 85% latencies:  1019           93
> > > > > > > > 90% latencies:  1346           96
> > > > > > > > 95% latencies:  5400          100
> > > > > > > > 99% latencies: 19044          110
> > > > > > > >
> > > > > > > > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > > > > > > > ---
> > > > > > >
> > > > > > > The ability to boost the latency sensitivity of a task seems very
> > > > > > > interesting. I have been playing around with these changes and have
> > > > > > > some observations.
> > > > > > >
> > > > > > > I tried 2 bursty tasks affinitized to the same CPU. The tasks sleep
> > > > > > > for 1ms and run for 10ms in a loop. I first tried it without adjusting
> > > > > > > the latency_nice value and took perf sched traces:
> > > > > >
> > > > > > The CPU is overloaded almost all the time as it can't run the 2 tasks
> > > > > > (2*10ms every 11ms)
> > > > > >
> > > > > > >
> > > > > > > latency_test:7040     |   2447.137 ms |        8 | avg:   6.546 ms |
> > > > > > > max:  10.674 ms | max start:   353.809487 s | max end:   353.820161 s
> > > > > > > latency_test:7028     |   2454.777 ms |        7 | avg:   4.494 ms |
> > > > > > > max:  10.609 ms | max start:   354.804386 s | max end:   354.814995 s
> > > > > > >
> > > > > > > Everything looked as expected, for a 5s run they had similar runtime
> > > > > > > and latency.
> > > > > > >
> > > > > > > I then adjusted one task to have a latency_nice of -20 (pid 8614
> > > > > > > below) and took another set of traces:
> > > > > > >
> > > > > > > latency_test:8618     |   1845.534 ms |      131 | avg:   9.764 ms |
> > > > > > > max:  10.686 ms | max start:  1405.737905 s | max end:  1405.748592 s
> > > > > > > latency_test:8614     |   3033.635 ms |       16 | avg:   3.559 ms |
> > > > > > > max:  10.467 ms | max start:  1407.594751 s | max end:  1407.605218 s
> > > > > > >
> > > > > > > The task with -20 latency_nice had significantly more runtime. The
> > > > > > > average latency was improved but the max roughly stayed the same. As
> > > > > > > expected the one with latency_nice value of 0 experienced more
> > > > > > > switches, but so did the one with latency_nice of -20.
> > > > > >
> > > > > > Your results look unexpected because the vruntime of the tasks is not
> > > > > > modified. So I can imagine that the thread with the low latency runs
> > > > > > first up to the offset at the beg of the test but then they should
> > > > > > switch regularly. I have tried a similar test with a modified rt-app
> > > > > > and the result seems ok. I have a small difference but not the
> > > > > > difference that you see.
> > > > > >
> > > > > > Could you share more details about your setup ? I'm going to try to
> > > > > > reproduce your sequence
> > > > >
> > > > > I was using an intel core i7 with this frequency details:
> > > > > CPU MHz:                         4200.000
> > > > > CPU max MHz:                  4800.0000
> > > > > CPU min MHz:                   400.0000
> > > > >
> > > > > This is a snippet of the test I was using:
> > > > >
> > > > > struct sched_attr attr;
> > > > > memset(&attr, 0, sizeof(struct sched_attr));
> > > > > attr.size = sizeof(struct sched_attr);
> > > > > attr.sched_latency_nice = nice_latency;
> > > > > attr.sched_flags = SCHED_FLAG_LATENCY_NICE;
> > > > >
> > > > > // set nice latency value
> > > > > int res = syscall(__NR_sched_setattr, 0, &attr, 0);
> > > > >
> > > > > while(1){
> > > > >   // wake up every ms
> > > > >   usleep(1000);
> > > > >   for(int i = 0; i < 40000000; i++){}
> > > > > }
> > > >
> > > > Between v2 and v3, the sched_feat(GENTLE_FAIR_SLEEPERS) has diseappeared when
> > > > I moved the computation of latency_offset at the setting of the latency prio
> > > > instead of runtime. As a result, the latency nice task can preempt up to
> > > > (sysctl_sched_latency - sysctl_sched_wakeup_granularity) but other threads are
> > > > cap to half of sysctl_sched_latency at wakeup.
> > > >
> > > > Could you try the patch below ?
> > >
> > > Thanks Vincent. That improved the runtime for the tests above.
> > >
> > > Running two bursty tasks (1ms sleep and 10ms running loop) one of
> > > which had a latency_nice of -20 (pid 9209) and the other 0:
> > >
> > > latency_test:9205     |   2429.467 ms |       14 | avg:   6.293 ms |
> > > max:  11.340 ms | max start:   209.543482 s | max end:   209.554821 s
> > > latency_test:9209     |   2467.423 ms |       17 | avg:   3.346 ms |
> > > max:  11.475 ms | max start:   208.374517 s | max end:   208.385992 s
> > >
> > > The task with -20 latency_nice had an improved average latency, but
> > > the max roughly stayed the same.
> >
> > This is expected because there is not enough cpu time to run both
> > tasks in your UC. Task with -20 latency will run 1st whenever it has
> > not used all its bandwidth but a some point, It will have to let the
> > other task runs 1st to not get an unfair amount of cpu bandwidth
> >
> > TL is task with -20 latency
> > TO is the ther task
> >
> > TO wakes up and start to run
> > After 3ms TL wakes  up
> > TL preempts TO because 3ms delta +12ms latency offset > 4ms
> > sched_akeup_granularity
> > TL  runs for 10ms then go to sleep
> > TL  vruntime is +7ms vs TO
> > TO starts to run
> > 1ms later TL wakeup
> > TL preempts TO because delta -6ms delta +12ms latency offset > 4ms
> > sched_akeup_granularity
> > TL  runs for 10ms then go to sleep
> > TL  vruntime is +16ms vs TO
> > TO starts to run
> > 1ms later TL wakeup
> > TL can't preempt TO because delta -15ms delta +12ms latency offset <
> > 4ms sched_wakeup_granularity
> >
> > TL has exhausted its cpu bandwidth and will wait like other cfs task
> >
> > I will look more deeply on the use cases below
>
> I have finally got time to look at the use cases. See my comment below
>
> >
> > >
> > > Running the same test with 3 bursty tasks, one of which had a
> > > latency_nice of -20 and the other two with latency_nice = 0. The
> > > results for this case roughly stayed the same after the changes:
> > >
> > > latency_test:26088    |   1641.458 ms |      158 | avg:  19.613 ms |
> > > max:  33.000 ms | max start:   871.707231 s | max end:   871.740231 s
> > > latency_test:26295    |   1639.766 ms |      238 | avg:  10.259 ms |
> > > max:  24.289 ms | max start:   873.917231 s | max end:   873.941519 s
> > > latency_test:26401    |   1643.580 ms |      241 | avg:  10.200 ms |
> > > max:  22.124 ms | max start:   876.289233 s | max end:   876.311357 s
> > >
> > > The task with latency_nice -20 seemed to have the highest average and
> > > max latencies.
> > >
> > > I also tried an additional test where we had a task that was cpu bound
> > > (while(1){}) and a task that ran for 1ms and slept for 1ms. The cpu
> > > bound task had the latency_nice value of -20 and the bursty had
> > > latency_nice of 0.
> > >
> > > latency_test:17353    |   4557.699 ms |      356 | avg:   1.058 ms |
> > > max:   1.181 ms | max start:  4324.806123 s | max end:  4324.807304 s
> > > latency_test:17452    |    377.804 ms |        1 | avg:   0.000 ms |
> > > max:   0.000 ms | max start:     0.000000 s | max end:     0.000000 s
> > >
> > > The cpu bound task (pid 17353 above) had significantly more runtime.
> > > If I reran the test with latency_nice = 0 for both:
> > >
> > > latency_test:20748    |   2478.014 ms |     2367 | avg:   1.037 ms |
> > > max:   1.182 ms | max start:  4460.769240 s | max end:  4460.770423 s
> > > latency_test:20972    |   2455.888 ms |        3 | avg:   0.001 ms |
> > > max:   0.002 ms | max start:  4460.628756 s | max end:  4460.628758 s
> > >
> > > Would decreasing the latency_offset as the task runs and increasing it
> > > as it sleeps help here?
>
> In fact, the problem above is not only linked to latency nice and you can
> face similar problem with nice priority. As an example, if the 2 tasks have
> a nice priority of 5 (without latency nice prio), the task that runs 1ms then
> sleep 1 ms, will get only a 1/8 of the bandwidth whereas it should get 50%.
>
> The problem comes from the fact that a task that wakes up with a
> vruntime == min_vruntime - sched_latency and doesn't preempt current, will
> lose some bandwidth because its vruntime is already capped to the max oldest
> vruntime and the new waiting time will be lost at next wakeup because
> vrutime is capped at min_vruntime - sched_latency
>
> The patch below fixes the problem for both:
>
> ---
>  kernel/sched/fair.c  | 54 +++++++++++++++++++++++---------------------
>  kernel/sched/sched.h | 19 ++++++++++++++++
>  2 files changed, 47 insertions(+), 26 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index d80fb154bb80..2c4a95633192 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4415,33 +4415,17 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
>  {
>         u64 vruntime = cfs_rq->min_vruntime;
>
> -       /*
> -        * The 'current' period is already promised to the current tasks,
> -        * however the extra weight of the new task will slow them down a
> -        * little, place the new task so that it fits in the slot that
> -        * stays open at the end.
> -        */
> -       if (initial && sched_feat(START_DEBIT))
> -               vruntime += sched_vslice(cfs_rq, se);
> -
> -       /* sleeps up to a single latency don't count. */
> -       if (!initial) {
> -               unsigned long thresh;
> -
> -               if (se_is_idle(se))
> -                       thresh = sysctl_sched_min_granularity;
> -               else
> -                       thresh = sysctl_sched_latency;
> -
> +       if (!initial)
> +               /* sleeps up to a single latency don't count. */
> +               vruntime -= get_sched_latency(se_is_idle(se));
> +       else if (sched_feat(START_DEBIT))
>                 /*
> -                * Halve their sleep time's effect, to allow
> -                * for a gentler effect of sleepers:
> +                * The 'current' period is already promised to the current tasks,
> +                * however the extra weight of the new task will slow them down a
> +                * little, place the new task so that it fits in the slot that
> +                * stays open at the end.
>                  */
> -               if (sched_feat(GENTLE_FAIR_SLEEPERS))
> -                       thresh >>= 1;
> -
> -               vruntime -= thresh;
> -       }
> +               vruntime += sched_vslice(cfs_rq, se);
>
>         /* ensure we never gain time by being placed backwards. */
>         se->vruntime = max_vruntime(se->vruntime, vruntime);
> @@ -7318,7 +7302,25 @@ static unsigned long wakeup_gran(struct sched_entity *se)
>  static int
>  wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>  {
> -       s64 gran, vdiff = curr->vruntime - se->vruntime;
> +       struct cfs_rq *cfs_rq = cfs_rq_of(curr);
> +       s64 min_vdiff, period, gran, vdiff;
> +
> +       min_vdiff = cfs_rq->min_vruntime - se->vruntime;
> +
> +       /*
> +        * At wake up, the vuntime of a task is updated to not be more older
> +        * than a sched_latency period behind the min_vruntime. This prevents
> +        * long sleeping task to get unlimited credit at wakeup.
> +        * Such waking up task should then preempt current to use it CPU
> +        * bandwidth.
> +        */
> +       period = get_sched_latency(false);
> +       period -= sysctl_sched_min_granularity;
> +       if (min_vdiff >= period)
> +               return 1;
> +
> +       /* Take into account latency priority */
> +       vdiff = curr->vruntime - se->vruntime;

The test above is misplaced and should happen after the test  below:

        if (vdiff <= 0)
                return -1;


I'm preparing a new version of the patchset

>
>         /* Take into account latency priority */
>         vdiff -= wakeup_latency_gran(curr, se);
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 76bca172585c..d1534a96f476 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -2483,6 +2483,25 @@ extern unsigned int sysctl_numa_balancing_scan_period_max;
>  extern unsigned int sysctl_numa_balancing_scan_size;
>  #endif
>
> +static inline unsigned long  get_sched_latency(bool idle)
> +{
> +       unsigned long thresh;
> +
> +       if (idle)
> +               thresh = sysctl_sched_min_granularity;
> +       else
> +               thresh = sysctl_sched_latency;
> +
> +       /*
> +        * Halve their sleep time's effect, to allow
> +        * for a gentler effect of sleepers:
> +        */
> +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> +               thresh >>= 1;
> +
> +       return thresh;
> +}
> +
>  #ifdef CONFIG_SCHED_HRTICK
>
>  /*
> --
> 2.17.1
>
>
> > >
> > >
> > > >
> > > > ---
> > > >  kernel/sched/core.c | 19 ++++++++++++++++---
> > > >  1 file changed, 16 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > > > index 8b44685ae247..68f9a83d7089 100644
> > > > --- a/kernel/sched/core.c
> > > > +++ b/kernel/sched/core.c
> > > > @@ -1287,11 +1287,15 @@ static void set_load_weight(struct task_struct *p, bool update_load)
> > > >  static void set_latency_offset(struct task_struct *p)
> > > >  {
> > > >         long weight = sched_latency_to_weight[p->latency_prio];
> > > > +       unsigned long period = sysctl_sched_latency;
> > > >         s64 offset;
> > > >
> > > > -       offset = sysctl_sched_latency * weight;
> > > > +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > > > +               period >>= 1;
> > > > +       offset = period * weight;
> > > >         offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX);
> > > >         p->se.latency_offset = (long)offset;
> > > > +       trace_printk("set_latency_offset pid %d offset %ld", p->pid, offset);
> > > >  }
> > > >
> > > >  #ifdef CONFIG_UCLAMP_TASK
> > > > @@ -10894,12 +10898,17 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,
> > > >  static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
> > > >                                     struct cftype *cft)
> > > >  {
> > > > +       unsigned long period = sysctl_sched_latency;
> > > >         int last_delta = INT_MAX;
> > > >         int prio, delta;
> > > >         s64 weight;
> > > >
> > > > +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > > > +               period >>= 1;
> > > > +
> > > >         weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX;
> > > > -       weight = div_s64(weight, sysctl_sched_latency);
> > > > +       period = sysctl_sched_latency;
> > > > +       weight = div_s64(weight, period);
> > > >
> > > >         /* Find the closest nice value to the current weight */
> > > >         for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) {
> > > > @@ -10915,6 +10924,7 @@ static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
> > > >  static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
> > > >                                      struct cftype *cft, s64 nice)
> > > >  {
> > > > +       unsigned long period;
> > > >         s64 latency_offset;
> > > >         long weight;
> > > >         int idx;
> > > > @@ -10926,7 +10936,10 @@ static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
> > > >         idx = array_index_nospec(idx, LATENCY_NICE_WIDTH);
> > > >         weight = sched_latency_to_weight[idx];
> > > >
> > > > -       latency_offset = sysctl_sched_latency * weight;
> > > > +       period = sysctl_sched_latency;
> > > > +       if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > > > +               period >>= 1;
> > > > +       latency_offset = period * weight;
> > > >         latency_offset = div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX);
> > > >
> > > >         return sched_group_set_latency(css_tg(css), latency_offset);
> > > > --
> > > > 2.17.1
> > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > Also tried running the same test but instead of using latency nice I
> > > > > > > adjusted the nice value as a comparison. In that case one task had a
> > > > > > > nice of -5 and the other was 0.
> > > > > > >
> > > > > > > nice_test:25219       |   1216.839 ms |      242 | avg:  10.295 ms |
> > > > > > > max:  11.927 ms | max start:  5877.881279 s | max end:  5877.893206 s
> > > > > > > nice_test:25235       |   3711.788 ms |        6 | avg:   1.026 ms |
> > > > > > > max:   6.143 ms | max start:  5875.603741 s | max end:  5875.609883 s
> > > > > > >
> > > > > > > As expected the one with a nice value of -5 had more runtime but also
> > > > > > > had better latency numbers than in the previous case of using
> > > > > > > latency_nice.
> > > > > > >
> > > > > > > I also tried a similar test with 3 bursty tasks instead of two. In
> > > > > > > this case all tasks had a latency_nice of 0:
> > > > > > >
> > > > > > > latency_test:11467    |   1641.131 ms |      161 | avg:  17.489 ms |
> > > > > > > max:  21.011 ms | max start:  1542.656275 s | max end:  1542.677286 s
> > > > > > > latency_test:11463    |   1644.809 ms |      161 | avg:  11.994 ms |
> > > > > > > max:  25.012 ms | max start:  1545.657776 s | max end:  1545.682788 s
> > > > > > > latency_test:11478    |   1643.211 ms |      160 | avg:  11.465 ms |
> > > > > > > max:  21.012 ms | max start:  1546.159026 s | max end:  1546.180038 s
> > > > > > >
> > > > > > > Next I tried two tasks with a latency_nice of 0 and a third one had a
> > > > > > > latency_nice of -20 (pid 11763 below):
> > > > > > >
> > > > > > > latency_test:11763    |   1645.482 ms |      159 | avg:  19.634 ms |
> > > > > > > max:  31.016 ms | max start:  1623.834862 s | max end:  1623.865877 s
> > > > > > > latency_test:11750    |   1644.276 ms |      259 | avg:   9.985 ms |
> > > > > > > max:  21.012 ms | max start:  1623.953921 s | max end:  1623.974933 s
> > > > > > > latency_test:11747    |   1642.745 ms |      262 | avg:   9.079 ms |
> > > > > > > max:  25.013 ms | max start:  1620.980435 s | max end:  1621.005447 s
> > > > > > >
> > > > > > > In this case it seemed like the runtime was not affected by the
> > > > > > > latency_nice value, but strangely the task with the latency nice of
> > > > > > > -20 had a worse average and max latency than the other two. The
> > > > > > > context switch times are also increased from the previous case.
> > > > > > >
> > > > > > > Have we considered an approach where the task that is marked as
> > > > > > > latency sensitive gets a boosted nice value when it sleeps and is
> > > > > > > either scaled down exponentially as it runs, or immediately reset to
> > > > > > > its default when it runs for one tick?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Youssef
> > > > > > >
> > > > > > >
> > > > > > > >  include/linux/sched.h |  2 +
> > > > > > > >  kernel/sched/fair.c   | 96 +++++++++++++++++++++++++++++++++++++++++--
> > > > > > > >  kernel/sched/sched.h  |  1 +
> > > > > > > >  3 files changed, 96 insertions(+), 3 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > > > > > > > index a74cad08e91e..0b92674e3664 100644
> > > > > > > > --- a/include/linux/sched.h
> > > > > > > > +++ b/include/linux/sched.h
> > > > > > > > @@ -547,6 +547,8 @@ struct sched_entity {
> > > > > > > >         /* For load-balancing: */
> > > > > > > >         struct load_weight              load;
> > > > > > > >         struct rb_node                  run_node;
> > > > > > > > +       struct rb_node                  latency_node;
> > > > > > > > +       unsigned int                    on_latency;
> > > > > > > >         struct list_head                group_node;
> > > > > > > >         unsigned int                    on_rq;
> > > > > > > >
> > > > > > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > > > > > > index e524e892d118..1a72f34136d8 100644
> > > > > > > > --- a/kernel/sched/fair.c
> > > > > > > > +++ b/kernel/sched/fair.c
> > > > > > > > @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> > > > > > > >
> > > > > > > >         return __node_2_se(last);
> > > > > > > >  }
> > > > > > > > +#endif
> > > > > > > >
> > > > > > > > +/**************************************************************
> > > > > > > > + * Scheduling class tree data structure manipulation methods:
> > > > > > > > + * for latency
> > > > > > > > + */
> > > > > > > > +
> > > > > > > > +static inline bool latency_before(struct sched_entity *a,
> > > > > > > > +                               struct sched_entity *b)
> > > > > > > > +{
> > > > > > > > +       return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +#define __latency_node_2_se(node) \
> > > > > > > > +       rb_entry((node), struct sched_entity, latency_node)
> > > > > > > > +
> > > > > > > > +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> > > > > > > > +{
> > > > > > > > +       return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +/*
> > > > > > > > + * Enqueue an entity into the latency rb-tree:
> > > > > > > > + */
> > > > > > > > +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > > > > +{
> > > > > > > > +
> > > > > > > > +       /* Only latency sensitive entity can be added to the list */
> > > > > > > > +       if (se->latency_offset >= 0)
> > > > > > > > +               return;
> > > > > > > > +
> > > > > > > > +       if (se->on_latency)
> > > > > > > > +               return;
> > > > > > > > +
> > > > > > > > +       /*
> > > > > > > > +        * An execution time less than sysctl_sched_min_granularity means that
> > > > > > > > +        * the entity has been preempted by a higher sched class or an entity
> > > > > > > > +        * with higher latency constraint.
> > > > > > > > +        * Put it back in the list so it gets a chance to run 1st during the
> > > > > > > > +        * next slice.
> > > > > > > > +        */
> > > > > > > > +       if (!(flags & ENQUEUE_WAKEUP)) {
> > > > > > > > +               u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> > > > > > > > +
> > > > > > > > +               if (delta_exec >= sysctl_sched_min_granularity)
> > > > > > > > +                       return;
> > > > > > > > +       }
> > > > > > > > +
> > > > > > > > +       rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> > > > > > > > +       se->on_latency = 1;
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > > > > > > +{
> > > > > > > > +       if (se->on_latency) {
> > > > > > > > +               rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> > > > > > > > +               se->on_latency = 0;
> > > > > > > > +       }
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> > > > > > > > +{
> > > > > > > > +       struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> > > > > > > > +
> > > > > > > > +       if (!left)
> > > > > > > > +               return NULL;
> > > > > > > > +
> > > > > > > > +       return __latency_node_2_se(left);
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +#ifdef CONFIG_SCHED_DEBUG
> > > > > > > >  /**************************************************************
> > > > > > > >   * Scheduling class statistics methods:
> > > > > > > >   */
> > > > > > > > @@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > > > >         check_schedstat_required();
> > > > > > > >         update_stats_enqueue_fair(cfs_rq, se, flags);
> > > > > > > >         check_spread(cfs_rq, se);
> > > > > > > > -       if (!curr)
> > > > > > > > +       if (!curr) {
> > > > > > > >                 __enqueue_entity(cfs_rq, se);
> > > > > > > > +               __enqueue_latency(cfs_rq, se, flags);
> > > > > > > > +       }
> > > > > > > >         se->on_rq = 1;
> > > > > > > >
> > > > > > > >         if (cfs_rq->nr_running == 1) {
> > > > > > > > @@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > > > > > > >
> > > > > > > >         clear_buddies(cfs_rq, se);
> > > > > > > >
> > > > > > > > -       if (se != cfs_rq->curr)
> > > > > > > > +       if (se != cfs_rq->curr) {
> > > > > > > >                 __dequeue_entity(cfs_rq, se);
> > > > > > > > +               __dequeue_latency(cfs_rq, se);
> > > > > > > > +       }
> > > > > > > >         se->on_rq = 0;
> > > > > > > >         account_entity_dequeue(cfs_rq, se);
> > > > > > > >
> > > > > > > > @@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > > > > > > >                  */
> > > > > > > >                 update_stats_wait_end_fair(cfs_rq, se);
> > > > > > > >                 __dequeue_entity(cfs_rq, se);
> > > > > > > > +               __dequeue_latency(cfs_rq, se);
> > > > > > > >                 update_load_avg(cfs_rq, se, UPDATE_TG);
> > > > > > > >         }
> > > > > > > >
> > > > > > > > @@ -4669,7 +4744,7 @@ static struct sched_entity *
> > > > > > > >  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > > > > > >  {
> > > > > > > >         struct sched_entity *left = __pick_first_entity(cfs_rq);
> > > > > > > > -       struct sched_entity *se;
> > > > > > > > +       struct sched_entity *latency, *se;
> > > > > > > >
> > > > > > > >         /*
> > > > > > > >          * If curr is set we have to see if its left of the leftmost entity
> > > > > > > > @@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> > > > > > > >                 se = cfs_rq->last;
> > > > > > > >         }
> > > > > > > >
> > > > > > > > +       /* Check for latency sensitive entity waiting for running */
> > > > > > > > +       latency = __pick_first_latency(cfs_rq);
> > > > > > > > +       if (latency && (latency != se) &&
> > > > > > > > +           wakeup_preempt_entity(latency, se) < 1)
> > > > > > > > +               se = latency;
> > > > > > > > +
> > > > > > > >         return se;
> > > > > > > >  }
> > > > > > > >
> > > > > > > > @@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> > > > > > > >                 update_stats_wait_start_fair(cfs_rq, prev);
> > > > > > > >                 /* Put 'current' back into the tree. */
> > > > > > > >                 __enqueue_entity(cfs_rq, prev);
> > > > > > > > +               __enqueue_latency(cfs_rq, prev, 0);
> > > > > > > >                 /* in !on_rq case, update occurred at dequeue */
> > > > > > > >                 update_load_avg(cfs_rq, prev, 0);
> > > > > > > >         }
> > > > > > > > @@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> > > > > > > >  void init_cfs_rq(struct cfs_rq *cfs_rq)
> > > > > > > >  {
> > > > > > > >         cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> > > > > > > > +       cfs_rq->latency_timeline = RB_ROOT_CACHED;
> > > > > > > >         u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> > > > > > > >  #ifdef CONFIG_SMP
> > > > > > > >         raw_spin_lock_init(&cfs_rq->removed.lock);
> > > > > > > > @@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency)
> > > > > > > >
> > > > > > > >         for_each_possible_cpu(i) {
> > > > > > > >                 struct sched_entity *se = tg->se[i];
> > > > > > > > +               struct rq *rq = cpu_rq(i);
> > > > > > > > +               struct rq_flags rf;
> > > > > > > > +
> > > > > > > > +               rq_lock_irqsave(rq, &rf);
> > > > > > > >
> > > > > > > > +               __dequeue_latency(se->cfs_rq, se);
> > > > > > > >                 WRITE_ONCE(se->latency_offset, latency);
> > > > > > > > +
> > > > > > > > +               rq_unlock_irqrestore(rq, &rf);
> > > > > > > >         }
> > > > > > > >
> > > > > > > >         mutex_unlock(&shares_mutex);
> > > > > > > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > > > > > > > index a15fb955092c..76bca172585c 100644
> > > > > > > > --- a/kernel/sched/sched.h
> > > > > > > > +++ b/kernel/sched/sched.h
> > > > > > > > @@ -599,6 +599,7 @@ struct cfs_rq {
> > > > > > > >  #endif
> > > > > > > >
> > > > > > > >         struct rb_root_cached   tasks_timeline;
> > > > > > > > +       struct rb_root_cached   latency_timeline;
> > > > > > > >
> > > > > > > >         /*
> > > > > > > >          * 'curr' points to currently running entity on this cfs_rq.
> > > > > > > > --
> > > > > > > > 2.17.1
> > > > > > > >
> > > > > > > >
> > > > > > > >

Re: [PATCH v5 4/7] sched/fair: Take into account latency priority at wakeup

[Chen Yu]

Hi Vincent,
On 2022-09-25 at 16:39:05 +0200, Vincent Guittot wrote:
[...]
> +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se)
> +{
> +	long latency_offset = se->latency_offset;
> +
> +	/*
> +	 * A negative latency offset means that the sched_entity has latency
> +	 * requirement that needs to be evaluated versus other entity.
> +	 * Otherwise, use the latency weight to evaluate how much scheduling
> +	 * delay is acceptable by se.
> +	 */
> +	if ((se->latency_offset < 0) || (curr->latency_offset < 0))
Maybe use latency_offset < 0 directly?
BTW, is it the policy that requires the user to provide a negative
latency nice so as to tell the kernel to compare between two tasks?
Maybe I missed the scenario, I'm thinking of the reason why we used
" || " rather than " && " above. To be more specific, why not comparing
se and curr only when they both have high requirement on latency(negative)?

The benefit of using "||" I'm thinking of is that, if se->latency_offset < 0
and curr->latency_offset > 0, the latency_offset would be even smaller than
se->latency_offset, which will make the preemption easier. And vice verse.

thanks,
Chenyu
> +		latency_offset -= curr->latency_offset;
> +
> +	return latency_offset;
> +}

Re: [PATCH v5 4/7] sched/fair: Take into account latency priority at wakeup

[Vincent Guittot]

On Sat, 22 Oct 2022 at 17:09, Chen Yu <yu.c.chen@intel.com> wrote:
>
> Hi Vincent,
> On 2022-09-25 at 16:39:05 +0200, Vincent Guittot wrote:
> [...]
> > +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se)
> > +{
> > +     long latency_offset = se->latency_offset;
> > +
> > +     /*
> > +      * A negative latency offset means that the sched_entity has latency
> > +      * requirement that needs to be evaluated versus other entity.
> > +      * Otherwise, use the latency weight to evaluate how much scheduling
> > +      * delay is acceptable by se.
> > +      */
> > +     if ((se->latency_offset < 0) || (curr->latency_offset < 0))
> Maybe use latency_offset < 0 directly?

yes

> BTW, is it the policy that requires the user to provide a negative
> latency nice so as to tell the kernel to compare between two tasks?

Only tasks with negative latency nice value want to be more aggressive
than the default behavior to preempt at wakeup.

> Maybe I missed the scenario, I'm thinking of the reason why we used
> " || " rather than " && " above. To be more specific, why not comparing
> se and curr only when they both have high requirement on latency(negative)?

When one task wants to be scheduled in priority (negative latency
nice), we want to keep the same offset for a same delta between the
latency nice prio.


>
> The benefit of using "||" I'm thinking of is that, if se->latency_offset < 0
> and curr->latency_offset > 0, the latency_offset would be even smaller than
> se->latency_offset, which will make the preemption easier. And vice verse.
>
> thanks,
> Chenyu
> > +             latency_offset -= curr->latency_offset;
> > +
> > +     return latency_offset;
> > +}

Re: [PATCH v5 0/7] Add latency priority for CFS class

[K Prateek Nayak]

Hello Vincent,

I've rerun some tests with a different configuration with more
contention for CPU and I can see a linear behavior. Sharing the
results below.

On 10/13/2022 8:54 PM, Vincent Guittot wrote:
>
> [..snip..]
>>
>> o Hackbench and Cyclictest in NPS1 configuration
>>
>> perf bench sched messaging -p -t -l 100000 -g 16&
>> cyclictest --policy other -D 5 -q -n -H 20000
>>
>> -----------------------------------------------------------------------------------------------------------------
>> |Hackbench     |      Cyclictest LN = 19        |         Cyclictest LN = 0       |      Cyclictest LN = -20    |
>> |LN            |--------------------------------|---------------------------------|-----------------------------|
>> |v             |   Min  |   Avg   |  Max        |     Min  |   Avg   |  Max       |     Min  |   Avg   |  Max   |
>> |--------------|--------|---------|-------------|----------|---------|------------|----------|---------|--------|
>> |0             |  54.00 |  117.00 | 3021.67     |    53.67 |  65.33  | 133.00     |    53.67 |  65.00  | 201.33 |  ^
>> |19            |  50.00 |  100.67 | 3099.33     |    41.00 |  64.33  | 1014.33    |    54.00 |  63.67  | 213.33 |
>> |-20           |  53.00 |  169.00 | 11661.67    |    53.67 |  217.33 | 14313.67   |    46.00 |  61.33  | 236.00 |  ^
>> -----------------------------------------------------------------------------------------------------------------
> 
> The latency results look good with Cyclictest LN:0 and hackbench LN:0.
> 133us max latency. This suggests that your system is not overloaded
> and cyclictest doesn't really compete with others to run.

Following is the result of running cyclictest alongside hackbench with 32 groups:

perf bench sched messaging -p -l 100000 -g 32&
cyclictest --policy other -D 5 -q -n -H 20000

----------------------------------------------------------------------------------------------------------
| Hackbench   |      Cyclictest LN = 19      |      Cyclictest LN = 0        |    Cyclictest LN = -20    |
| LN          |------------------------------|-------------------------------|---------------------------|
|             |   Min  |   Avg   |  Max      |   Min  |   Avg   |   Max      |   Min  |  Avg  |   Max    |
|-------------|--------|---------|-----------|--------|---------|------------|--------|-------|----------|
| 0           |  54.00 |  165.00 | 6899.00   |  22.00 |  85.00  |  3294.00   |  23.00 | 64.00 |  276.00  |
| 19          |  53.00 |  173.00 | 3275.00   |  40.00 |  60.00  |  2276.00   |  13.00 | 59.00 |  94.00   |
| -20         |  52.00 |  293.00 | 19980.00  |  52.00 |  280.00 |  14305.00  |  53.00 | 95.00 |  5713.00 |
----------------------------------------------------------------------------------------------------------

I see a spike for Max in (0, 0) configuration and the latency decreases
monotonically with lower latency nice value.

> 
>>
>> o Hackbench and schbench in NPS1 configuration
>>
>> perf bench sched messaging -p -t -l 1000000 -g 16&
>> schebcnh -m 1 -t 64 -s 30s
>>
>> ------------------------------------------------------------------------------------------------------------
>> |Hackbench     |   schbench LN = 19         |        schbench LN = 0         |       schbench LN = -20     |
>> |LN            |----------------------------|--------------------------------|-----------------------------|
>> |v             |  90th  |  95th  |  99th    |   90th  |  95th   |  99th      |   90th  |   95th   | 99th   |
>> |--------------|--------|--------|----------|---------|---------|------------|---------|----------|--------|
>> |0             |  4264  |  6744  |  15664   |   17952 |  32672  |  55488     |   15088 |   25312  | 50112  |
>> |19            |  288   |  613   |  2332    |   274   |  1015   |  3628      |   374   |   1394   | 4424   |
>> |-20           |  35904 |  47680 |  79744   |   87168 |  113536 |  176896    |   13008 |   21216  | 42560  |   ^
>> ------------------------------------------------------------------------------------------------------------
> 
> For the schbench, your test is 30 seconds long which is longer than
> the duration of perf bench sched messaging -p -t -l 1000000 -g 16&
> 
> The duration of the latter varies depending of latency nice value so
> schbench is disturb more time in some cases

I've rerun this with hackbench running 128 groups alongside schbench
with 2 messenger and 1 worker each. With larger worker count, I still
see non-monotonic behavior in 99th percentile latency of schbench.
I also see number of latency samples collected by schbench to vary
over the 30 second run for different latency nice values which could
also pay a part in seeing the unexpected behavior. For lower worker
count, I see the number of samples collected is similar. Following
is the configuration and the latency reported by schbench:

perf bench sched messaging -p -t -l 150000 -g 128&
schbench -m 2 -t 1 -s 30s

Note: In all cases, hackbench runs longer than schbench.

-------------------------------------------------------------------------------------------------
| Hackbench |     schbench LN = 19       |      schbench LN = 0      |     schbench LN = -20    |
| LN        |----------------------------|---------------------------|--------------------------|
|           |  90th  |  95th  |  99th    |  90th  |  95th  |  99th   |  90th  |  95th  |  99th  |
|-----------|--------|--------|----------|--------|--------|---------|--------|--------|--------|
| 0         |  42    |  92    |  2972    |  26    |  49    |  2356   |  9     |  11    |  20    |
| 19        |  35    |  424   |  4984    |  13    |  390   |  5096   |  8     |  10    |  14    | ^
| -19       |  144   |  3516  |  110208  |  61    |  807   |  34880  |  25    |  39    |  295   |
-------------------------------------------------------------------------------------------------

I see 90th and 95th percentile latency decrease monotonically with
latency nice value of schbench (for a fixed latency nice value of
hackbench) but there are cases where 99th percentile latency
reported by schbench may not strictly decrease with lower latency
nice value (Marked with ^)

Note: Only a small number of bad samples can affect the 99th
percentile latency for the above configuration. The monotonic
behavior in 90th and 95th percentile latency is a good data point
to show latency nice is indeed working as expected.

If there is any specific workload you would like me to run on the
test system, or any additional data you would like for above
workloads, please do let me know.

--
Thanks and Regards,
Prateek

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Dietmar Eggemann]

On 12/10/2022 17:21, Vincent Guittot wrote:
> Le mardi 11 oct. 2022 � 18:54:27 (-0500), Youssef Esmat a �crit :
>> Hi Vincent,
>>
>> On Tue, Oct 11, 2022 at 12:10 PM Vincent Guittot
>> <vincent.guittot@linaro.org> wrote:

[...]

> @@ -10894,12 +10898,17 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,
>  static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
>  				    struct cftype *cft)
>  {
> +	unsigned long period = sysctl_sched_latency;
>  	int last_delta = INT_MAX;
>  	int prio, delta;
>  	s64 weight;
>  
> +	if (sched_feat(GENTLE_FAIR_SLEEPERS))
> +		period >>= 1;
> +
>  	weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX;
> -	weight = div_s64(weight, sysctl_sched_latency);
> +	period = sysctl_sched_latency;

Looks like this line is wrong here, period has been set already for
GENTLE_FAIR_SLEEPERS and !GENTLE_FAIR_SLEEPERS?

> +	weight = div_s64(weight, period);
>  
>  	/* Find the closest nice value to the current weight */
>  	for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) {

[...]

Re: [PATCH v5 7/7] sched/fair: Add latency list

[Vincent Guittot]

On Wed, 26 Oct 2022 at 12:44, Dietmar Eggemann <dietmar.eggemann@arm.com> wrote:
>
> On 12/10/2022 17:21, Vincent Guittot wrote:
> > Le mardi 11 oct. 2022 � 18:54:27 (-0500), Youssef Esmat a �crit :
> >> Hi Vincent,
> >>
> >> On Tue, Oct 11, 2022 at 12:10 PM Vincent Guittot
> >> <vincent.guittot@linaro.org> wrote:
>
> [...]
>
> > @@ -10894,12 +10898,17 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,
> >  static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
> >                                   struct cftype *cft)
> >  {
> > +     unsigned long period = sysctl_sched_latency;
> >       int last_delta = INT_MAX;
> >       int prio, delta;
> >       s64 weight;
> >
> > +     if (sched_feat(GENTLE_FAIR_SLEEPERS))
> > +             period >>= 1;
> > +
> >       weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX;
> > -     weight = div_s64(weight, sysctl_sched_latency);
> > +     period = sysctl_sched_latency;
>
> Looks like this line is wrong here, period has been set already for
> GENTLE_FAIR_SLEEPERS and !GENTLE_FAIR_SLEEPERS?

Yes, I have factorized this in a function and the new version that I'm
preparing so The formula is not duplicated with the risk of error like
above

>
> > +     weight = div_s64(weight, period);
> >
> >       /* Find the closest nice value to the current weight */
> >       for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) {
>
> [...]
>

Re: [PATCH v5 0/7] Add latency priority for CFS class

[Vincent Guittot]

Hi Prateek,

On Tue, 25 Oct 2022 at 08:36, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Hello Vincent,
>
> I've rerun some tests with a different configuration with more
> contention for CPU and I can see a linear behavior. Sharing the
> results below.
>
> On 10/13/2022 8:54 PM, Vincent Guittot wrote:
> >
> > [..snip..]
> >>
> >> o Hackbench and Cyclictest in NPS1 configuration
> >>
> >> perf bench sched messaging -p -t -l 100000 -g 16&
> >> cyclictest --policy other -D 5 -q -n -H 20000
> >>
> >> -----------------------------------------------------------------------------------------------------------------
> >> |Hackbench     |      Cyclictest LN = 19        |         Cyclictest LN = 0       |      Cyclictest LN = -20    |
> >> |LN            |--------------------------------|---------------------------------|-----------------------------|
> >> |v             |   Min  |   Avg   |  Max        |     Min  |   Avg   |  Max       |     Min  |   Avg   |  Max   |
> >> |--------------|--------|---------|-------------|----------|---------|------------|----------|---------|--------|
> >> |0             |  54.00 |  117.00 | 3021.67     |    53.67 |  65.33  | 133.00     |    53.67 |  65.00  | 201.33 |  ^
> >> |19            |  50.00 |  100.67 | 3099.33     |    41.00 |  64.33  | 1014.33    |    54.00 |  63.67  | 213.33 |
> >> |-20           |  53.00 |  169.00 | 11661.67    |    53.67 |  217.33 | 14313.67   |    46.00 |  61.33  | 236.00 |  ^
> >> -----------------------------------------------------------------------------------------------------------------
> >
> > The latency results look good with Cyclictest LN:0 and hackbench LN:0.
> > 133us max latency. This suggests that your system is not overloaded
> > and cyclictest doesn't really compete with others to run.
>
> Following is the result of running cyclictest alongside hackbench with 32 groups:
>
> perf bench sched messaging -p -l 100000 -g 32&
> cyclictest --policy other -D 5 -q -n -H 20000
>
> ----------------------------------------------------------------------------------------------------------
> | Hackbench   |      Cyclictest LN = 19      |      Cyclictest LN = 0        |    Cyclictest LN = -20    |
> | LN          |------------------------------|-------------------------------|---------------------------|
> |             |   Min  |   Avg   |  Max      |   Min  |   Avg   |   Max      |   Min  |  Avg  |   Max    |
> |-------------|--------|---------|-----------|--------|---------|------------|--------|-------|----------|
> | 0           |  54.00 |  165.00 | 6899.00   |  22.00 |  85.00  |  3294.00   |  23.00 | 64.00 |  276.00  |
> | 19          |  53.00 |  173.00 | 3275.00   |  40.00 |  60.00  |  2276.00   |  13.00 | 59.00 |  94.00   |
> | -20         |  52.00 |  293.00 | 19980.00  |  52.00 |  280.00 |  14305.00  |  53.00 | 95.00 |  5713.00 |
> ----------------------------------------------------------------------------------------------------------
>
> I see a spike for Max in (0, 0) configuration and the latency decreases
> monotonically with lower latency nice value.

Your results looks good

>
> >
> >>
> >> o Hackbench and schbench in NPS1 configuration
> >>
> >> perf bench sched messaging -p -t -l 1000000 -g 16&
> >> schebcnh -m 1 -t 64 -s 30s
> >>
> >> ------------------------------------------------------------------------------------------------------------
> >> |Hackbench     |   schbench LN = 19         |        schbench LN = 0         |       schbench LN = -20     |
> >> |LN            |----------------------------|--------------------------------|-----------------------------|
> >> |v             |  90th  |  95th  |  99th    |   90th  |  95th   |  99th      |   90th  |   95th   | 99th   |
> >> |--------------|--------|--------|----------|---------|---------|------------|---------|----------|--------|
> >> |0             |  4264  |  6744  |  15664   |   17952 |  32672  |  55488     |   15088 |   25312  | 50112  |
> >> |19            |  288   |  613   |  2332    |   274   |  1015   |  3628      |   374   |   1394   | 4424   |
> >> |-20           |  35904 |  47680 |  79744   |   87168 |  113536 |  176896    |   13008 |   21216  | 42560  |   ^
> >> ------------------------------------------------------------------------------------------------------------
> >
> > For the schbench, your test is 30 seconds long which is longer than
> > the duration of perf bench sched messaging -p -t -l 1000000 -g 16&
> >
> > The duration of the latter varies depending of latency nice value so
> > schbench is disturb more time in some cases
>
> I've rerun this with hackbench running 128 groups alongside schbench
> with 2 messenger and 1 worker each. With larger worker count, I still
> see non-monotonic behavior in 99th percentile latency of schbench.
> I also see number of latency samples collected by schbench to vary
> over the 30 second run for different latency nice values which could
> also pay a part in seeing the unexpected behavior. For lower worker
> count, I see the number of samples collected is similar. Following
> is the configuration and the latency reported by schbench:
>
> perf bench sched messaging -p -t -l 150000 -g 128&
> schbench -m 2 -t 1 -s 30s
>
> Note: In all cases, hackbench runs longer than schbench.
>
> -------------------------------------------------------------------------------------------------
> | Hackbench |     schbench LN = 19       |      schbench LN = 0      |     schbench LN = -20    |
> | LN        |----------------------------|---------------------------|--------------------------|
> |           |  90th  |  95th  |  99th    |  90th  |  95th  |  99th   |  90th  |  95th  |  99th  |
> |-----------|--------|--------|----------|--------|--------|---------|--------|--------|--------|
> | 0         |  42    |  92    |  2972    |  26    |  49    |  2356   |  9     |  11    |  20    |
> | 19        |  35    |  424   |  4984    |  13    |  390   |  5096   |  8     |  10    |  14    | ^
> | -19       |  144   |  3516  |  110208  |  61    |  807   |  34880  |  25    |  39    |  295   |
> -------------------------------------------------------------------------------------------------
>
> I see 90th and 95th percentile latency decrease monotonically with
> latency nice value of schbench (for a fixed latency nice value of
> hackbench) but there are cases where 99th percentile latency
> reported by schbench may not strictly decrease with lower latency
> nice value (Marked with ^)
>
> Note: Only a small number of bad samples can affect the 99th
> percentile latency for the above configuration. The monotonic
> behavior in 90th and 95th percentile latency is a good data point
> to show latency nice is indeed working as expected.

Yes, I think you are right that the 99th percentile is not stable
enough because it can be impacted by a small number of bad samples

>
> If there is any specific workload you would like me to run on the
> test system, or any additional data you would like for above
> workloads, please do let me know.

Thanks a lot for your tests.
I'm about to send v6

>
> --
> Thanks and Regards,
> Prateek