[PATCH v12 0/8] Add latency priority for CFS class

[PATCH v12 0/8] Add latency priority for CFS class

[Vincent Guittot]

This patchset restarts the work about adding a latency priority to describe
the latency tolerance of cfs tasks.

Patch [1] is a new one that has been added with v6. It fixes an
unfairness for low prio tasks because of wakeup_gran() being bigger
than the maximum vruntime credit that a waking task can keep after
sleeping.

The patches [2-4] have been done by Parth:
https://lore.kernel.org/lkml/20200228090755.22829-1-parth@linux.ibm.com/

I have just rebased and moved the set of latency priority outside the
priority update. I have removed the reviewed tag because the patches
are 2 years old.

This aims to be a generic interface and the following patches is one use
of it to improve the scheduling latency of cfs tasks.

Patch [5] uses latency nice priority to define a latency offset
and then decide if a cfs task can or should preempt the current
running task. The patch gives some tests results with cyclictests and
hackbench to highlight the benefit of latency priority for short
interactive task or long intensive tasks.

Patch [6] adds the support of latency nice priority to task group by
adding a cpu.latency.nice field. The range is [-20:19] as for setting task
latency priority.

Patch [7] makes sched_core taking into account the latency offset.

Patch [8] adds a rb tree to cover some corner cases where the latency
sensitive task (priority < 0) is preempted by high priority task (RT/DL)
or fails to preempt them. This patch ensures that tasks will have at least
a slice of sched_min_granularity in priority at wakeup.

I have also backported the patchset on a dragonboard RB3 with an android
mainline kernel based on v5.18 for a quick test. I have used the
TouchLatency app which is part of AOSP and described to be a very good
test to highlight jitter and jank frame sources of a system [1].
In addition to the app, I have added some short running tasks waking-up
regularly (to use the 8 cpus for 4 ms every 37777us) to stress the system
without overloading it (and disabling EAS). The 1st results shows that the
patchset helps to reduce the missed deadline frames from 5% to less than
0.1% when the cpu.latency.nice of task group are set. I haven't rerun the
test with latest version.

I have also tested the patchset with the modified version of the alsa
latency test that has been shared by Tim. The test quickly xruns with
default latency nice priority 0 but is able to run without underuns with
a latency -20 and hackbench running simultaneously.

While preparing the version 8, I have evaluated the benefit of using an
augmented rbtree instead of adding a rbtree for latency sensitive entities,
which was a relevant suggestion done by PeterZ. Although the augmented
rbtree enables to sort additional information in the tree with a limited
overhead, it has more impact on legacy use cases (latency_nice >= 0)
because the augmented callbacks are always called to maintain this
additional information even when there is no sensitive tasks. In such
cases, the dedicated rbtree remains empty and the overhead is reduced to
loading a cached null node pointer. Nevertheless, we might want to
reconsider the augmented rbtree once the use of negative latency_nice will
be more widlely deployed. At now, the different tests that I have done,
have not shown improvements with augmented rbtree.

Below are some hackbench results (from v10):
        2 rbtrees               augmented rbtree        augmented rbtree	
                                sorted by vruntime      sorted by wakeup_vruntime
sched	pipe	
avg     26311,000               25976,667               25839,556
stdev   0,15 %                  0,28 %                  0,24 %
vs tip  0,50 %                  -0,78 %                 -1,31 %
hackbench	1 group	
avg     1,315                   1,344                   1,359
stdev   0,88 %                  1,55 %                  1,82 %
vs tip  -0,47 %                 -2,68 %                 -3,87 %
hackbench	4 groups
avg     1,339                   1,365                   1,367
stdev   2,39 %                  2,26 %                  3,58 %
vs tip  -0,08 %                 -2,01 %                 -2,22 %
hackbench	8 groups
avg     1,233                   1,286                   1,301
stdev   0,74 %                  1,09 %                  1,52 %
vs tip  0,29 %                  -4,05 %                 -5,27 %
hackbench	16 groups	
avg     1,268                   1,313                   1,319
stdev   0,85 %                  1,60 %                  0,68 %
vs tip  -0,02 %                 -3,56 %                 -4,01 %

[1] https://source.android.com/docs/core/debug/eval_perf#touchlatency

Change since v11:
- init latency_node of task group entity

Change since v10:
- remove sched_latency_to_weight array and use a calc_latency_offset() instead
- save latency_prio instead for task group instead of latency offset
- enqueue back an entity when changing the latency nice prio fo a task group

Change since v9:
- Rebase
- add tags

Change since v8:
- Rename get_sched_latency by get_sleep_latency
- move latency nice defines in sched/prio.h and fix latency_prio init value
- Fix typo and comments

Change since v7:
- Replaced se->on_latency by using RB_CLEAR_NODE() and RB_EMPTY_NODE()
- Clarify the limit behavior fo the cgroup cpu.latenyc_nice

Change since v6:
- Fix compilation error for !CONFIG_SCHED_DEBUG

Change since v5:
- Add patch 1 to fix unfairness for low prio task. This has been
  discovered while studying Youssef's tests results with latency nice
  which were hitting the same problem.
- Fixed latency_offset computation to take into account
  GENTLE_FAIR_SLEEPERS. This has diseappeared with v2and has been raised
  by Youssef's tests.
- Reworked and optimized how latency_offset in used to check for
  preempting current task at wakeup and tick. This cover more cases too.
- Add patch 9 to remove check_preempt_from_others() which is not needed
  anymore with the rb tree.

Change since v4:
- Removed permission checks to set latency priority. This enables user
  without elevated privilege like audio application to set their latency
  priority as requested by Tim.
- Removed cpu.latency and replaced it by cpu.latency.nice so we keep a
  generic interface not tied to latency_offset which can be used to
  implement other latency features.
- Added an entry in Documentation/admin-guide/cgroup-v2.rst to describe
  cpu.latency.nice.
- Fix some typos.

Change since v3:
- Fix 2 compilation warnings raised by kernel test robot <lkp@intel.com>

Change since v2:
- Set a latency_offset field instead of saving a weight and computing it
  on the fly.
- Make latency_offset available for task group: cpu.latency
- Fix some corner cases to make latency sensitive tasks schedule first and
  add a rb tree for latency sensitive task.

Change since v1:
- fix typo
- move some codes in the right patch to make bisect happy
- simplify and fixed how the weight is computed
- added support of sched core patch 7

Parth Shah (3):
  sched: Introduce latency-nice as a per-task attribute
  sched/core: Propagate parent task's latency requirements to the child
    task
  sched: Allow sched_{get,set}attr to change latency_nice of the task

Vincent Guittot (5):
  sched/fair: fix unfairness at wakeup
  sched/fair: Take into account latency priority at wakeup
  sched/fair: Add sched group latency support
  sched/core: Support latency priority with sched core
  sched/fair: Add latency list

 Documentation/admin-guide/cgroup-v2.rst |  10 ++
 include/linux/sched.h                   |   4 +
 include/linux/sched/prio.h              |  27 +++
 include/uapi/linux/sched.h              |   4 +-
 include/uapi/linux/sched/types.h        |  19 ++
 init/init_task.c                        |   1 +
 kernel/sched/core.c                     |  65 +++++++
 kernel/sched/debug.c                    |   1 +
 kernel/sched/fair.c                     | 222 ++++++++++++++++++++----
 kernel/sched/sched.h                    |  50 +++++-
 tools/include/uapi/linux/sched.h        |   4 +-
 11 files changed, 371 insertions(+), 36 deletions(-)

-- 
2.34.1

[PATCH v12 1/8] sched/fair: fix unfairness at wakeup

[Vincent Guittot]

At wake up, the vruntime of a task is updated to not be more older than
a sched_latency period behind the min_vruntime. This prevents long sleeping
task to get unlimited credit at wakeup.
Such waking task should preempt current one to use its CPU bandwidth but
wakeup_gran() can be larger than sched_latency, filter out the
wakeup preemption and as a results steals some CPU bandwidth to
the waking task.

Make sure that a task, which vruntime has been capped, will preempt current
task and use its CPU bandwidth even if wakeup_gran() is in the same range
as sched_latency.

If the waking task failed to preempt current it could to wait up to
sysctl_sched_min_granularity before preempting it during next tick.

Strictly speaking, we should use cfs->min_vruntime instead of
curr->vruntime but it doesn't worth the additional overhead and complexity
as the vruntime of current should be close to min_vruntime if not equal.

Reported-by: Youssef Esmat <youssefesmat@chromium.org>
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 kernel/sched/fair.c  | 46 ++++++++++++++++++++------------------------
 kernel/sched/sched.h | 34 +++++++++++++++++++++++++++++++-
 2 files changed, 54 insertions(+), 26 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ff4dbbae3b10..81bef11eb660 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4654,33 +4654,17 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
 	u64 vruntime = cfs_rq->min_vruntime;
 	u64 sleep_time;
 
-	/*
-	 * The 'current' period is already promised to the current tasks,
-	 * however the extra weight of the new task will slow them down a
-	 * little, place the new task so that it fits in the slot that
-	 * stays open at the end.
-	 */
-	if (initial && sched_feat(START_DEBIT))
-		vruntime += sched_vslice(cfs_rq, se);
-
-	/* sleeps up to a single latency don't count. */
-	if (!initial) {
-		unsigned long thresh;
-
-		if (se_is_idle(se))
-			thresh = sysctl_sched_min_granularity;
-		else
-			thresh = sysctl_sched_latency;
-
+	if (!initial)
+		/* sleeps up to a single latency don't count. */
+		vruntime -= get_sleep_latency(se_is_idle(se));
+	else if (sched_feat(START_DEBIT))
 		/*
-		 * Halve their sleep time's effect, to allow
-		 * for a gentler effect of sleepers:
+		 * The 'current' period is already promised to the current tasks,
+		 * however the extra weight of the new task will slow them down a
+		 * little, place the new task so that it fits in the slot that
+		 * stays open at the end.
 		 */
-		if (sched_feat(GENTLE_FAIR_SLEEPERS))
-			thresh >>= 1;
-
-		vruntime -= thresh;
-	}
+		vruntime += sched_vslice(cfs_rq, se);
 
 	/*
 	 * Pull vruntime of the entity being placed to the base level of
@@ -7721,6 +7705,18 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
 		return -1;
 
 	gran = wakeup_gran(se);
+
+	/*
+	 * At wake up, the vruntime of a task is capped to not be older than
+	 * a sched_latency period compared to min_vruntime. This prevents long
+	 * sleeping task to get unlimited credit at wakeup. Such waking up task
+	 * has to preempt current in order to not lose its share of CPU
+	 * bandwidth but wakeup_gran() can become higher than scheduling period
+	 * for low priority task. Make sure that long sleeping task will get a
+	 * chance to preempt current.
+	 */
+	gran = min_t(s64, gran, get_latency_max());
+
 	if (vdiff > gran)
 		return 1;
 
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 3e8df6d31c1e..51ba0af7fb27 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2458,9 +2458,9 @@ extern void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags);
 extern const_debug unsigned int sysctl_sched_nr_migrate;
 extern const_debug unsigned int sysctl_sched_migration_cost;
 
-#ifdef CONFIG_SCHED_DEBUG
 extern unsigned int sysctl_sched_latency;
 extern unsigned int sysctl_sched_min_granularity;
+#ifdef CONFIG_SCHED_DEBUG
 extern unsigned int sysctl_sched_idle_min_granularity;
 extern unsigned int sysctl_sched_wakeup_granularity;
 extern int sysctl_resched_latency_warn_ms;
@@ -2475,6 +2475,38 @@ extern unsigned int sysctl_numa_balancing_scan_size;
 extern unsigned int sysctl_numa_balancing_hot_threshold;
 #endif
 
+static inline unsigned long get_sleep_latency(bool idle)
+{
+	unsigned long thresh;
+
+	if (idle)
+		thresh = sysctl_sched_min_granularity;
+	else
+		thresh = sysctl_sched_latency;
+
+	/*
+	 * Halve their sleep time's effect, to allow
+	 * for a gentler effect of sleepers:
+	 */
+	if (sched_feat(GENTLE_FAIR_SLEEPERS))
+		thresh >>= 1;
+
+	return thresh;
+}
+
+static inline unsigned long get_latency_max(void)
+{
+	unsigned long thresh = get_sleep_latency(false);
+
+	 /*
+	  * If the waking task failed to preempt current it could to wait up to
+	  * sysctl_sched_min_granularity before preempting it during next tick.
+	  */
+	thresh -= sysctl_sched_min_granularity;
+
+	return thresh;
+}
+
 #ifdef CONFIG_SCHED_HRTICK
 
 /*
-- 
2.34.1

[PATCH v12 2/8] sched: Introduce latency-nice as a per-task attribute

[Vincent Guittot]

From: Parth Shah <parth@linux.ibm.com>

Latency-nice indicates the latency requirements of a task with respect
to the other tasks in the system. The value of the attribute can be within
the range of [-20, 19] both inclusive to be in-line with the values just
like task nice values.

latency_nice = -20 indicates the task to have the least latency as
compared to the tasks having latency_nice = +19.

The latency_nice may affect only the CFS SCHED_CLASS by getting
latency requirements from the userspace.

Additionally, add debugging bits for newly added latency_nice attribute.

Signed-off-by: Parth Shah <parth@linux.ibm.com>
[rebase, move defines in sched/prio.h]
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 include/linux/sched.h      |  1 +
 include/linux/sched/prio.h | 18 ++++++++++++++++++
 kernel/sched/debug.c       |  1 +
 3 files changed, 20 insertions(+)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 4df2b3e76b30..6c61bde49152 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -784,6 +784,7 @@ struct task_struct {
 	int				static_prio;
 	int				normal_prio;
 	unsigned int			rt_priority;
+	int				latency_nice;
 
 	struct sched_entity		se;
 	struct sched_rt_entity		rt;
diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h
index ab83d85e1183..bfcd7f1d1e11 100644
--- a/include/linux/sched/prio.h
+++ b/include/linux/sched/prio.h
@@ -42,4 +42,22 @@ static inline long rlimit_to_nice(long prio)
 	return (MAX_NICE - prio + 1);
 }
 
+/*
+ * Latency nice is meant to provide scheduler hints about the relative
+ * latency requirements of a task with respect to other tasks.
+ * Thus a task with latency_nice == 19 can be hinted as the task with no
+ * latency requirements, in contrast to the task with latency_nice == -20
+ * which should be given priority in terms of lower latency.
+ */
+#define MAX_LATENCY_NICE	19
+#define MIN_LATENCY_NICE	-20
+
+#define LATENCY_NICE_WIDTH	\
+	(MAX_LATENCY_NICE - MIN_LATENCY_NICE + 1)
+
+/*
+ * Default tasks should be treated as a task with latency_nice = 0.
+ */
+#define DEFAULT_LATENCY_NICE	0
+
 #endif /* _LINUX_SCHED_PRIO_H */
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 1637b65ba07a..68be7a3e42a3 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1043,6 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 #endif
 	P(policy);
 	P(prio);
+	P(latency_nice);
 	if (task_has_dl_policy(p)) {
 		P(dl.runtime);
 		P(dl.deadline);
-- 
2.34.1

[PATCH v12 3/8] sched/core: Propagate parent task's latency requirements to the child task

[Vincent Guittot]

From: Parth Shah <parth@linux.ibm.com>

Clone parent task's latency_nice attribute to the forked child task.

Reset the latency_nice value to default value when the child task is
set to sched_reset_on_fork.

Also, initialize init_task.latency_nice value with DEFAULT_LATENCY_NICE
value

Signed-off-by: Parth Shah <parth@linux.ibm.com>
[rebase]
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 init/init_task.c    | 1 +
 kernel/sched/core.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/init/init_task.c b/init/init_task.c
index ff6c4b9bfe6b..7dd71dd2d261 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -78,6 +78,7 @@ struct task_struct init_task
 	.prio		= MAX_PRIO - 20,
 	.static_prio	= MAX_PRIO - 20,
 	.normal_prio	= MAX_PRIO - 20,
+	.latency_nice	= DEFAULT_LATENCY_NICE,
 	.policy		= SCHED_NORMAL,
 	.cpus_ptr	= &init_task.cpus_mask,
 	.user_cpus_ptr	= NULL,
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 4580fe3e1d0c..28b397f9698b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4681,6 +4681,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
 		p->prio = p->normal_prio = p->static_prio;
 		set_load_weight(p, false);
 
+		p->latency_nice = DEFAULT_LATENCY_NICE;
 		/*
 		 * We don't need the reset flag anymore after the fork. It has
 		 * fulfilled its duty:
-- 
2.34.1

[PATCH v12 4/8] sched: Allow sched_{get,set}attr to change latency_nice of the task

[Vincent Guittot]

From: Parth Shah <parth@linux.ibm.com>

Introduce the latency_nice attribute to sched_attr and provide a
mechanism to change the value with the use of sched_setattr/sched_getattr
syscall.

Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in
latency_nice of the task on every sched_setattr syscall.

Signed-off-by: Parth Shah <parth@linux.ibm.com>
[rebase and add a dedicated __setscheduler_latency ]
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 include/uapi/linux/sched.h       |  4 +++-
 include/uapi/linux/sched/types.h | 19 +++++++++++++++++++
 kernel/sched/core.c              | 24 ++++++++++++++++++++++++
 tools/include/uapi/linux/sched.h |  4 +++-
 4 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h
index 3bac0a8ceab2..b2e932c25be6 100644
--- a/include/uapi/linux/sched.h
+++ b/include/uapi/linux/sched.h
@@ -132,6 +132,7 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_LATENCY_NICE		0x80
 
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
@@ -143,6 +144,7 @@ struct clone_args {
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_LATENCY_NICE)
 
 #endif /* _UAPI_LINUX_SCHED_H */
diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/types.h
index f2c4589d4dbf..db1e8199e8c8 100644
--- a/include/uapi/linux/sched/types.h
+++ b/include/uapi/linux/sched/types.h
@@ -10,6 +10,7 @@ struct sched_param {
 
 #define SCHED_ATTR_SIZE_VER0	48	/* sizeof first published struct */
 #define SCHED_ATTR_SIZE_VER1	56	/* add: util_{min,max} */
+#define SCHED_ATTR_SIZE_VER2	60	/* add: latency_nice */
 
 /*
  * Extended scheduling parameters data structure.
@@ -98,6 +99,22 @@ struct sched_param {
  * scheduled on a CPU with no more capacity than the specified value.
  *
  * A task utilization boundary can be reset by setting the attribute to -1.
+ *
+ * Latency Tolerance Attributes
+ * ===========================
+ *
+ * A subset of sched_attr attributes allows to specify the relative latency
+ * requirements of a task with respect to the other tasks running/queued in the
+ * system.
+ *
+ * @ sched_latency_nice	task's latency_nice value
+ *
+ * The latency_nice of a task can have any value in a range of
+ * [MIN_LATENCY_NICE..MAX_LATENCY_NICE].
+ *
+ * A task with latency_nice with the value of LATENCY_NICE_MIN can be
+ * taken for a task requiring a lower latency as opposed to the task with
+ * higher latency_nice.
  */
 struct sched_attr {
 	__u32 size;
@@ -120,6 +137,8 @@ struct sched_attr {
 	__u32 sched_util_min;
 	__u32 sched_util_max;
 
+	/* latency requirement hints */
+	__s32 sched_latency_nice;
 };
 
 #endif /* _UAPI_LINUX_SCHED_TYPES_H */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 28b397f9698b..d327614c70b0 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7443,6 +7443,14 @@ static void __setscheduler_params(struct task_struct *p,
 	p->rt_priority = attr->sched_priority;
 	p->normal_prio = normal_prio(p);
 	set_load_weight(p, true);
+
+}
+
+static void __setscheduler_latency(struct task_struct *p,
+		const struct sched_attr *attr)
+{
+	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
+		p->latency_nice = attr->sched_latency_nice;
 }
 
 /*
@@ -7585,6 +7593,13 @@ static int __sched_setscheduler(struct task_struct *p,
 			return retval;
 	}
 
+	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
+		if (attr->sched_latency_nice > MAX_LATENCY_NICE)
+			return -EINVAL;
+		if (attr->sched_latency_nice < MIN_LATENCY_NICE)
+			return -EINVAL;
+	}
+
 	if (pi)
 		cpuset_read_lock();
 
@@ -7619,6 +7634,9 @@ static int __sched_setscheduler(struct task_struct *p,
 			goto change;
 		if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
 			goto change;
+		if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
+		    attr->sched_latency_nice != p->latency_nice)
+			goto change;
 
 		p->sched_reset_on_fork = reset_on_fork;
 		retval = 0;
@@ -7707,6 +7725,7 @@ static int __sched_setscheduler(struct task_struct *p,
 		__setscheduler_params(p, attr);
 		__setscheduler_prio(p, newprio);
 	}
+	__setscheduler_latency(p, attr);
 	__setscheduler_uclamp(p, attr);
 
 	if (queued) {
@@ -7917,6 +7936,9 @@ static int sched_copy_attr(struct sched_attr __user *uattr, struct sched_attr *a
 	    size < SCHED_ATTR_SIZE_VER1)
 		return -EINVAL;
 
+	if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) &&
+	    size < SCHED_ATTR_SIZE_VER2)
+		return -EINVAL;
 	/*
 	 * XXX: Do we want to be lenient like existing syscalls; or do we want
 	 * to be strict and return an error on out-of-bounds values?
@@ -8154,6 +8176,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
 	get_params(p, &kattr);
 	kattr.sched_flags &= SCHED_FLAG_ALL;
 
+	kattr.sched_latency_nice = p->latency_nice;
+
 #ifdef CONFIG_UCLAMP_TASK
 	/*
 	 * This could race with another potential updater, but this is fine
diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sched.h
index 3bac0a8ceab2..b2e932c25be6 100644
--- a/tools/include/uapi/linux/sched.h
+++ b/tools/include/uapi/linux/sched.h
@@ -132,6 +132,7 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_LATENCY_NICE		0x80
 
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
@@ -143,6 +144,7 @@ struct clone_args {
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_LATENCY_NICE)
 
 #endif /* _UAPI_LINUX_SCHED_H */
-- 
2.34.1

[PATCH v12 5/8] sched/fair: Take into account latency priority at wakeup

[Vincent Guittot]

Take into account the latency priority of a thread when deciding to
preempt the current running thread. We don't want to provide more CPU
bandwidth to a thread but reorder the scheduling to run latency sensitive
task first whenever possible.

As long as a thread didn't use its bandwidth, it will be able to preempt
the current thread.

At the opposite, a thread with a low latency priority will preempt current
thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will
wait for the tick to get its sched slice.

                                   curr vruntime
                                       |
                      sysctl_sched_wakeup_granularity
                                   <-->
----------------------------------|----|-----------------------|---------------
                                  |    |<--------------------->
                                  |    .  sysctl_sched_latency
                                  |    .
default/current latency entity    |    .
                                  |    .
1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-
se preempts curr at wakeup ------>|<- se doesn't preempt curr -----------------
                                  |    .
                                  |    .
                                  |    .
low latency entity                |    .
                                   ---------------------->|
                               % of sysctl_sched_latency  |
1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-
preempt ------------------------------------------------->|<- do not preempt --
                                  |    .
                                  |    .
                                  |    .
high latency entity               |    .
         |<-----------------------|----.
         | % of sysctl_sched_latency   .
111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1
preempt->|<- se doesn't preempt curr ------------------------------------------

Tests results of nice latency impact on heavy load like hackbench:

hackbench -l (2560 / group) -g group
group        latency 0             latency 19
1            1.378(+/-  1%)      1.337(+/- 1%) + 3%
4            1.393(+/-  3%)      1.312(+/- 3%) + 6%
8            1.308(+/-  2%)      1.279(+/- 1%) + 2%
16           1.347(+/-  1%)      1.317(+/- 1%) + 2%

hackbench -p -l (2560 / group) -g group
group
1            1.836(+/- 17%)      1.148(+/- 5%) +37%
4            1.586(+/-  6%)      1.109(+/- 8%) +30%
8            1.209(+/-  4%)      0.780(+/- 4%) +35%
16           0.805(+/-  5%)      0.728(+/- 4%) +10%

By deacreasing the latency prio, we reduce the number of preemption at
wakeup and help hackbench making progress.

Test results of nice latency impact on short live load like cyclictest
while competing with heavy load like hackbench:

hackbench -l 10000 -g $group &
cyclictest --policy other -D 5 -q -n
        latency 0           latency -20
group   min  avg    max     min  avg    max
0       16    19     29      17   18     29
1       43   299   7359      63   84   3422
4       56   449  14806      45   83    284
8       63   820  51123      63   83    283
16      64  1326  70684      41  157  26852

group = 0 means that hackbench is not running.

The avg is significantly improved with nice latency -20 especially with
large number of groups but min and max remain quite similar. If we add the
histogram parameter to get details of latency, we have :

hackbench -l 10000 -g 16 &
cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
              latency 0    latency -20
Min Latencies:    64           62
Avg Latencies:  1170          107
Max Latencies: 88069        10417
50% latencies:   122           86
75% latencies:   614           91
85% latencies:   961           94
90% latencies:  1225           97
95% latencies:  6120          102
99% latencies: 18328          159

With percentile details, we see the benefit of nice latency -20 as
only 1% of the latencies are above 159us whereas the default latency
has got 15% around ~1ms or above and 5% over the 6ms.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 include/linux/sched.h      |  4 +++-
 include/linux/sched/prio.h |  9 +++++++++
 init/init_task.c           |  2 +-
 kernel/sched/core.c        | 19 ++++++++++++++-----
 kernel/sched/debug.c       |  2 +-
 kernel/sched/fair.c        | 32 +++++++++++++++++++++++++++-----
 kernel/sched/sched.h       | 11 +++++++++++
 7 files changed, 66 insertions(+), 13 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 6c61bde49152..38decae3e156 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -568,6 +568,8 @@ struct sched_entity {
 	/* cached value of my_q->h_nr_running */
 	unsigned long			runnable_weight;
 #endif
+	/* preemption offset in ns */
+	long				latency_offset;
 
 #ifdef CONFIG_SMP
 	/*
@@ -784,7 +786,7 @@ struct task_struct {
 	int				static_prio;
 	int				normal_prio;
 	unsigned int			rt_priority;
-	int				latency_nice;
+	int				latency_prio;
 
 	struct sched_entity		se;
 	struct sched_rt_entity		rt;
diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h
index bfcd7f1d1e11..be79503d86af 100644
--- a/include/linux/sched/prio.h
+++ b/include/linux/sched/prio.h
@@ -59,5 +59,14 @@ static inline long rlimit_to_nice(long prio)
  * Default tasks should be treated as a task with latency_nice = 0.
  */
 #define DEFAULT_LATENCY_NICE	0
+#define DEFAULT_LATENCY_PRIO	(DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2)
+
+/*
+ * Convert user-nice values [ -20 ... 0 ... 19 ]
+ * to static latency [ 0..39 ],
+ * and back.
+ */
+#define NICE_TO_LATENCY(nice)	((nice) + DEFAULT_LATENCY_PRIO)
+#define LATENCY_TO_NICE(prio)	((prio) - DEFAULT_LATENCY_PRIO)
 
 #endif /* _LINUX_SCHED_PRIO_H */
diff --git a/init/init_task.c b/init/init_task.c
index 7dd71dd2d261..071deff8dbd1 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -78,7 +78,7 @@ struct task_struct init_task
 	.prio		= MAX_PRIO - 20,
 	.static_prio	= MAX_PRIO - 20,
 	.normal_prio	= MAX_PRIO - 20,
-	.latency_nice	= DEFAULT_LATENCY_NICE,
+	.latency_prio	= DEFAULT_LATENCY_PRIO,
 	.policy		= SCHED_NORMAL,
 	.cpus_ptr	= &init_task.cpus_mask,
 	.user_cpus_ptr	= NULL,
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index d327614c70b0..d5b7e237d79b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1285,6 +1285,11 @@ static void set_load_weight(struct task_struct *p, bool update_load)
 	}
 }
 
+static void set_latency_offset(struct task_struct *p)
+{
+	p->se.latency_offset = calc_latency_offset(p->latency_prio);
+}
+
 #ifdef CONFIG_UCLAMP_TASK
 /*
  * Serializes updates of utilization clamp values
@@ -4681,7 +4686,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
 		p->prio = p->normal_prio = p->static_prio;
 		set_load_weight(p, false);
 
-		p->latency_nice = DEFAULT_LATENCY_NICE;
+		p->latency_prio = NICE_TO_LATENCY(0);
+		set_latency_offset(p);
+
 		/*
 		 * We don't need the reset flag anymore after the fork. It has
 		 * fulfilled its duty:
@@ -7449,8 +7456,10 @@ static void __setscheduler_params(struct task_struct *p,
 static void __setscheduler_latency(struct task_struct *p,
 		const struct sched_attr *attr)
 {
-	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
-		p->latency_nice = attr->sched_latency_nice;
+	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
+		p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice);
+		set_latency_offset(p);
+	}
 }
 
 /*
@@ -7635,7 +7644,7 @@ static int __sched_setscheduler(struct task_struct *p,
 		if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
 			goto change;
 		if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
-		    attr->sched_latency_nice != p->latency_nice)
+		    attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio))
 			goto change;
 
 		p->sched_reset_on_fork = reset_on_fork;
@@ -8176,7 +8185,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
 	get_params(p, &kattr);
 	kattr.sched_flags &= SCHED_FLAG_ALL;
 
-	kattr.sched_latency_nice = p->latency_nice;
+	kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio);
 
 #ifdef CONFIG_UCLAMP_TASK
 	/*
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 68be7a3e42a3..b3922184af91 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1043,7 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 #endif
 	P(policy);
 	P(prio);
-	P(latency_nice);
+	P(latency_prio);
 	if (task_has_dl_policy(p)) {
 		P(dl.runtime);
 		P(dl.deadline);
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 81bef11eb660..414b6243208b 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4877,6 +4877,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 		update_idle_cfs_rq_clock_pelt(cfs_rq);
 }
 
+static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se);
+
 /*
  * Preempt the current task with a newly woken task if needed:
  */
@@ -4885,7 +4887,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 {
 	unsigned long ideal_runtime, delta_exec;
 	struct sched_entity *se;
-	s64 delta;
+	s64 delta, offset;
 
 	/*
 	 * When many tasks blow up the sched_period; it is possible that
@@ -4916,10 +4918,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 	se = __pick_first_entity(cfs_rq);
 	delta = curr->vruntime - se->vruntime;
 
-	if (delta < 0)
+	offset = wakeup_latency_gran(curr, se);
+	if (delta < offset)
 		return;
 
-	if (delta > ideal_runtime)
+	if ((delta > ideal_runtime) ||
+	    (delta > get_latency_max()))
 		resched_curr(rq_of(cfs_rq));
 }
 
@@ -7662,6 +7666,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 }
 #endif /* CONFIG_SMP */
 
+static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se)
+{
+	long latency_offset = se->latency_offset;
+
+	/*
+	 * A negative latency offset means that the sched_entity has latency
+	 * requirement that needs to be evaluated versus other entity.
+	 * Otherwise, use the latency weight to evaluate how much scheduling
+	 * delay is acceptable by se.
+	 */
+	if ((latency_offset < 0) || (curr->latency_offset < 0))
+		latency_offset -= curr->latency_offset;
+	latency_offset = min_t(long, latency_offset, get_latency_max());
+
+	return latency_offset;
+}
+
 static unsigned long wakeup_gran(struct sched_entity *se)
 {
 	unsigned long gran = sysctl_sched_wakeup_granularity;
@@ -7700,11 +7721,12 @@ static int
 wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
 {
 	s64 gran, vdiff = curr->vruntime - se->vruntime;
+	s64 offset = wakeup_latency_gran(curr, se);
 
-	if (vdiff <= 0)
+	if (vdiff < offset)
 		return -1;
 
-	gran = wakeup_gran(se);
+	gran = offset + wakeup_gran(se);
 
 	/*
 	 * At wake up, the vruntime of a task is capped to not be older than
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 51ba0af7fb27..3f42f86105d4 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2494,6 +2494,17 @@ static inline unsigned long get_sleep_latency(bool idle)
 	return thresh;
 }
 
+/*
+ * Calculate the latency offset for a priority level.
+ * We use a linear mapping of the priority in the range:
+ *     [-sysctl_sched_latency:sysctl_sched_latency]
+ */
+static inline long calc_latency_offset(int prio)
+{
+	return (long)get_sleep_latency(false) * LATENCY_TO_NICE(prio) /
+			(LATENCY_NICE_WIDTH/2);
+}
+
 static inline unsigned long get_latency_max(void)
 {
 	unsigned long thresh = get_sleep_latency(false);
-- 
2.34.1

[PATCH v12 6/8] sched/fair: Add sched group latency support

[Vincent Guittot]

Task can set its latency priority with sched_setattr(), which is then used
to set the latency offset of its sched_enity, but sched group entities
still have the default latency offset value.

Add a latency.nice field in cpu cgroup controller to set the latency
priority of the group similarly to sched_setattr(). The latency priority
is then used to set the offset of the sched_entities of the group.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 Documentation/admin-guide/cgroup-v2.rst | 10 ++++++++
 kernel/sched/core.c                     | 30 +++++++++++++++++++++++
 kernel/sched/fair.c                     | 32 +++++++++++++++++++++++++
 kernel/sched/sched.h                    |  4 ++++
 4 files changed, 76 insertions(+)

diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 1b3ed1c3b3f1..c08424593e4a 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1121,6 +1121,16 @@ All time durations are in microseconds.
         values similar to the sched_setattr(2). This maximum utilization
         value is used to clamp the task specific maximum utilization clamp.
 
+  cpu.latency.nice
+	A read-write single value file which exists on non-root
+	cgroups.  The default is "0".
+
+	The nice value is in the range [-20, 19].
+
+	This interface file allows reading and setting latency using the
+	same values used by sched_setattr(2). The latency_nice of a group is
+	used to limit the impact of the latency_nice of a task outside the
+	group.
 
 
 Memory
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index d5b7e237d79b..093cc1af73dc 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -11059,6 +11059,25 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,
 {
 	return sched_group_set_idle(css_tg(css), idle);
 }
+
+static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css,
+				    struct cftype *cft)
+{
+	return LATENCY_TO_NICE(css_tg(css)->latency_prio);
+}
+
+static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css,
+				     struct cftype *cft, s64 nice)
+{
+	int prio;
+
+	if (nice < MIN_LATENCY_NICE || nice > MAX_LATENCY_NICE)
+		return -ERANGE;
+
+	prio = NICE_TO_LATENCY(nice);
+
+	return sched_group_set_latency(css_tg(css), prio);
+}
 #endif
 
 static struct cftype cpu_legacy_files[] = {
@@ -11073,6 +11092,11 @@ static struct cftype cpu_legacy_files[] = {
 		.read_s64 = cpu_idle_read_s64,
 		.write_s64 = cpu_idle_write_s64,
 	},
+	{
+		.name = "latency.nice",
+		.read_s64 = cpu_latency_nice_read_s64,
+		.write_s64 = cpu_latency_nice_write_s64,
+	},
 #endif
 #ifdef CONFIG_CFS_BANDWIDTH
 	{
@@ -11290,6 +11314,12 @@ static struct cftype cpu_files[] = {
 		.read_s64 = cpu_idle_read_s64,
 		.write_s64 = cpu_idle_write_s64,
 	},
+	{
+		.name = "latency.nice",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.read_s64 = cpu_latency_nice_read_s64,
+		.write_s64 = cpu_latency_nice_write_s64,
+	},
 #endif
 #ifdef CONFIG_CFS_BANDWIDTH
 	{
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 414b6243208b..dc7570f43ebe 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12274,6 +12274,7 @@ int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
 		goto err;
 
 	tg->shares = NICE_0_LOAD;
+	tg->latency_prio = DEFAULT_LATENCY_PRIO;
 
 	init_cfs_bandwidth(tg_cfs_bandwidth(tg));
 
@@ -12372,6 +12373,9 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
 	}
 
 	se->my_q = cfs_rq;
+
+	se->latency_offset = calc_latency_offset(tg->latency_prio);
+
 	/* guarantee group entities always have weight */
 	update_load_set(&se->load, NICE_0_LOAD);
 	se->parent = parent;
@@ -12502,6 +12506,34 @@ int sched_group_set_idle(struct task_group *tg, long idle)
 	return 0;
 }
 
+int sched_group_set_latency(struct task_group *tg, int prio)
+{
+	long latency_offset;
+	int i;
+
+	if (tg == &root_task_group)
+		return -EINVAL;
+
+	mutex_lock(&shares_mutex);
+
+	if (tg->latency_prio == prio) {
+		mutex_unlock(&shares_mutex);
+		return 0;
+	}
+
+	tg->latency_prio = prio;
+	latency_offset = calc_latency_offset(prio);
+
+	for_each_possible_cpu(i) {
+		struct sched_entity *se = tg->se[i];
+
+		WRITE_ONCE(se->latency_offset, latency_offset);
+	}
+
+	mutex_unlock(&shares_mutex);
+	return 0;
+}
+
 #else /* CONFIG_FAIR_GROUP_SCHED */
 
 void free_fair_sched_group(struct task_group *tg) { }
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 3f42f86105d4..9a2e71231083 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -378,6 +378,8 @@ struct task_group {
 
 	/* A positive value indicates that this is a SCHED_IDLE group. */
 	int			idle;
+	/* latency priority of the group. */
+	int			latency_prio;
 
 #ifdef	CONFIG_SMP
 	/*
@@ -488,6 +490,8 @@ extern int sched_group_set_shares(struct task_group *tg, unsigned long shares);
 
 extern int sched_group_set_idle(struct task_group *tg, long idle);
 
+extern int sched_group_set_latency(struct task_group *tg, int prio);
+
 #ifdef CONFIG_SMP
 extern void set_task_rq_fair(struct sched_entity *se,
 			     struct cfs_rq *prev, struct cfs_rq *next);
-- 
2.34.1

[PATCH v12 7/8] sched/core: Support latency priority with sched core

[Vincent Guittot]

Take into account wakeup_latency_gran() when ordering the cfs threads.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 kernel/sched/fair.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index dc7570f43ebe..125a6ff53378 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -11949,6 +11949,9 @@ bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b,
 	delta = (s64)(sea->vruntime - seb->vruntime) +
 		(s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi);
 
+	/* Take into account latency offset */
+	delta -= wakeup_latency_gran(sea, seb);
+
 	return delta > 0;
 }
 #else
-- 
2.34.1

[PATCH v12 8/8] sched/fair: Add latency list

[Vincent Guittot]

Add a rb tree for latency sensitive entities so we can schedule the most
sensitive one first even when it failed to preempt current at wakeup or
when it got quickly preempted by another entity of higher priority.

In order to keep fairness, the latency is used once at wakeup to get a
minimum slice and not during the following scheduling slice to prevent
long running entity to got more running time than allocated to his nice
priority.

The rb tree enables to cover the last corner case where latency
sensitive entity can't got schedule quickly after the wakeup.

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 include/linux/sched.h |   1 +
 kernel/sched/core.c   |   1 +
 kernel/sched/fair.c   | 109 ++++++++++++++++++++++++++++++++++++++++--
 kernel/sched/sched.h  |   1 +
 4 files changed, 109 insertions(+), 3 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 38decae3e156..41bb92be5ecc 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -548,6 +548,7 @@ struct sched_entity {
 	/* For load-balancing: */
 	struct load_weight		load;
 	struct rb_node			run_node;
+	struct rb_node			latency_node;
 	struct list_head		group_node;
 	unsigned int			on_rq;
 
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 093cc1af73dc..752fd364216c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 	p->se.nr_migrations		= 0;
 	p->se.vruntime			= 0;
 	INIT_LIST_HEAD(&p->se.group_node);
+	RB_CLEAR_NODE(&p->se.latency_node);
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	p->se.cfs_rq			= NULL;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 125a6ff53378..e2aeb4511686 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
 
 	return __node_2_se(last);
 }
+#endif
 
+/**************************************************************
+ * Scheduling class tree data structure manipulation methods:
+ * for latency
+ */
+
+static inline bool latency_before(struct sched_entity *a,
+				struct sched_entity *b)
+{
+	return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
+}
+
+#define __latency_node_2_se(node) \
+	rb_entry((node), struct sched_entity, latency_node)
+
+static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
+{
+	return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
+}
+
+/*
+ * Enqueue an entity into the latency rb-tree:
+ */
+static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
+{
+
+	/* Only latency sensitive entity can be added to the list */
+	if (se->latency_offset >= 0)
+		return;
+
+	if (!RB_EMPTY_NODE(&se->latency_node))
+		return;
+
+	/*
+	 * The entity is always added the latency list at wakeup.
+	 * Then, a not waking up entity that is put back in the list after an
+	 * execution time less than sysctl_sched_min_granularity, means that
+	 * the entity has been preempted by a higher sched class or an entity
+	 * with higher latency constraint. In thi case, the entity is also put
+	 * back in the latency list so it gets a chance to run 1st during the
+	 * next slice.
+	 */
+	if (!(flags & ENQUEUE_WAKEUP)) {
+		u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
+
+		if (delta_exec >= sysctl_sched_min_granularity)
+			return;
+	}
+
+	rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
+}
+
+/*
+ * Dequeue an entity from the latency rb-tree and return true if it was really
+ * part of the rb-tree:
+ */
+static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
+{
+	if (!RB_EMPTY_NODE(&se->latency_node)) {
+		rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
+		RB_CLEAR_NODE(&se->latency_node);
+		return true;
+	}
+
+	return false;
+}
+
+static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
+{
+	struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
+
+	if (!left)
+		return NULL;
+
+	return __latency_node_2_se(left);
+}
+
+#ifdef CONFIG_SCHED_DEBUG
 /**************************************************************
  * Scheduling class statistics methods:
  */
@@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	check_schedstat_required();
 	update_stats_enqueue_fair(cfs_rq, se, flags);
 	check_spread(cfs_rq, se);
-	if (!curr)
+	if (!curr) {
 		__enqueue_entity(cfs_rq, se);
+		__enqueue_latency(cfs_rq, se, flags);
+	}
 	se->on_rq = 1;
 
 	if (cfs_rq->nr_running == 1) {
@@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
 	clear_buddies(cfs_rq, se);
 
-	if (se != cfs_rq->curr)
+	if (se != cfs_rq->curr) {
 		__dequeue_entity(cfs_rq, se);
+		__dequeue_latency(cfs_rq, se);
+	}
 	se->on_rq = 0;
 	account_entity_dequeue(cfs_rq, se);
 
@@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 		 */
 		update_stats_wait_end_fair(cfs_rq, se);
 		__dequeue_entity(cfs_rq, se);
+		__dequeue_latency(cfs_rq, se);
 		update_load_avg(cfs_rq, se, UPDATE_TG);
 	}
 
@@ -4979,7 +5062,7 @@ static struct sched_entity *
 pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 {
 	struct sched_entity *left = __pick_first_entity(cfs_rq);
-	struct sched_entity *se;
+	struct sched_entity *latency, *se;
 
 	/*
 	 * If curr is set we have to see if its left of the leftmost entity
@@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 		se = cfs_rq->last;
 	}
 
+	/* Check for latency sensitive entity waiting for running */
+	latency = __pick_first_latency(cfs_rq);
+	if (latency && (latency != se) &&
+	    wakeup_preempt_entity(latency, se) < 1)
+		se = latency;
+
 	return se;
 }
 
@@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
 		update_stats_wait_start_fair(cfs_rq, prev);
 		/* Put 'current' back into the tree. */
 		__enqueue_entity(cfs_rq, prev);
+		__enqueue_latency(cfs_rq, prev, 0);
 		/* in !on_rq case, update occurred at dequeue */
 		update_load_avg(cfs_rq, prev, 0);
 	}
@@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 void init_cfs_rq(struct cfs_rq *cfs_rq)
 {
 	cfs_rq->tasks_timeline = RB_ROOT_CACHED;
+	cfs_rq->latency_timeline = RB_ROOT_CACHED;
 	u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
 #ifdef CONFIG_SMP
 	raw_spin_lock_init(&cfs_rq->removed.lock);
@@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
 	se->my_q = cfs_rq;
 
 	se->latency_offset = calc_latency_offset(tg->latency_prio);
+	RB_CLEAR_NODE(&se->latency_node);
 
 	/* guarantee group entities always have weight */
 	update_load_set(&se->load, NICE_0_LOAD);
@@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
 
 	for_each_possible_cpu(i) {
 		struct sched_entity *se = tg->se[i];
+		struct rq *rq = cpu_rq(i);
+		struct rq_flags rf;
+		bool queued;
+
+		rq_lock_irqsave(rq, &rf);
 
+		queued = __dequeue_latency(se->cfs_rq, se);
 		WRITE_ONCE(se->latency_offset, latency_offset);
+		if (queued)
+			__enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
+
+
+		rq_unlock_irqrestore(rq, &rf);
 	}
 
 	mutex_unlock(&shares_mutex);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 9a2e71231083..21dd309e98a9 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -570,6 +570,7 @@ struct cfs_rq {
 #endif
 
 	struct rb_root_cached	tasks_timeline;
+	struct rb_root_cached	latency_timeline;
 
 	/*
 	 * 'curr' points to currently running entity on this cfs_rq.
-- 
2.34.1

Re: [PATCH v12 6/8] sched/fair: Add sched group latency support

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 1367 bytes --]

Hello Vincent.

On Fri, Feb 24, 2023 at 10:34:52AM +0100, Vincent Guittot <vincent.guittot@linaro.org> wrote:
> +  cpu.latency.nice
> +	A read-write single value file which exists on non-root
> +	cgroups.  The default is "0".
> +
> +	The nice value is in the range [-20, 19].
> +
> +	This interface file allows reading and setting latency using the
> +	same values used by sched_setattr(2). The latency_nice of a group is
> +	used to limit the impact of the latency_nice of a task outside the
> +	group.

IIUC, the latency priority is taken into account when deciding between
entitites at the same level (as in pick_next_entity() or
check_preempt_wake()/find_matchig_se()).

So this group attribute is relevant in context of siblings (i.e. like
cpu.weight ~ bandwidth priority)?

I'm thus confused when it's referred to as a limit (in vertical sense).
You somewhat imply that in [1]:

> Regarding the behavior, the rule remains the same that a sched_entity
> attached to a cgroup will not get more (latency in this case) than
> what has been set for the group entity.

But I don't see where such a constraint would be implemented in the
code. (My cursory understanding above tends to horizontal comparisons.)

Could you please hint me which is right?

Thanks,
Michal

[1] https://lore.kernel.org/r/CAKfTPtDu=c-psGnHkoWSPRWoh1Z0VBBfsN++g+krv4B1SJmFjg@mail.gmail.com/


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v12 6/8] sched/fair: Add sched group latency support

[Vincent Guittot]

On Fri, 24 Feb 2023 at 20:29, Michal Koutný <mkoutny@suse.com> wrote:
>
> Hello Vincent.
>
> On Fri, Feb 24, 2023 at 10:34:52AM +0100, Vincent Guittot <vincent.guittot@linaro.org> wrote:
> > +  cpu.latency.nice
> > +     A read-write single value file which exists on non-root
> > +     cgroups.  The default is "0".
> > +
> > +     The nice value is in the range [-20, 19].
> > +
> > +     This interface file allows reading and setting latency using the
> > +     same values used by sched_setattr(2). The latency_nice of a group is
> > +     used to limit the impact of the latency_nice of a task outside the
> > +     group.
>
> IIUC, the latency priority is taken into account when deciding between
> entitites at the same level (as in pick_next_entity() or
> check_preempt_wake()/find_matchig_se()).
>
> So this group attribute is relevant in context of siblings (i.e. like
> cpu.weight ~ bandwidth priority)?

Yes

>
> I'm thus confused when it's referred to as a limit (in vertical sense).
> You somewhat imply that in [1]:

There were discussions about adding more features that could make use
of the latency nice. This comment mainly wants to describe how this
would behave in case that we need to compre entities/tasks not at the
same level.

Regarding the current use of latency nice to set a latency offset, the
problem doesn't appear because latency offset applies between entities
at the same level as you mentioned above

>
> > Regarding the behavior, the rule remains the same that a sched_entity
> > attached to a cgroup will not get more (latency in this case) than
> > what has been set for the group entity.
>
> But I don't see where such a constraint would be implemented in the
> code. (My cursory understanding above tends to horizontal comparisons.)
>
> Could you please hint me which is right?

Does  my explanation above make sense to you ?

>
> Thanks,
> Michal
>
> [1] https://lore.kernel.org/r/CAKfTPtDu=c-psGnHkoWSPRWoh1Z0VBBfsN++g+krv4B1SJmFjg@mail.gmail.com/
>

Re: [PATCH v12 6/8] sched/fair: Add sched group latency support

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Mon, Feb 27, 2023 at 02:44:22PM +0100, Vincent Guittot <vincent.guittot@linaro.org> wrote:
> Regarding the current use of latency nice to set a latency offset, the
> problem doesn't appear because latency offset applies between entities
> at the same level as you mentioned above

Splendid, it turned out that way (latency nice analogous to bandwidth
nice).

> Does  my explanation above make sense to you ?

Yes, thank you.

Thus, I'd like to propose avoiding the use of "limit" in this context and
stress the horizontal scope. For example:

> +     This interface file allows reading and setting latency using the
> +     same values used by sched_setattr(2). The latency_nice of a group is
> +     used to limit the impact of the latency_nice of a task outside the
> +     group.

+     This interface file allows reading and setting latency using the
+     same values used by sched_setattr(2). The latency_nice of a group is
+     used to modify group members' latency with respect to sibling groups.

Regards,
Michal

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v12 6/8] sched/fair: Add sched group latency support

[Vincent Guittot]

On Mon, 27 Feb 2023 at 15:42, Michal Koutný <mkoutny@suse.com> wrote:
>
> On Mon, Feb 27, 2023 at 02:44:22PM +0100, Vincent Guittot <vincent.guittot@linaro.org> wrote:
> > Regarding the current use of latency nice to set a latency offset, the
> > problem doesn't appear because latency offset applies between entities
> > at the same level as you mentioned above
>
> Splendid, it turned out that way (latency nice analogous to bandwidth
> nice).
>
> > Does  my explanation above make sense to you ?
>
> Yes, thank you.
>
> Thus, I'd like to propose avoiding the use of "limit" in this context and
> stress the horizontal scope. For example:
>
> > +     This interface file allows reading and setting latency using the
> > +     same values used by sched_setattr(2). The latency_nice of a group is
> > +     used to limit the impact of the latency_nice of a task outside the
> > +     group.
>
> +     This interface file allows reading and setting latency using the
> +     same values used by sched_setattr(2). The latency_nice of a group is
> +     used to modify group members' latency with respect to sibling groups.

That sounds reasonable to me.

>
> Regards,
> Michal

Re: [PATCH v12 8/8] sched/fair: Add latency list

[shrikanth hegde]

On 2/24/23 3:04 PM, Vincent Guittot wrote:
> Add a rb tree for latency sensitive entities so we can schedule the most
> sensitive one first even when it failed to preempt current at wakeup or
> when it got quickly preempted by another entity of higher priority.
>
> In order to keep fairness, the latency is used once at wakeup to get a
> minimum slice and not during the following scheduling slice to prevent
> long running entity to got more running time than allocated to his nice
> priority.
>
> The rb tree enables to cover the last corner case where latency
> sensitive entity can't got schedule quickly after the wakeup.
>
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> ---
>  include/linux/sched.h |   1 +
>  kernel/sched/core.c   |   1 +
>  kernel/sched/fair.c   | 109 ++++++++++++++++++++++++++++++++++++++++--
>  kernel/sched/sched.h  |   1 +
>  4 files changed, 109 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 38decae3e156..41bb92be5ecc 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -548,6 +548,7 @@ struct sched_entity {
>  	/* For load-balancing: */
>  	struct load_weight		load;
>  	struct rb_node			run_node;
> +	struct rb_node			latency_node;

Ran pahole to see if the frequently accessed fields change across cachelines.
There is no change in cachelines of task_struct, whereas sched_entity differs
due to latency_node.  Maybe the latency_node could be placed after
runnable_weight as there is space available in that cacheline.


6.2
#pahole sched_entity
struct sched_entity {
	struct load_weight         load;                 /*     0    16 */

	/* XXX last struct has 4 bytes of padding */

	struct rb_node             run_node;             /*    16    24 */
	struct list_head           group_node;           /*    40    16 */
	unsigned int               on_rq;                /*    56     4 */

	/* XXX 4 bytes hole, try to pack */

	u64                        exec_start;           /*    64     8 */
	u64                        sum_exec_runtime;     /*    72     8 */
	u64                        vruntime;             /*    80     8 */
	u64                        prev_sum_exec_runtime; /*    88     8 */
	u64                        nr_migrations;        /*    96     8 */
	int                        depth;                /*   104     4 */

	/* XXX 4 bytes hole, try to pack */

	struct sched_entity *      parent;               /*   112     8 */
	struct cfs_rq *            cfs_rq;               /*   120     8 */
	/* --- cacheline 1 boundary (128 bytes) --- */
	struct cfs_rq *            my_q;                 /*   128     8 */
	long unsigned int          runnable_weight;      /*   136     8 */

	/* XXX 112 bytes hole, try to pack */

	/* --- cacheline 2 boundary (256 bytes) --- */
	struct sched_avg           avg;                  /*   256   128 */


6.2 + V12 patch
#pahole sched_entity
struct sched_entity {
	struct load_weight         load;                 /*     0    16 */

	/* XXX last struct has 4 bytes of padding */

	struct rb_node             run_node;             /*    16    24 */
	struct rb_node             latency_node;         /*    40    24 */
	struct list_head           group_node;           /*    64    16 */
	unsigned int               on_rq;                /*    80     4 */

	/* XXX 4 bytes hole, try to pack */

	u64                        exec_start;           /*    88     8 */
	u64                        sum_exec_runtime;     /*    96     8 */
	u64                        vruntime;             /*   104     8 */
	u64                        prev_sum_exec_runtime; /*   112     8 */
	u64                        nr_migrations;        /*   120     8 */
	/* --- cacheline 1 boundary (128 bytes) --- */
	int                        depth;                /*   128     4 */

	/* XXX 4 bytes hole, try to pack */

	struct sched_entity *      parent;               /*   136     8 */
	struct cfs_rq *            cfs_rq;               /*   144     8 */
	struct cfs_rq *            my_q;                 /*   152     8 */


6.2 + V12 patch + Re-shuffle of latency_node
#pahole sched_entity
struct sched_entity {
	struct load_weight         load;                 /*     0    16 */

	/* XXX last struct has 4 bytes of padding */

	struct rb_node             run_node;             /*    16    24 */
	struct list_head           group_node;           /*    40    16 */
	unsigned int               on_rq;                /*    56     4 */

	/* XXX 4 bytes hole, try to pack */

	u64                        exec_start;           /*    64     8 */
	u64                        sum_exec_runtime;     /*    72     8 */
	u64                        vruntime;             /*    80     8 */
	u64                        prev_sum_exec_runtime; /*    88     8 */
	u64                        nr_migrations;        /*    96     8 */
	int                        depth;                /*   104     4 */

	/* XXX 4 bytes hole, try to pack */

	struct sched_entity *      parent;               /*   112     8 */
	struct cfs_rq *            cfs_rq;               /*   120     8 */
	/* --- cacheline 1 boundary (128 bytes) --- */
	struct cfs_rq *            my_q;                 /*   128     8 */
	long unsigned int          runnable_weight;      /*   136     8 */
	struct rb_node             latency_node;         /*   144    24 */
	long int                   latency_offset;       /*   168     8 */

	/* XXX 80 bytes hole, try to pack */

	/* --- cacheline 2 boundary (256 bytes) --- */



diff --git a/include/linux/sched.h b/include/linux/sched.h
index a2b52cf5e1bb..1e93aaaeead2 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -548,7 +548,6 @@ struct sched_entity {
        /* For load-balancing: */
        struct load_weight              load;
        struct rb_node                  run_node;
-       struct rb_node                  latency_node;
        struct list_head                group_node;
        unsigned int                    on_rq;

@@ -569,6 +568,7 @@ struct sched_entity {
        /* cached value of my_q->h_nr_running */
        unsigned long                   runnable_weight;
 #endif
+       struct rb_node                  latency_node;
        /* preemption offset in ns */
        long                            latency_offset;


Ran the schbench and hackbench with this patch series. Here comparison is
between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
and other is running these benchmarks (group2). Set the latency nice
of group2 to -20. These are run on Power system with 12 cores with SMT=8.
Total of 96 CPU.

schbench gets lower latency compared to stabletree. Whereas hackbench seems
to regress under this case. Maybe i am doing something wrong. I will re-run
and attach the numbers to series. 
Please suggest if any variation in the test i need to try.

Re-arrange seems to help the patch series by avoiding an cacheline miss.

=========================
schbench
=========================
		 6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
1 Thread
  50.0th:	 9.00  |    9.00	|	 9.50
  75.0th:	10.50  |   10.00	|	 9.50
  90.0th:	11.00  |   11.00	|	10.50
  95.0th:	11.00  |   11.00	|	11.00
  99.0th:	11.50  |   11.50	|	11.50
  99.5th:	12.50  |   12.00	|	12.00
  99.9th:	14.50  |   13.50	|	12.00
2 Threads
  50.0th:	 9.50  |    9.50	|	 8.50
  75.0th:	11.00  |   10.50	|	 9.50
  90.0th:	13.50  |   11.50	|	10.50
  95.0th:	14.00  |   12.00	|	11.00
  99.0th:	15.50  |   13.50	|	12.00
  99.5th:	16.00  |   14.00	|	12.00
  99.9th:	17.00  |   16.00	|	16.50
4 Threads
  50.0th:	11.50  |   11.50        |	10.50
  75.0th:	13.50  |   12.50        |	12.50
  90.0th:	15.50  |   14.50        |	14.00
  95.0th:	16.50  |   15.50        |	14.50
  99.0th:	20.00  |   17.50        |	16.50
  99.5th:	20.50  |   18.50        |	17.00
  99.9th:	22.50  |   21.00        |	19.00
8 Threads
  50.0th:	14.00  |   14.00        |	14.00
  75.0th:	16.00  |   16.00        |	16.00
  90.0th:	18.00  |   18.00        |	17.50
  95.0th:	18.50  |   18.50        |	18.50
  99.0th:	20.00  |   20.00        |	20.00
  99.5th:	20.50  |   21.50        |	21.00
  99.9th:	22.50  |   23.50        |	23.00
16 Threads
  50.0th:	19.00  |   18.50        |	19.00
  75.0th:	23.00  |   22.50        |	23.00
  90.0th:	25.00  |   25.50        |	25.00
  95.0th:	26.50  |   26.50        |	26.00
  99.0th:	28.50  |   29.00        |	28.50
  99.5th:	31.00  |   30.00        |	30.00
  99.9th:     5626.00  | 4761.50        |	32.50
32 Threads
  50.0th:	27.00  |   27.50        |	29.00
  75.0th:	35.50  |   36.50        |	38.50
  90.0th:	42.00  |   44.00        |	50.50
  95.0th:      447.50  | 2959.00        |     8544.00
  99.0th:     7372.00  | 17032.00       |    19136.00
  99.5th:    15360.00  | 19808.00       |    20704.00
  99.9th:    20640.00  | 30048.00       |    30048.00

====================
hackbench
====================
			6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange

Process 10 Time:	0.35    |	0.42        |		0.41
Process 20 Time:	0.61    |	0.76        |		0.76
Process 30 Time:	0.87    |	1.06        |		1.05
thread 10 Time:		0.35    |	0.43        |		0.42
thread 20 Time:		0.66    |	0.79        |		0.78
Process(Pipe) 10 Time:	0.21    |	0.33        |		0.32
Process(Pipe) 20 Time:	0.34    |	0.52        |		0.52
Process(Pipe) 30 Time:	0.46    |	0.72        |		0.71
thread(Pipe) 10 Time:	0.21    |	0.34        |		0.34
thread(Pipe) 20 Time:	0.36    |	0.56        |		0.56


>  	struct list_head		group_node;
>  	unsigned int			on_rq;
>  
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 093cc1af73dc..752fd364216c 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
>  	p->se.nr_migrations		= 0;
>  	p->se.vruntime			= 0;
>  	INIT_LIST_HEAD(&p->se.group_node);
> +	RB_CLEAR_NODE(&p->se.latency_node);
>  
>  #ifdef CONFIG_FAIR_GROUP_SCHED
>  	p->se.cfs_rq			= NULL;
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 125a6ff53378..e2aeb4511686 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
>  
>  	return __node_2_se(last);
>  }
> +#endif
>  
> +/**************************************************************
> + * Scheduling class tree data structure manipulation methods:
> + * for latency
> + */
> +
> +static inline bool latency_before(struct sched_entity *a,
> +				struct sched_entity *b)
> +{
> +	return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> +}
> +
> +#define __latency_node_2_se(node) \
> +	rb_entry((node), struct sched_entity, latency_node)
> +
> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> +{
> +	return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> +}
> +
> +/*
> + * Enqueue an entity into the latency rb-tree:
> + */
> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> +{
> +
> +	/* Only latency sensitive entity can be added to the list */
> +	if (se->latency_offset >= 0)
> +		return;
> +
> +	if (!RB_EMPTY_NODE(&se->latency_node))
> +		return;
> +
> +	/*
> +	 * The entity is always added the latency list at wakeup.
> +	 * Then, a not waking up entity that is put back in the list after an
> +	 * execution time less than sysctl_sched_min_granularity, means that
> +	 * the entity has been preempted by a higher sched class or an entity
> +	 * with higher latency constraint. In thi case, the entity is also put
> +	 * back in the latency list so it gets a chance to run 1st during the
> +	 * next slice.
> +	 */
> +	if (!(flags & ENQUEUE_WAKEUP)) {
> +		u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> +
> +		if (delta_exec >= sysctl_sched_min_granularity)
> +			return;
> +	}
> +
> +	rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> +}
> +
> +/*
> + * Dequeue an entity from the latency rb-tree and return true if it was really
> + * part of the rb-tree:
> + */
> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> +{
> +	if (!RB_EMPTY_NODE(&se->latency_node)) {
> +		rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> +		RB_CLEAR_NODE(&se->latency_node);
> +		return true;
> +	}
> +
> +	return false;
> +}
> +
> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> +{
> +	struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> +
> +	if (!left)
> +		return NULL;
> +
> +	return __latency_node_2_se(left);
> +}
> +
> +#ifdef CONFIG_SCHED_DEBUG
>  /**************************************************************
>   * Scheduling class statistics methods:
>   */
> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>  	check_schedstat_required();
>  	update_stats_enqueue_fair(cfs_rq, se, flags);
>  	check_spread(cfs_rq, se);
> -	if (!curr)
> +	if (!curr) {
>  		__enqueue_entity(cfs_rq, se);
> +		__enqueue_latency(cfs_rq, se, flags);
> +	}
>  	se->on_rq = 1;
>  
>  	if (cfs_rq->nr_running == 1) {
> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>  
>  	clear_buddies(cfs_rq, se);
>  
> -	if (se != cfs_rq->curr)
> +	if (se != cfs_rq->curr) {
>  		__dequeue_entity(cfs_rq, se);
> +		__dequeue_latency(cfs_rq, se);
> +	}
>  	se->on_rq = 0;
>  	account_entity_dequeue(cfs_rq, se);
>  
> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>  		 */
>  		update_stats_wait_end_fair(cfs_rq, se);
>  		__dequeue_entity(cfs_rq, se);
> +		__dequeue_latency(cfs_rq, se);
>  		update_load_avg(cfs_rq, se, UPDATE_TG);
>  	}
>  
> @@ -4979,7 +5062,7 @@ static struct sched_entity *
>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>  {
>  	struct sched_entity *left = __pick_first_entity(cfs_rq);
> -	struct sched_entity *se;
> +	struct sched_entity *latency, *se;
>  
>  	/*
>  	 * If curr is set we have to see if its left of the leftmost entity
> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>  		se = cfs_rq->last;
>  	}
>  
> +	/* Check for latency sensitive entity waiting for running */
> +	latency = __pick_first_latency(cfs_rq);
> +	if (latency && (latency != se) &&
> +	    wakeup_preempt_entity(latency, se) < 1)
> +		se = latency;
> +
>  	return se;
>  }
>  
> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
>  		update_stats_wait_start_fair(cfs_rq, prev);
>  		/* Put 'current' back into the tree. */
>  		__enqueue_entity(cfs_rq, prev);
> +		__enqueue_latency(cfs_rq, prev, 0);
>  		/* in !on_rq case, update occurred at dequeue */
>  		update_load_avg(cfs_rq, prev, 0);
>  	}
> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>  void init_cfs_rq(struct cfs_rq *cfs_rq)
>  {
>  	cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> +	cfs_rq->latency_timeline = RB_ROOT_CACHED;
>  	u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
>  #ifdef CONFIG_SMP
>  	raw_spin_lock_init(&cfs_rq->removed.lock);
> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
>  	se->my_q = cfs_rq;
>  
>  	se->latency_offset = calc_latency_offset(tg->latency_prio);
> +	RB_CLEAR_NODE(&se->latency_node);
>  
>  	/* guarantee group entities always have weight */
>  	update_load_set(&se->load, NICE_0_LOAD);
> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
>  
>  	for_each_possible_cpu(i) {
>  		struct sched_entity *se = tg->se[i];
> +		struct rq *rq = cpu_rq(i);
> +		struct rq_flags rf;
> +		bool queued;
> +
> +		rq_lock_irqsave(rq, &rf);
>  
> +		queued = __dequeue_latency(se->cfs_rq, se);
>  		WRITE_ONCE(se->latency_offset, latency_offset);
> +		if (queued)
> +			__enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
> +
> +
> +		rq_unlock_irqrestore(rq, &rf);
>  	}
>  
>  	mutex_unlock(&shares_mutex);
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 9a2e71231083..21dd309e98a9 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -570,6 +570,7 @@ struct cfs_rq {
>  #endif
>  
>  	struct rb_root_cached	tasks_timeline;
> +	struct rb_root_cached	latency_timeline;
>  
>  	/*
>  	 * 'curr' points to currently running entity on this cfs_rq.

Re: [PATCH v12 5/8] sched/fair: Take into account latency priority at wakeup

[shrikanth hegde]

On 2/24/23 3:04 PM, Vincent Guittot wrote:
> Take into account the latency priority of a thread when deciding to
> preempt the current running thread. We don't want to provide more CPU
> bandwidth to a thread but reorder the scheduling to run latency sensitive
> task first whenever possible.
>
> As long as a thread didn't use its bandwidth, it will be able to preempt
> the current thread.
>
> At the opposite, a thread with a low latency priority will preempt current
> thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will
> wait for the tick to get its sched slice.
>
>                                    curr vruntime
>                                        |
>                       sysctl_sched_wakeup_granularity
>                                    <-->
> ----------------------------------|----|-----------------------|---------------
>                                   |    |<--------------------->
>                                   |    .  sysctl_sched_latency
>                                   |    .
> default/current latency entity    |    .
>                                   |    .
> 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-
> se preempts curr at wakeup ------>|<- se doesn't preempt curr -----------------
>                                   |    .
>                                   |    .
>                                   |    .
> low latency entity                |    .
>                                    ---------------------->|
>                                % of sysctl_sched_latency  |
> 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-
> preempt ------------------------------------------------->|<- do not preempt --
>                                   |    .
>                                   |    .
>                                   |    .
> high latency entity               |    .
>          |<-----------------------|----.
>          | % of sysctl_sched_latency   .
> 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1
> preempt->|<- se doesn't preempt curr ------------------------------------------
>
> Tests results of nice latency impact on heavy load like hackbench:
>
> hackbench -l (2560 / group) -g group
> group        latency 0             latency 19
> 1            1.378(+/-  1%)      1.337(+/- 1%) + 3%
> 4            1.393(+/-  3%)      1.312(+/- 3%) + 6%
> 8            1.308(+/-  2%)      1.279(+/- 1%) + 2%
> 16           1.347(+/-  1%)      1.317(+/- 1%) + 2%
>
> hackbench -p -l (2560 / group) -g group
> group
> 1            1.836(+/- 17%)      1.148(+/- 5%) +37%
> 4            1.586(+/-  6%)      1.109(+/- 8%) +30%
> 8            1.209(+/-  4%)      0.780(+/- 4%) +35%
> 16           0.805(+/-  5%)      0.728(+/- 4%) +10%
>
> By deacreasing the latency prio, we reduce the number of preemption at
> wakeup and help hackbench making progress.
>
> Test results of nice latency impact on short live load like cyclictest
> while competing with heavy load like hackbench:
>
> hackbench -l 10000 -g $group &
> cyclictest --policy other -D 5 -q -n
>         latency 0           latency -20
> group   min  avg    max     min  avg    max
> 0       16    19     29      17   18     29
> 1       43   299   7359      63   84   3422
> 4       56   449  14806      45   83    284
> 8       63   820  51123      63   83    283
> 16      64  1326  70684      41  157  26852
>
> group = 0 means that hackbench is not running.
>
> The avg is significantly improved with nice latency -20 especially with
> large number of groups but min and max remain quite similar. If we add the
> histogram parameter to get details of latency, we have :
>
> hackbench -l 10000 -g 16 &
> cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
>               latency 0    latency -20
> Min Latencies:    64           62
> Avg Latencies:  1170          107
> Max Latencies: 88069        10417
> 50% latencies:   122           86
> 75% latencies:   614           91
> 85% latencies:   961           94
> 90% latencies:  1225           97
> 95% latencies:  6120          102
> 99% latencies: 18328          159
>
> With percentile details, we see the benefit of nice latency -20 as
> only 1% of the latencies are above 159us whereas the default latency
> has got 15% around ~1ms or above and 5% over the 6ms.
>
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> ---
>  include/linux/sched.h      |  4 +++-
>  include/linux/sched/prio.h |  9 +++++++++
>  init/init_task.c           |  2 +-
>  kernel/sched/core.c        | 19 ++++++++++++++-----
>  kernel/sched/debug.c       |  2 +-
>  kernel/sched/fair.c        | 32 +++++++++++++++++++++++++++-----
>  kernel/sched/sched.h       | 11 +++++++++++
>  7 files changed, 66 insertions(+), 13 deletions(-)
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 6c61bde49152..38decae3e156 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -568,6 +568,8 @@ struct sched_entity {
>  	/* cached value of my_q->h_nr_running */
>  	unsigned long			runnable_weight;
>  #endif
> +	/* preemption offset in ns */
> +	long				latency_offset;
>  
>  #ifdef CONFIG_SMP
>  	/*
> @@ -784,7 +786,7 @@ struct task_struct {
>  	int				static_prio;
>  	int				normal_prio;
>  	unsigned int			rt_priority;
> -	int				latency_nice;
> +	int				latency_prio;
>  
>  	struct sched_entity		se;
>  	struct sched_rt_entity		rt;
> diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h
> index bfcd7f1d1e11..be79503d86af 100644
> --- a/include/linux/sched/prio.h
> +++ b/include/linux/sched/prio.h
> @@ -59,5 +59,14 @@ static inline long rlimit_to_nice(long prio)
>   * Default tasks should be treated as a task with latency_nice = 0.
>   */
>  #define DEFAULT_LATENCY_NICE	0
> +#define DEFAULT_LATENCY_PRIO	(DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2)
> +
> +/*
> + * Convert user-nice values [ -20 ... 0 ... 19 ]
> + * to static latency [ 0..39 ],
> + * and back.
> + */
> +#define NICE_TO_LATENCY(nice)	((nice) + DEFAULT_LATENCY_PRIO)
> +#define LATENCY_TO_NICE(prio)	((prio) - DEFAULT_LATENCY_PRIO)
>  
>  #endif /* _LINUX_SCHED_PRIO_H */
> diff --git a/init/init_task.c b/init/init_task.c
> index 7dd71dd2d261..071deff8dbd1 100644
> --- a/init/init_task.c
> +++ b/init/init_task.c
> @@ -78,7 +78,7 @@ struct task_struct init_task
>  	.prio		= MAX_PRIO - 20,
>  	.static_prio	= MAX_PRIO - 20,
>  	.normal_prio	= MAX_PRIO - 20,
> -	.latency_nice	= DEFAULT_LATENCY_NICE,
> +	.latency_prio	= DEFAULT_LATENCY_PRIO,
>  	.policy		= SCHED_NORMAL,
>  	.cpus_ptr	= &init_task.cpus_mask,
>  	.user_cpus_ptr	= NULL,
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index d327614c70b0..d5b7e237d79b 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -1285,6 +1285,11 @@ static void set_load_weight(struct task_struct *p, bool update_load)
>  	}
>  }
>  
> +static void set_latency_offset(struct task_struct *p)
> +{
> +	p->se.latency_offset = calc_latency_offset(p->latency_prio);
> +}
> +
>  #ifdef CONFIG_UCLAMP_TASK
>  /*
>   * Serializes updates of utilization clamp values
> @@ -4681,7 +4686,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
>  		p->prio = p->normal_prio = p->static_prio;
>  		set_load_weight(p, false);
>  
> -		p->latency_nice = DEFAULT_LATENCY_NICE;
> +		p->latency_prio = NICE_TO_LATENCY(0);
> +		set_latency_offset(p);
> +
>  		/*
>  		 * We don't need the reset flag anymore after the fork. It has
>  		 * fulfilled its duty:
> @@ -7449,8 +7456,10 @@ static void __setscheduler_params(struct task_struct *p,
>  static void __setscheduler_latency(struct task_struct *p,
>  		const struct sched_attr *attr)
>  {
> -	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
> -		p->latency_nice = attr->sched_latency_nice;
> +	if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
> +		p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice);
> +		set_latency_offset(p);
> +	}
>  }
>  
>  /*
> @@ -7635,7 +7644,7 @@ static int __sched_setscheduler(struct task_struct *p,
>  		if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
>  			goto change;
>  		if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
> -		    attr->sched_latency_nice != p->latency_nice)
> +		    attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio))
>  			goto change;
>  
>  		p->sched_reset_on_fork = reset_on_fork;
> @@ -8176,7 +8185,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
>  	get_params(p, &kattr);
>  	kattr.sched_flags &= SCHED_FLAG_ALL;
>  
> -	kattr.sched_latency_nice = p->latency_nice;
> +	kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio);
>  
>  #ifdef CONFIG_UCLAMP_TASK
>  	/*
> diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
> index 68be7a3e42a3..b3922184af91 100644
> --- a/kernel/sched/debug.c
> +++ b/kernel/sched/debug.c
> @@ -1043,7 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
>  #endif
>  	P(policy);
>  	P(prio);
> -	P(latency_nice);
> +	P(latency_prio);

/proc/<pid>/sched  should update if the latency values are updated
for the cgroup right? That doesn't seem to happen.

#cd /sys/fs/cgroup/cpu
# echo -20 >  task1/cpu.latency.nice 
# cat task1/cgroup.procs 
1897
1998
1999
# cat /proc/1999/sched | grep latency
latency_prio                                 :                   20
# echo 0 >  task1/cpu.latency.nice 
# cat /proc/1999/sched | grep latency
latency_prio                                 :                   20
# echo 19 >  task1/cpu.latency.nice 
# cat /proc/1999/sched | grep latency
latency_prio                                 :                   20


>  	if (task_has_dl_policy(p)) {
>  		P(dl.runtime);
>  		P(dl.deadline);
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 81bef11eb660..414b6243208b 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4877,6 +4877,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>  		update_idle_cfs_rq_clock_pelt(cfs_rq);
>  }
>  
> +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se);
> +
>  /*
>   * Preempt the current task with a newly woken task if needed:
>   */
> @@ -4885,7 +4887,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>  {
>  	unsigned long ideal_runtime, delta_exec;
>  	struct sched_entity *se;
> -	s64 delta;
> +	s64 delta, offset;
>  
>  	/*
>  	 * When many tasks blow up the sched_period; it is possible that
> @@ -4916,10 +4918,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>  	se = __pick_first_entity(cfs_rq);
>  	delta = curr->vruntime - se->vruntime;
>  
> -	if (delta < 0)
> +	offset = wakeup_latency_gran(curr, se);
> +	if (delta < offset)
>  		return;
>  
> -	if (delta > ideal_runtime)
> +	if ((delta > ideal_runtime) ||
> +	    (delta > get_latency_max()))
>  		resched_curr(rq_of(cfs_rq));
>  }
>  
> @@ -7662,6 +7666,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
>  }
>  #endif /* CONFIG_SMP */
>  
> +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se)
> +{
> +	long latency_offset = se->latency_offset;
> +
> +	/*
> +	 * A negative latency offset means that the sched_entity has latency
> +	 * requirement that needs to be evaluated versus other entity.
> +	 * Otherwise, use the latency weight to evaluate how much scheduling
> +	 * delay is acceptable by se.
> +	 */
> +	if ((latency_offset < 0) || (curr->latency_offset < 0))
> +		latency_offset -= curr->latency_offset;
> +	latency_offset = min_t(long, latency_offset, get_latency_max());
> +
> +	return latency_offset;
> +}
> +
>  static unsigned long wakeup_gran(struct sched_entity *se)
>  {
>  	unsigned long gran = sysctl_sched_wakeup_granularity;
> @@ -7700,11 +7721,12 @@ static int
>  wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>  {
>  	s64 gran, vdiff = curr->vruntime - se->vruntime;
> +	s64 offset = wakeup_latency_gran(curr, se);
>  
> -	if (vdiff <= 0)
> +	if (vdiff < offset)
>  		return -1;
>  
> -	gran = wakeup_gran(se);
> +	gran = offset + wakeup_gran(se);
>  
>  	/*
>  	 * At wake up, the vruntime of a task is capped to not be older than
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 51ba0af7fb27..3f42f86105d4 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -2494,6 +2494,17 @@ static inline unsigned long get_sleep_latency(bool idle)
>  	return thresh;
>  }
>  
> +/*
> + * Calculate the latency offset for a priority level.
> + * We use a linear mapping of the priority in the range:
> + *     [-sysctl_sched_latency:sysctl_sched_latency]
> + */
> +static inline long calc_latency_offset(int prio)
> +{
> +	return (long)get_sleep_latency(false) * LATENCY_TO_NICE(prio) /
> +			(LATENCY_NICE_WIDTH/2);
> +}
> +
>  static inline unsigned long get_latency_max(void)
>  {
>  	unsigned long thresh = get_sleep_latency(false);

Re: [PATCH v12 8/8] sched/fair: Add latency list

[shrikanth hegde]

On 2/24/23 3:04 PM, Vincent Guittot wrote:
> Add a rb tree for latency sensitive entities so we can schedule the most
> sensitive one first even when it failed to preempt current at wakeup or
> when it got quickly preempted by another entity of higher priority.
>
> In order to keep fairness, the latency is used once at wakeup to get a
> minimum slice and not during the following scheduling slice to prevent
> long running entity to got more running time than allocated to his nice
> priority.
>
> The rb tree enables to cover the last corner case where latency
> sensitive entity can't got schedule quickly after the wakeup.
>
> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> ---
>  include/linux/sched.h |   1 +
>  kernel/sched/core.c   |   1 +
>  kernel/sched/fair.c   | 109 ++++++++++++++++++++++++++++++++++++++++--
>  kernel/sched/sched.h  |   1 +
>  4 files changed, 109 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 38decae3e156..41bb92be5ecc 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -548,6 +548,7 @@ struct sched_entity {
>  	/* For load-balancing: */
>  	struct load_weight		load;
>  	struct rb_node			run_node;
> +	struct rb_node			latency_node;
>  	struct list_head		group_node;
>  	unsigned int			on_rq;
>  
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 093cc1af73dc..752fd364216c 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
>  	p->se.nr_migrations		= 0;
>  	p->se.vruntime			= 0;
>  	INIT_LIST_HEAD(&p->se.group_node);
> +	RB_CLEAR_NODE(&p->se.latency_node);
>  
>  #ifdef CONFIG_FAIR_GROUP_SCHED
>  	p->se.cfs_rq			= NULL;
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 125a6ff53378..e2aeb4511686 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
>  
>  	return __node_2_se(last);
>  }
> +#endif
>  
> +/**************************************************************
> + * Scheduling class tree data structure manipulation methods:
> + * for latency
> + */
> +
> +static inline bool latency_before(struct sched_entity *a,
> +				struct sched_entity *b)
> +{
> +	return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> +}
> +
> +#define __latency_node_2_se(node) \
> +	rb_entry((node), struct sched_entity, latency_node)
> +
> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> +{
> +	return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> +}
> +
> +/*
> + * Enqueue an entity into the latency rb-tree:
> + */
> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> +{
> +
> +	/* Only latency sensitive entity can be added to the list */
> +	if (se->latency_offset >= 0)
> +		return;
> +
> +	if (!RB_EMPTY_NODE(&se->latency_node))
> +		return;
> +
> +	/*
> +	 * The entity is always added the latency list at wakeup.
> +	 * Then, a not waking up entity that is put back in the list after an
> +	 * execution time less than sysctl_sched_min_granularity, means that
> +	 * the entity has been preempted by a higher sched class or an entity
> +	 * with higher latency constraint. In thi case, the entity is also put
> +	 * back in the latency list so it gets a chance to run 1st during the
> +	 * next slice.
> +	 */
> +	if (!(flags & ENQUEUE_WAKEUP)) {
> +		u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> +
> +		if (delta_exec >= sysctl_sched_min_granularity)
> +			return;
> +	}
> +
> +	rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> +}
> +
> +/*
> + * Dequeue an entity from the latency rb-tree and return true if it was really
> + * part of the rb-tree:
> + */
> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> +{
> +	if (!RB_EMPTY_NODE(&se->latency_node)) {
> +		rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> +		RB_CLEAR_NODE(&se->latency_node);
> +		return true;
> +	}
> +
> +	return false;
> +}
> +
> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> +{
> +	struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> +
> +	if (!left)
> +		return NULL;
> +
> +	return __latency_node_2_se(left);
> +}
> +
> +#ifdef CONFIG_SCHED_DEBUG
>  /**************************************************************
>   * Scheduling class statistics methods:
>   */
> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>  	check_schedstat_required();
>  	update_stats_enqueue_fair(cfs_rq, se, flags);
>  	check_spread(cfs_rq, se);
> -	if (!curr)
> +	if (!curr) {
>  		__enqueue_entity(cfs_rq, se);
> +		__enqueue_latency(cfs_rq, se, flags);
> +	}
>  	se->on_rq = 1;
>  
>  	if (cfs_rq->nr_running == 1) {
> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>  
>  	clear_buddies(cfs_rq, se);
>  
> -	if (se != cfs_rq->curr)
> +	if (se != cfs_rq->curr) {
>  		__dequeue_entity(cfs_rq, se);
> +		__dequeue_latency(cfs_rq, se);
> +	}
>  	se->on_rq = 0;
>  	account_entity_dequeue(cfs_rq, se);
>  
> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>  		 */
>  		update_stats_wait_end_fair(cfs_rq, se);
>  		__dequeue_entity(cfs_rq, se);
> +		__dequeue_latency(cfs_rq, se);
>  		update_load_avg(cfs_rq, se, UPDATE_TG);
>  	}
>  
> @@ -4979,7 +5062,7 @@ static struct sched_entity *
>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>  {
>  	struct sched_entity *left = __pick_first_entity(cfs_rq);
> -	struct sched_entity *se;
> +	struct sched_entity *latency, *se;

can this variable be se_latency? 

Sorry, I should have put this in the previous reply itself.

>  
>  	/*
>  	 * If curr is set we have to see if its left of the leftmost entity
> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>  		se = cfs_rq->last;
>  	}
>  
> +	/* Check for latency sensitive entity waiting for running */
> +	latency = __pick_first_latency(cfs_rq);
> +	if (latency && (latency != se) &&
> +	    wakeup_preempt_entity(latency, se) < 1)
> +		se = latency;
> +
>  	return se;
>  }
>  
> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
>  		update_stats_wait_start_fair(cfs_rq, prev);
>  		/* Put 'current' back into the tree. */
>  		__enqueue_entity(cfs_rq, prev);
> +		__enqueue_latency(cfs_rq, prev, 0);
>  		/* in !on_rq case, update occurred at dequeue */
>  		update_load_avg(cfs_rq, prev, 0);
>  	}
> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>  void init_cfs_rq(struct cfs_rq *cfs_rq)
>  {
>  	cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> +	cfs_rq->latency_timeline = RB_ROOT_CACHED;
>  	u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
>  #ifdef CONFIG_SMP
>  	raw_spin_lock_init(&cfs_rq->removed.lock);
> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
>  	se->my_q = cfs_rq;
>  
>  	se->latency_offset = calc_latency_offset(tg->latency_prio);
> +	RB_CLEAR_NODE(&se->latency_node);
>  
>  	/* guarantee group entities always have weight */
>  	update_load_set(&se->load, NICE_0_LOAD);
> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
>  
>  	for_each_possible_cpu(i) {
>  		struct sched_entity *se = tg->se[i];
> +		struct rq *rq = cpu_rq(i);
> +		struct rq_flags rf;
> +		bool queued;
> +
> +		rq_lock_irqsave(rq, &rf);
>  
> +		queued = __dequeue_latency(se->cfs_rq, se);
>  		WRITE_ONCE(se->latency_offset, latency_offset);
> +		if (queued)
> +			__enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
> +
> +
> +		rq_unlock_irqrestore(rq, &rf);
>  	}
>  
>  	mutex_unlock(&shares_mutex);
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 9a2e71231083..21dd309e98a9 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -570,6 +570,7 @@ struct cfs_rq {
>  #endif
>  
>  	struct rb_root_cached	tasks_timeline;
> +	struct rb_root_cached	latency_timeline;
>  
>  	/*
>  	 * 'curr' points to currently running entity on this cfs_rq.

Re: [PATCH v12 5/8] sched/fair: Take into account latency priority at wakeup

[Vincent Guittot]

On Wed, 1 Mar 2023 at 20:29, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>
>
>
> On 2/24/23 3:04 PM, Vincent Guittot wrote:
> > Take into account the latency priority of a thread when deciding to
> > preempt the current running thread. We don't want to provide more CPU
> > bandwidth to a thread but reorder the scheduling to run latency sensitive
> > task first whenever possible.
> >
> > As long as a thread didn't use its bandwidth, it will be able to preempt
> > the current thread.
> >
> > At the opposite, a thread with a low latency priority will preempt current
> > thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will
> > wait for the tick to get its sched slice.
> >
> >                                    curr vruntime
> >                                        |
> >                       sysctl_sched_wakeup_granularity
> >                                    <-->
> > ----------------------------------|----|-----------------------|---------------
> >                                   |    |<--------------------->
> >                                   |    .  sysctl_sched_latency
> >                                   |    .
> > default/current latency entity    |    .
> >                                   |    .
> > 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-
> > se preempts curr at wakeup ------>|<- se doesn't preempt curr -----------------
> >                                   |    .
> >                                   |    .
> >                                   |    .
> > low latency entity                |    .
> >                                    ---------------------->|
> >                                % of sysctl_sched_latency  |
> > 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-
> > preempt ------------------------------------------------->|<- do not preempt --
> >                                   |    .
> >                                   |    .
> >                                   |    .
> > high latency entity               |    .
> >          |<-----------------------|----.
> >          | % of sysctl_sched_latency   .
> > 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1
> > preempt->|<- se doesn't preempt curr ------------------------------------------
> >
> > Tests results of nice latency impact on heavy load like hackbench:
> >
> > hackbench -l (2560 / group) -g group
> > group        latency 0             latency 19
> > 1            1.378(+/-  1%)      1.337(+/- 1%) + 3%
> > 4            1.393(+/-  3%)      1.312(+/- 3%) + 6%
> > 8            1.308(+/-  2%)      1.279(+/- 1%) + 2%
> > 16           1.347(+/-  1%)      1.317(+/- 1%) + 2%
> >
> > hackbench -p -l (2560 / group) -g group
> > group
> > 1            1.836(+/- 17%)      1.148(+/- 5%) +37%
> > 4            1.586(+/-  6%)      1.109(+/- 8%) +30%
> > 8            1.209(+/-  4%)      0.780(+/- 4%) +35%
> > 16           0.805(+/-  5%)      0.728(+/- 4%) +10%
> >
> > By deacreasing the latency prio, we reduce the number of preemption at
> > wakeup and help hackbench making progress.
> >
> > Test results of nice latency impact on short live load like cyclictest
> > while competing with heavy load like hackbench:
> >
> > hackbench -l 10000 -g $group &
> > cyclictest --policy other -D 5 -q -n
> >         latency 0           latency -20
> > group   min  avg    max     min  avg    max
> > 0       16    19     29      17   18     29
> > 1       43   299   7359      63   84   3422
> > 4       56   449  14806      45   83    284
> > 8       63   820  51123      63   83    283
> > 16      64  1326  70684      41  157  26852
> >
> > group = 0 means that hackbench is not running.
> >
> > The avg is significantly improved with nice latency -20 especially with
> > large number of groups but min and max remain quite similar. If we add the
> > histogram parameter to get details of latency, we have :
> >
> > hackbench -l 10000 -g 16 &
> > cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
> >               latency 0    latency -20
> > Min Latencies:    64           62
> > Avg Latencies:  1170          107
> > Max Latencies: 88069        10417
> > 50% latencies:   122           86
> > 75% latencies:   614           91
> > 85% latencies:   961           94
> > 90% latencies:  1225           97
> > 95% latencies:  6120          102
> > 99% latencies: 18328          159
> >
> > With percentile details, we see the benefit of nice latency -20 as
> > only 1% of the latencies are above 159us whereas the default latency
> > has got 15% around ~1ms or above and 5% over the 6ms.
> >
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> > ---
> >  include/linux/sched.h      |  4 +++-
> >  include/linux/sched/prio.h |  9 +++++++++
> >  init/init_task.c           |  2 +-
> >  kernel/sched/core.c        | 19 ++++++++++++++-----
> >  kernel/sched/debug.c       |  2 +-
> >  kernel/sched/fair.c        | 32 +++++++++++++++++++++++++++-----
> >  kernel/sched/sched.h       | 11 +++++++++++
> >  7 files changed, 66 insertions(+), 13 deletions(-)
> >
> > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > index 6c61bde49152..38decae3e156 100644
> > --- a/include/linux/sched.h
> > +++ b/include/linux/sched.h
> > @@ -568,6 +568,8 @@ struct sched_entity {
> >       /* cached value of my_q->h_nr_running */
> >       unsigned long                   runnable_weight;
> >  #endif
> > +     /* preemption offset in ns */
> > +     long                            latency_offset;
> >
> >  #ifdef CONFIG_SMP
> >       /*
> > @@ -784,7 +786,7 @@ struct task_struct {
> >       int                             static_prio;
> >       int                             normal_prio;
> >       unsigned int                    rt_priority;
> > -     int                             latency_nice;
> > +     int                             latency_prio;
> >
> >       struct sched_entity             se;
> >       struct sched_rt_entity          rt;
> > diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h
> > index bfcd7f1d1e11..be79503d86af 100644
> > --- a/include/linux/sched/prio.h
> > +++ b/include/linux/sched/prio.h
> > @@ -59,5 +59,14 @@ static inline long rlimit_to_nice(long prio)
> >   * Default tasks should be treated as a task with latency_nice = 0.
> >   */
> >  #define DEFAULT_LATENCY_NICE 0
> > +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2)
> > +
> > +/*
> > + * Convert user-nice values [ -20 ... 0 ... 19 ]
> > + * to static latency [ 0..39 ],
> > + * and back.
> > + */
> > +#define NICE_TO_LATENCY(nice)        ((nice) + DEFAULT_LATENCY_PRIO)
> > +#define LATENCY_TO_NICE(prio)        ((prio) - DEFAULT_LATENCY_PRIO)
> >
> >  #endif /* _LINUX_SCHED_PRIO_H */
> > diff --git a/init/init_task.c b/init/init_task.c
> > index 7dd71dd2d261..071deff8dbd1 100644
> > --- a/init/init_task.c
> > +++ b/init/init_task.c
> > @@ -78,7 +78,7 @@ struct task_struct init_task
> >       .prio           = MAX_PRIO - 20,
> >       .static_prio    = MAX_PRIO - 20,
> >       .normal_prio    = MAX_PRIO - 20,
> > -     .latency_nice   = DEFAULT_LATENCY_NICE,
> > +     .latency_prio   = DEFAULT_LATENCY_PRIO,
> >       .policy         = SCHED_NORMAL,
> >       .cpus_ptr       = &init_task.cpus_mask,
> >       .user_cpus_ptr  = NULL,
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index d327614c70b0..d5b7e237d79b 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -1285,6 +1285,11 @@ static void set_load_weight(struct task_struct *p, bool update_load)
> >       }
> >  }
> >
> > +static void set_latency_offset(struct task_struct *p)
> > +{
> > +     p->se.latency_offset = calc_latency_offset(p->latency_prio);
> > +}
> > +
> >  #ifdef CONFIG_UCLAMP_TASK
> >  /*
> >   * Serializes updates of utilization clamp values
> > @@ -4681,7 +4686,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
> >               p->prio = p->normal_prio = p->static_prio;
> >               set_load_weight(p, false);
> >
> > -             p->latency_nice = DEFAULT_LATENCY_NICE;
> > +             p->latency_prio = NICE_TO_LATENCY(0);
> > +             set_latency_offset(p);
> > +
> >               /*
> >                * We don't need the reset flag anymore after the fork. It has
> >                * fulfilled its duty:
> > @@ -7449,8 +7456,10 @@ static void __setscheduler_params(struct task_struct *p,
> >  static void __setscheduler_latency(struct task_struct *p,
> >               const struct sched_attr *attr)
> >  {
> > -     if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
> > -             p->latency_nice = attr->sched_latency_nice;
> > +     if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
> > +             p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice);
> > +             set_latency_offset(p);
> > +     }
> >  }
> >
> >  /*
> > @@ -7635,7 +7644,7 @@ static int __sched_setscheduler(struct task_struct *p,
> >               if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
> >                       goto change;
> >               if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
> > -                 attr->sched_latency_nice != p->latency_nice)
> > +                 attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio))
> >                       goto change;
> >
> >               p->sched_reset_on_fork = reset_on_fork;
> > @@ -8176,7 +8185,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
> >       get_params(p, &kattr);
> >       kattr.sched_flags &= SCHED_FLAG_ALL;
> >
> > -     kattr.sched_latency_nice = p->latency_nice;
> > +     kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio);
> >
> >  #ifdef CONFIG_UCLAMP_TASK
> >       /*
> > diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
> > index 68be7a3e42a3..b3922184af91 100644
> > --- a/kernel/sched/debug.c
> > +++ b/kernel/sched/debug.c
> > @@ -1043,7 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
> >  #endif
> >       P(policy);
> >       P(prio);
> > -     P(latency_nice);
> > +     P(latency_prio);
>
> /proc/<pid>/sched  should update if the latency values are updated
> for the cgroup right? That doesn't seem to happen.

No It's not. The cgroup latency_nice value applies the the
sched_entity of the group in which the task are scheduled

>
> #cd /sys/fs/cgroup/cpu
> # echo -20 >  task1/cpu.latency.nice
> # cat task1/cgroup.procs
> 1897
> 1998
> 1999
> # cat /proc/1999/sched | grep latency
> latency_prio                                 :                   20
> # echo 0 >  task1/cpu.latency.nice
> # cat /proc/1999/sched | grep latency
> latency_prio                                 :                   20
> # echo 19 >  task1/cpu.latency.nice
> # cat /proc/1999/sched | grep latency
> latency_prio                                 :                   20
>
>
> >       if (task_has_dl_policy(p)) {
> >               P(dl.runtime);
> >               P(dl.deadline);
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 81bef11eb660..414b6243208b 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -4877,6 +4877,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >               update_idle_cfs_rq_clock_pelt(cfs_rq);
> >  }
> >
> > +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se);
> > +
> >  /*
> >   * Preempt the current task with a newly woken task if needed:
> >   */
> > @@ -4885,7 +4887,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >  {
> >       unsigned long ideal_runtime, delta_exec;
> >       struct sched_entity *se;
> > -     s64 delta;
> > +     s64 delta, offset;
> >
> >       /*
> >        * When many tasks blow up the sched_period; it is possible that
> > @@ -4916,10 +4918,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >       se = __pick_first_entity(cfs_rq);
> >       delta = curr->vruntime - se->vruntime;
> >
> > -     if (delta < 0)
> > +     offset = wakeup_latency_gran(curr, se);
> > +     if (delta < offset)
> >               return;
> >
> > -     if (delta > ideal_runtime)
> > +     if ((delta > ideal_runtime) ||
> > +         (delta > get_latency_max()))
> >               resched_curr(rq_of(cfs_rq));
> >  }
> >
> > @@ -7662,6 +7666,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
> >  }
> >  #endif /* CONFIG_SMP */
> >
> > +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se)
> > +{
> > +     long latency_offset = se->latency_offset;
> > +
> > +     /*
> > +      * A negative latency offset means that the sched_entity has latency
> > +      * requirement that needs to be evaluated versus other entity.
> > +      * Otherwise, use the latency weight to evaluate how much scheduling
> > +      * delay is acceptable by se.
> > +      */
> > +     if ((latency_offset < 0) || (curr->latency_offset < 0))
> > +             latency_offset -= curr->latency_offset;
> > +     latency_offset = min_t(long, latency_offset, get_latency_max());
> > +
> > +     return latency_offset;
> > +}
> > +
> >  static unsigned long wakeup_gran(struct sched_entity *se)
> >  {
> >       unsigned long gran = sysctl_sched_wakeup_granularity;
> > @@ -7700,11 +7721,12 @@ static int
> >  wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >  {
> >       s64 gran, vdiff = curr->vruntime - se->vruntime;
> > +     s64 offset = wakeup_latency_gran(curr, se);
> >
> > -     if (vdiff <= 0)
> > +     if (vdiff < offset)
> >               return -1;
> >
> > -     gran = wakeup_gran(se);
> > +     gran = offset + wakeup_gran(se);
> >
> >       /*
> >        * At wake up, the vruntime of a task is capped to not be older than
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index 51ba0af7fb27..3f42f86105d4 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -2494,6 +2494,17 @@ static inline unsigned long get_sleep_latency(bool idle)
> >       return thresh;
> >  }
> >
> > +/*
> > + * Calculate the latency offset for a priority level.
> > + * We use a linear mapping of the priority in the range:
> > + *     [-sysctl_sched_latency:sysctl_sched_latency]
> > + */
> > +static inline long calc_latency_offset(int prio)
> > +{
> > +     return (long)get_sleep_latency(false) * LATENCY_TO_NICE(prio) /
> > +                     (LATENCY_NICE_WIDTH/2);
> > +}
> > +
> >  static inline unsigned long get_latency_max(void)
> >  {
> >       unsigned long thresh = get_sleep_latency(false);
>

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Vincent Guittot]

On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>
>
> On 2/24/23 3:04 PM, Vincent Guittot wrote:
> > Add a rb tree for latency sensitive entities so we can schedule the most
> > sensitive one first even when it failed to preempt current at wakeup or
> > when it got quickly preempted by another entity of higher priority.
> >
> > In order to keep fairness, the latency is used once at wakeup to get a
> > minimum slice and not during the following scheduling slice to prevent
> > long running entity to got more running time than allocated to his nice
> > priority.
> >
> > The rb tree enables to cover the last corner case where latency
> > sensitive entity can't got schedule quickly after the wakeup.
> >
> > Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> > Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> > ---
> >  include/linux/sched.h |   1 +
> >  kernel/sched/core.c   |   1 +
> >  kernel/sched/fair.c   | 109 ++++++++++++++++++++++++++++++++++++++++--
> >  kernel/sched/sched.h  |   1 +
> >  4 files changed, 109 insertions(+), 3 deletions(-)
> >
> > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > index 38decae3e156..41bb92be5ecc 100644
> > --- a/include/linux/sched.h
> > +++ b/include/linux/sched.h
> > @@ -548,6 +548,7 @@ struct sched_entity {
> >       /* For load-balancing: */
> >       struct load_weight              load;
> >       struct rb_node                  run_node;
> > +     struct rb_node                  latency_node;
>
> Ran pahole to see if the frequently accessed fields change across cachelines.
> There is no change in cachelines of task_struct, whereas sched_entity differs
> due to latency_node.  Maybe the latency_node could be placed after
> runnable_weight as there is space available in that cacheline.

I will run some test on my system to confimr your results but we can
move latency_node field if it helps cache hit stats

>
>
> 6.2
> #pahole sched_entity
> struct sched_entity {
>         struct load_weight         load;                 /*     0    16 */
>
>         /* XXX last struct has 4 bytes of padding */
>
>         struct rb_node             run_node;             /*    16    24 */
>         struct list_head           group_node;           /*    40    16 */
>         unsigned int               on_rq;                /*    56     4 */
>
>         /* XXX 4 bytes hole, try to pack */
>
>         u64                        exec_start;           /*    64     8 */
>         u64                        sum_exec_runtime;     /*    72     8 */
>         u64                        vruntime;             /*    80     8 */
>         u64                        prev_sum_exec_runtime; /*    88     8 */
>         u64                        nr_migrations;        /*    96     8 */
>         int                        depth;                /*   104     4 */
>
>         /* XXX 4 bytes hole, try to pack */
>
>         struct sched_entity *      parent;               /*   112     8 */
>         struct cfs_rq *            cfs_rq;               /*   120     8 */
>         /* --- cacheline 1 boundary (128 bytes) --- */
>         struct cfs_rq *            my_q;                 /*   128     8 */
>         long unsigned int          runnable_weight;      /*   136     8 */
>
>         /* XXX 112 bytes hole, try to pack */
>
>         /* --- cacheline 2 boundary (256 bytes) --- */
>         struct sched_avg           avg;                  /*   256   128 */
>
>
> 6.2 + V12 patch
> #pahole sched_entity
> struct sched_entity {
>         struct load_weight         load;                 /*     0    16 */
>
>         /* XXX last struct has 4 bytes of padding */
>
>         struct rb_node             run_node;             /*    16    24 */
>         struct rb_node             latency_node;         /*    40    24 */
>         struct list_head           group_node;           /*    64    16 */
>         unsigned int               on_rq;                /*    80     4 */
>
>         /* XXX 4 bytes hole, try to pack */
>
>         u64                        exec_start;           /*    88     8 */
>         u64                        sum_exec_runtime;     /*    96     8 */
>         u64                        vruntime;             /*   104     8 */
>         u64                        prev_sum_exec_runtime; /*   112     8 */
>         u64                        nr_migrations;        /*   120     8 */
>         /* --- cacheline 1 boundary (128 bytes) --- */
>         int                        depth;                /*   128     4 */
>
>         /* XXX 4 bytes hole, try to pack */
>
>         struct sched_entity *      parent;               /*   136     8 */
>         struct cfs_rq *            cfs_rq;               /*   144     8 */
>         struct cfs_rq *            my_q;                 /*   152     8 */
>
>
> 6.2 + V12 patch + Re-shuffle of latency_node
> #pahole sched_entity
> struct sched_entity {
>         struct load_weight         load;                 /*     0    16 */
>
>         /* XXX last struct has 4 bytes of padding */
>
>         struct rb_node             run_node;             /*    16    24 */
>         struct list_head           group_node;           /*    40    16 */
>         unsigned int               on_rq;                /*    56     4 */
>
>         /* XXX 4 bytes hole, try to pack */
>
>         u64                        exec_start;           /*    64     8 */
>         u64                        sum_exec_runtime;     /*    72     8 */
>         u64                        vruntime;             /*    80     8 */
>         u64                        prev_sum_exec_runtime; /*    88     8 */
>         u64                        nr_migrations;        /*    96     8 */
>         int                        depth;                /*   104     4 */
>
>         /* XXX 4 bytes hole, try to pack */
>
>         struct sched_entity *      parent;               /*   112     8 */
>         struct cfs_rq *            cfs_rq;               /*   120     8 */
>         /* --- cacheline 1 boundary (128 bytes) --- */
>         struct cfs_rq *            my_q;                 /*   128     8 */
>         long unsigned int          runnable_weight;      /*   136     8 */
>         struct rb_node             latency_node;         /*   144    24 */
>         long int                   latency_offset;       /*   168     8 */
>
>         /* XXX 80 bytes hole, try to pack */
>
>         /* --- cacheline 2 boundary (256 bytes) --- */
>
>
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index a2b52cf5e1bb..1e93aaaeead2 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -548,7 +548,6 @@ struct sched_entity {
>         /* For load-balancing: */
>         struct load_weight              load;
>         struct rb_node                  run_node;
> -       struct rb_node                  latency_node;
>         struct list_head                group_node;
>         unsigned int                    on_rq;
>
> @@ -569,6 +568,7 @@ struct sched_entity {
>         /* cached value of my_q->h_nr_running */
>         unsigned long                   runnable_weight;
>  #endif
> +       struct rb_node                  latency_node;
>         /* preemption offset in ns */
>         long                            latency_offset;
>
>
> Ran the schbench and hackbench with this patch series. Here comparison is
> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
> and other is running these benchmarks (group2). Set the latency nice
> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
> Total of 96 CPU.
>
> schbench gets lower latency compared to stabletree. Whereas hackbench seems
> to regress under this case. Maybe i am doing something wrong. I will re-run
> and attach the numbers to series.
> Please suggest if any variation in the test i need to try.

hackbench takes advanatge of a latency nice 19 as it mainly wants to
run longer slice to move forward rather than preempting others all the
time

>
> Re-arrange seems to help the patch series by avoiding an cacheline miss.
>
> =========================
> schbench
> =========================
>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
> 1 Thread
>   50.0th:        9.00  |    9.00        |        9.50
>   75.0th:       10.50  |   10.00        |        9.50
>   90.0th:       11.00  |   11.00        |       10.50
>   95.0th:       11.00  |   11.00        |       11.00
>   99.0th:       11.50  |   11.50        |       11.50
>   99.5th:       12.50  |   12.00        |       12.00
>   99.9th:       14.50  |   13.50        |       12.00
> 2 Threads
>   50.0th:        9.50  |    9.50        |        8.50
>   75.0th:       11.00  |   10.50        |        9.50
>   90.0th:       13.50  |   11.50        |       10.50
>   95.0th:       14.00  |   12.00        |       11.00
>   99.0th:       15.50  |   13.50        |       12.00
>   99.5th:       16.00  |   14.00        |       12.00
>   99.9th:       17.00  |   16.00        |       16.50
> 4 Threads
>   50.0th:       11.50  |   11.50        |       10.50
>   75.0th:       13.50  |   12.50        |       12.50
>   90.0th:       15.50  |   14.50        |       14.00
>   95.0th:       16.50  |   15.50        |       14.50
>   99.0th:       20.00  |   17.50        |       16.50
>   99.5th:       20.50  |   18.50        |       17.00
>   99.9th:       22.50  |   21.00        |       19.00
> 8 Threads
>   50.0th:       14.00  |   14.00        |       14.00
>   75.0th:       16.00  |   16.00        |       16.00
>   90.0th:       18.00  |   18.00        |       17.50
>   95.0th:       18.50  |   18.50        |       18.50
>   99.0th:       20.00  |   20.00        |       20.00
>   99.5th:       20.50  |   21.50        |       21.00
>   99.9th:       22.50  |   23.50        |       23.00
> 16 Threads
>   50.0th:       19.00  |   18.50        |       19.00
>   75.0th:       23.00  |   22.50        |       23.00
>   90.0th:       25.00  |   25.50        |       25.00
>   95.0th:       26.50  |   26.50        |       26.00
>   99.0th:       28.50  |   29.00        |       28.50
>   99.5th:       31.00  |   30.00        |       30.00
>   99.9th:     5626.00  | 4761.50        |       32.50
> 32 Threads
>   50.0th:       27.00  |   27.50        |       29.00
>   75.0th:       35.50  |   36.50        |       38.50
>   90.0th:       42.00  |   44.00        |       50.50
>   95.0th:      447.50  | 2959.00        |     8544.00
>   99.0th:     7372.00  | 17032.00       |    19136.00
>   99.5th:    15360.00  | 19808.00       |    20704.00
>   99.9th:    20640.00  | 30048.00       |    30048.00
>
> ====================
> hackbench
> ====================
>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
>
> Process 10 Time:        0.35    |       0.42        |           0.41
> Process 20 Time:        0.61    |       0.76        |           0.76
> Process 30 Time:        0.87    |       1.06        |           1.05
> thread 10 Time:         0.35    |       0.43        |           0.42
> thread 20 Time:         0.66    |       0.79        |           0.78
> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
>
>
> >       struct list_head                group_node;
> >       unsigned int                    on_rq;
> >
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 093cc1af73dc..752fd364216c 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
> >       p->se.nr_migrations             = 0;
> >       p->se.vruntime                  = 0;
> >       INIT_LIST_HEAD(&p->se.group_node);
> > +     RB_CLEAR_NODE(&p->se.latency_node);
> >
> >  #ifdef CONFIG_FAIR_GROUP_SCHED
> >       p->se.cfs_rq                    = NULL;
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 125a6ff53378..e2aeb4511686 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> >
> >       return __node_2_se(last);
> >  }
> > +#endif
> >
> > +/**************************************************************
> > + * Scheduling class tree data structure manipulation methods:
> > + * for latency
> > + */
> > +
> > +static inline bool latency_before(struct sched_entity *a,
> > +                             struct sched_entity *b)
> > +{
> > +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> > +}
> > +
> > +#define __latency_node_2_se(node) \
> > +     rb_entry((node), struct sched_entity, latency_node)
> > +
> > +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> > +{
> > +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> > +}
> > +
> > +/*
> > + * Enqueue an entity into the latency rb-tree:
> > + */
> > +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> > +{
> > +
> > +     /* Only latency sensitive entity can be added to the list */
> > +     if (se->latency_offset >= 0)
> > +             return;
> > +
> > +     if (!RB_EMPTY_NODE(&se->latency_node))
> > +             return;
> > +
> > +     /*
> > +      * The entity is always added the latency list at wakeup.
> > +      * Then, a not waking up entity that is put back in the list after an
> > +      * execution time less than sysctl_sched_min_granularity, means that
> > +      * the entity has been preempted by a higher sched class or an entity
> > +      * with higher latency constraint. In thi case, the entity is also put
> > +      * back in the latency list so it gets a chance to run 1st during the
> > +      * next slice.
> > +      */
> > +     if (!(flags & ENQUEUE_WAKEUP)) {
> > +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> > +
> > +             if (delta_exec >= sysctl_sched_min_granularity)
> > +                     return;
> > +     }
> > +
> > +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> > +}
> > +
> > +/*
> > + * Dequeue an entity from the latency rb-tree and return true if it was really
> > + * part of the rb-tree:
> > + */
> > +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> > +{
> > +     if (!RB_EMPTY_NODE(&se->latency_node)) {
> > +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> > +             RB_CLEAR_NODE(&se->latency_node);
> > +             return true;
> > +     }
> > +
> > +     return false;
> > +}
> > +
> > +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> > +{
> > +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> > +
> > +     if (!left)
> > +             return NULL;
> > +
> > +     return __latency_node_2_se(left);
> > +}
> > +
> > +#ifdef CONFIG_SCHED_DEBUG
> >  /**************************************************************
> >   * Scheduling class statistics methods:
> >   */
> > @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >       check_schedstat_required();
> >       update_stats_enqueue_fair(cfs_rq, se, flags);
> >       check_spread(cfs_rq, se);
> > -     if (!curr)
> > +     if (!curr) {
> >               __enqueue_entity(cfs_rq, se);
> > +             __enqueue_latency(cfs_rq, se, flags);
> > +     }
> >       se->on_rq = 1;
> >
> >       if (cfs_rq->nr_running == 1) {
> > @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >
> >       clear_buddies(cfs_rq, se);
> >
> > -     if (se != cfs_rq->curr)
> > +     if (se != cfs_rq->curr) {
> >               __dequeue_entity(cfs_rq, se);
> > +             __dequeue_latency(cfs_rq, se);
> > +     }
> >       se->on_rq = 0;
> >       account_entity_dequeue(cfs_rq, se);
> >
> > @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >                */
> >               update_stats_wait_end_fair(cfs_rq, se);
> >               __dequeue_entity(cfs_rq, se);
> > +             __dequeue_latency(cfs_rq, se);
> >               update_load_avg(cfs_rq, se, UPDATE_TG);
> >       }
> >
> > @@ -4979,7 +5062,7 @@ static struct sched_entity *
> >  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >  {
> >       struct sched_entity *left = __pick_first_entity(cfs_rq);
> > -     struct sched_entity *se;
> > +     struct sched_entity *latency, *se;
> >
> >       /*
> >        * If curr is set we have to see if its left of the leftmost entity
> > @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >               se = cfs_rq->last;
> >       }
> >
> > +     /* Check for latency sensitive entity waiting for running */
> > +     latency = __pick_first_latency(cfs_rq);
> > +     if (latency && (latency != se) &&
> > +         wakeup_preempt_entity(latency, se) < 1)
> > +             se = latency;
> > +
> >       return se;
> >  }
> >
> > @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> >               update_stats_wait_start_fair(cfs_rq, prev);
> >               /* Put 'current' back into the tree. */
> >               __enqueue_entity(cfs_rq, prev);
> > +             __enqueue_latency(cfs_rq, prev, 0);
> >               /* in !on_rq case, update occurred at dequeue */
> >               update_load_avg(cfs_rq, prev, 0);
> >       }
> > @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> >  void init_cfs_rq(struct cfs_rq *cfs_rq)
> >  {
> >       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> > +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
> >       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> >  #ifdef CONFIG_SMP
> >       raw_spin_lock_init(&cfs_rq->removed.lock);
> > @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
> >       se->my_q = cfs_rq;
> >
> >       se->latency_offset = calc_latency_offset(tg->latency_prio);
> > +     RB_CLEAR_NODE(&se->latency_node);
> >
> >       /* guarantee group entities always have weight */
> >       update_load_set(&se->load, NICE_0_LOAD);
> > @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
> >
> >       for_each_possible_cpu(i) {
> >               struct sched_entity *se = tg->se[i];
> > +             struct rq *rq = cpu_rq(i);
> > +             struct rq_flags rf;
> > +             bool queued;
> > +
> > +             rq_lock_irqsave(rq, &rf);
> >
> > +             queued = __dequeue_latency(se->cfs_rq, se);
> >               WRITE_ONCE(se->latency_offset, latency_offset);
> > +             if (queued)
> > +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
> > +
> > +
> > +             rq_unlock_irqrestore(rq, &rf);
> >       }
> >
> >       mutex_unlock(&shares_mutex);
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index 9a2e71231083..21dd309e98a9 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -570,6 +570,7 @@ struct cfs_rq {
> >  #endif
> >
> >       struct rb_root_cached   tasks_timeline;
> > +     struct rb_root_cached   latency_timeline;
> >
> >       /*
> >        * 'curr' points to currently running entity on this cfs_rq.
>

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Shrikanth Hegde]

On 3/2/23 1:20 PM, Vincent Guittot wrote:
> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>>
>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
>>> Add a rb tree for latency sensitive entities so we can schedule the most
>>> sensitive one first even when it failed to preempt current at wakeup or
>>> when it got quickly preempted by another entity of higher priority.
>>>
>>> In order to keep fairness, the latency is used once at wakeup to get a
>>> minimum slice and not during the following scheduling slice to prevent
>>> long running entity to got more running time than allocated to his nice
>>> priority.
>>>
>>> The rb tree enables to cover the last corner case where latency
>>> sensitive entity can't got schedule quickly after the wakeup.
>>>
>>> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
>>> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
>>> ---
>>>  include/linux/sched.h |   1 +
>>>  kernel/sched/core.c   |   1 +
>>>  kernel/sched/fair.c   | 109 ++++++++++++++++++++++++++++++++++++++++--
>>>  kernel/sched/sched.h  |   1 +
>>>  4 files changed, 109 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>>> index 38decae3e156..41bb92be5ecc 100644
>>> --- a/include/linux/sched.h
>>> +++ b/include/linux/sched.h
>>> @@ -548,6 +548,7 @@ struct sched_entity {
>>>       /* For load-balancing: */
>>>       struct load_weight              load;
>>>       struct rb_node                  run_node;
>>> +     struct rb_node                  latency_node;
>> Ran pahole to see if the frequently accessed fields change across cachelines.
>> There is no change in cachelines of task_struct, whereas sched_entity differs
>> due to latency_node.  Maybe the latency_node could be placed after
>> runnable_weight as there is space available in that cacheline.
> I will run some test on my system to confimr your results but we can
> move latency_node field if it helps cache hit stats
>
>>
>> 6.2
>> #pahole sched_entity
>> struct sched_entity {
>>         struct load_weight         load;                 /*     0    16 */
>>
>>         /* XXX last struct has 4 bytes of padding */
>>
>>         struct rb_node             run_node;             /*    16    24 */
>>         struct list_head           group_node;           /*    40    16 */
>>         unsigned int               on_rq;                /*    56     4 */
>>
>>         /* XXX 4 bytes hole, try to pack */
>>
>>         u64                        exec_start;           /*    64     8 */
>>         u64                        sum_exec_runtime;     /*    72     8 */
>>         u64                        vruntime;             /*    80     8 */
>>         u64                        prev_sum_exec_runtime; /*    88     8 */
>>         u64                        nr_migrations;        /*    96     8 */
>>         int                        depth;                /*   104     4 */
>>
>>         /* XXX 4 bytes hole, try to pack */
>>
>>         struct sched_entity *      parent;               /*   112     8 */
>>         struct cfs_rq *            cfs_rq;               /*   120     8 */
>>         /* --- cacheline 1 boundary (128 bytes) --- */
>>         struct cfs_rq *            my_q;                 /*   128     8 */
>>         long unsigned int          runnable_weight;      /*   136     8 */
>>
>>         /* XXX 112 bytes hole, try to pack */
>>
>>         /* --- cacheline 2 boundary (256 bytes) --- */
>>         struct sched_avg           avg;                  /*   256   128 */
>>
>>
>> 6.2 + V12 patch
>> #pahole sched_entity
>> struct sched_entity {
>>         struct load_weight         load;                 /*     0    16 */
>>
>>         /* XXX last struct has 4 bytes of padding */
>>
>>         struct rb_node             run_node;             /*    16    24 */
>>         struct rb_node             latency_node;         /*    40    24 */
>>         struct list_head           group_node;           /*    64    16 */
>>         unsigned int               on_rq;                /*    80     4 */
>>
>>         /* XXX 4 bytes hole, try to pack */
>>
>>         u64                        exec_start;           /*    88     8 */
>>         u64                        sum_exec_runtime;     /*    96     8 */
>>         u64                        vruntime;             /*   104     8 */
>>         u64                        prev_sum_exec_runtime; /*   112     8 */
>>         u64                        nr_migrations;        /*   120     8 */
>>         /* --- cacheline 1 boundary (128 bytes) --- */
>>         int                        depth;                /*   128     4 */
>>
>>         /* XXX 4 bytes hole, try to pack */
>>
>>         struct sched_entity *      parent;               /*   136     8 */
>>         struct cfs_rq *            cfs_rq;               /*   144     8 */
>>         struct cfs_rq *            my_q;                 /*   152     8 */
>>
>>
>> 6.2 + V12 patch + Re-shuffle of latency_node
>> #pahole sched_entity
>> struct sched_entity {
>>         struct load_weight         load;                 /*     0    16 */
>>
>>         /* XXX last struct has 4 bytes of padding */
>>
>>         struct rb_node             run_node;             /*    16    24 */
>>         struct list_head           group_node;           /*    40    16 */
>>         unsigned int               on_rq;                /*    56     4 */
>>
>>         /* XXX 4 bytes hole, try to pack */
>>
>>         u64                        exec_start;           /*    64     8 */
>>         u64                        sum_exec_runtime;     /*    72     8 */
>>         u64                        vruntime;             /*    80     8 */
>>         u64                        prev_sum_exec_runtime; /*    88     8 */
>>         u64                        nr_migrations;        /*    96     8 */
>>         int                        depth;                /*   104     4 */
>>
>>         /* XXX 4 bytes hole, try to pack */
>>
>>         struct sched_entity *      parent;               /*   112     8 */
>>         struct cfs_rq *            cfs_rq;               /*   120     8 */
>>         /* --- cacheline 1 boundary (128 bytes) --- */
>>         struct cfs_rq *            my_q;                 /*   128     8 */
>>         long unsigned int          runnable_weight;      /*   136     8 */
>>         struct rb_node             latency_node;         /*   144    24 */
>>         long int                   latency_offset;       /*   168     8 */
>>
>>         /* XXX 80 bytes hole, try to pack */
>>
>>         /* --- cacheline 2 boundary (256 bytes) --- */
>>
>>
>>
>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>> index a2b52cf5e1bb..1e93aaaeead2 100644
>> --- a/include/linux/sched.h
>> +++ b/include/linux/sched.h
>> @@ -548,7 +548,6 @@ struct sched_entity {
>>         /* For load-balancing: */
>>         struct load_weight              load;
>>         struct rb_node                  run_node;
>> -       struct rb_node                  latency_node;
>>         struct list_head                group_node;
>>         unsigned int                    on_rq;
>>
>> @@ -569,6 +568,7 @@ struct sched_entity {
>>         /* cached value of my_q->h_nr_running */
>>         unsigned long                   runnable_weight;
>>  #endif
>> +       struct rb_node                  latency_node;
>>         /* preemption offset in ns */
>>         long                            latency_offset;
>>
>>
>> Ran the schbench and hackbench with this patch series. Here comparison is
>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
>> and other is running these benchmarks (group2). Set the latency nice
>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
>> Total of 96 CPU.
>>
>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
>> to regress under this case. Maybe i am doing something wrong. I will re-run
>> and attach the numbers to series.
>> Please suggest if any variation in the test i need to try.
> hackbench takes advanatge of a latency nice 19 as it mainly wants to
> run longer slice to move forward rather than preempting others all the
> time

hackbench still seems to regress in different latency nice values compared to
baseline of 6.2 in this case. up to 50% in some cases.

12 core powerpc system  with SMT=8 i.e 96 CPU
running 2 CPU cgroups. No quota assigned.
1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
latency is not changed for this cgroup.
2nd cgroup is running hackbench. This cgroup is assigned the different latency
nice values of 0, -20 and 19. 

Numbers are average of 10 runs in each case. Time is in seconds

type	   groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
		  |	       | lat nice=0    | lat nice=-20| lat nice=+19
		  |	       |	       |             |
Process       10  |   0.36     |     0.41      |    0.43     |    0.42
Process	      20  |   0.62     |     0.76      |    0.75     |    0.75
Process       30  |   0.87     |     1.05      |    1.04     |    1.06
Process       40  |   1.13     |     1.34      |    1.33     |    1.33
Process       50  |   1.38     |     1.62      |    1.66     |    1.63
Process       60  |   1.64     |     1.91      |    1.97     |    1.90
thread        10  |   0.35     |     0.41      |    0.44     |    0.42
thread        20  |   0.64     |     0.78      |    0.77     |    0.79
Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55



>> Re-arrange seems to help the patch series by avoiding an cacheline miss.
>>
>> =========================
>> schbench
>> =========================
>>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
>> 1 Thread
>>   50.0th:        9.00  |    9.00        |        9.50
>>   75.0th:       10.50  |   10.00        |        9.50
>>   90.0th:       11.00  |   11.00        |       10.50
>>   95.0th:       11.00  |   11.00        |       11.00
>>   99.0th:       11.50  |   11.50        |       11.50
>>   99.5th:       12.50  |   12.00        |       12.00
>>   99.9th:       14.50  |   13.50        |       12.00
>> 2 Threads
>>   50.0th:        9.50  |    9.50        |        8.50
>>   75.0th:       11.00  |   10.50        |        9.50
>>   90.0th:       13.50  |   11.50        |       10.50
>>   95.0th:       14.00  |   12.00        |       11.00
>>   99.0th:       15.50  |   13.50        |       12.00
>>   99.5th:       16.00  |   14.00        |       12.00
>>   99.9th:       17.00  |   16.00        |       16.50
>> 4 Threads
>>   50.0th:       11.50  |   11.50        |       10.50
>>   75.0th:       13.50  |   12.50        |       12.50
>>   90.0th:       15.50  |   14.50        |       14.00
>>   95.0th:       16.50  |   15.50        |       14.50
>>   99.0th:       20.00  |   17.50        |       16.50
>>   99.5th:       20.50  |   18.50        |       17.00
>>   99.9th:       22.50  |   21.00        |       19.00
>> 8 Threads
>>   50.0th:       14.00  |   14.00        |       14.00
>>   75.0th:       16.00  |   16.00        |       16.00
>>   90.0th:       18.00  |   18.00        |       17.50
>>   95.0th:       18.50  |   18.50        |       18.50
>>   99.0th:       20.00  |   20.00        |       20.00
>>   99.5th:       20.50  |   21.50        |       21.00
>>   99.9th:       22.50  |   23.50        |       23.00
>> 16 Threads
>>   50.0th:       19.00  |   18.50        |       19.00
>>   75.0th:       23.00  |   22.50        |       23.00
>>   90.0th:       25.00  |   25.50        |       25.00
>>   95.0th:       26.50  |   26.50        |       26.00
>>   99.0th:       28.50  |   29.00        |       28.50
>>   99.5th:       31.00  |   30.00        |       30.00
>>   99.9th:     5626.00  | 4761.50        |       32.50
>> 32 Threads
>>   50.0th:       27.00  |   27.50        |       29.00
>>   75.0th:       35.50  |   36.50        |       38.50
>>   90.0th:       42.00  |   44.00        |       50.50
>>   95.0th:      447.50  | 2959.00        |     8544.00
>>   99.0th:     7372.00  | 17032.00       |    19136.00
>>   99.5th:    15360.00  | 19808.00       |    20704.00
>>   99.9th:    20640.00  | 30048.00       |    30048.00
>>
>> ====================
>> hackbench
>> ====================
>>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
>>
>> Process 10 Time:        0.35    |       0.42        |           0.41
>> Process 20 Time:        0.61    |       0.76        |           0.76
>> Process 30 Time:        0.87    |       1.06        |           1.05
>> thread 10 Time:         0.35    |       0.43        |           0.42
>> thread 20 Time:         0.66    |       0.79        |           0.78
>> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
>> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
>> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
>> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
>> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
>>
>>
>>>       struct list_head                group_node;
>>>       unsigned int                    on_rq;
>>>
>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>> index 093cc1af73dc..752fd364216c 100644
>>> --- a/kernel/sched/core.c
>>> +++ b/kernel/sched/core.c
>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
>>>       p->se.nr_migrations             = 0;
>>>       p->se.vruntime                  = 0;
>>>       INIT_LIST_HEAD(&p->se.group_node);
>>> +     RB_CLEAR_NODE(&p->se.latency_node);
>>>
>>>  #ifdef CONFIG_FAIR_GROUP_SCHED
>>>       p->se.cfs_rq                    = NULL;
>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>> index 125a6ff53378..e2aeb4511686 100644
>>> --- a/kernel/sched/fair.c
>>> +++ b/kernel/sched/fair.c
>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
>>>
>>>       return __node_2_se(last);
>>>  }
>>> +#endif
>>>
>>> +/**************************************************************
>>> + * Scheduling class tree data structure manipulation methods:
>>> + * for latency
>>> + */
>>> +
>>> +static inline bool latency_before(struct sched_entity *a,
>>> +                             struct sched_entity *b)
>>> +{
>>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
>>> +}
>>> +
>>> +#define __latency_node_2_se(node) \
>>> +     rb_entry((node), struct sched_entity, latency_node)
>>> +
>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
>>> +{
>>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
>>> +}
>>> +
>>> +/*
>>> + * Enqueue an entity into the latency rb-tree:
>>> + */
>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>> +{
>>> +
>>> +     /* Only latency sensitive entity can be added to the list */
>>> +     if (se->latency_offset >= 0)
>>> +             return;
>>> +
>>> +     if (!RB_EMPTY_NODE(&se->latency_node))
>>> +             return;
>>> +
>>> +     /*
>>> +      * The entity is always added the latency list at wakeup.
>>> +      * Then, a not waking up entity that is put back in the list after an
>>> +      * execution time less than sysctl_sched_min_granularity, means that
>>> +      * the entity has been preempted by a higher sched class or an entity
>>> +      * with higher latency constraint. In thi case, the entity is also put
>>> +      * back in the latency list so it gets a chance to run 1st during the
>>> +      * next slice.
>>> +      */
>>> +     if (!(flags & ENQUEUE_WAKEUP)) {
>>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
>>> +
>>> +             if (delta_exec >= sysctl_sched_min_granularity)
>>> +                     return;
>>> +     }
>>> +
>>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
>>> +}
>>> +
>>> +/*
>>> + * Dequeue an entity from the latency rb-tree and return true if it was really
>>> + * part of the rb-tree:
>>> + */
>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>> +{
>>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
>>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
>>> +             RB_CLEAR_NODE(&se->latency_node);
>>> +             return true;
>>> +     }
>>> +
>>> +     return false;
>>> +}
>>> +
>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
>>> +{
>>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
>>> +
>>> +     if (!left)
>>> +             return NULL;
>>> +
>>> +     return __latency_node_2_se(left);
>>> +}
>>> +
>>> +#ifdef CONFIG_SCHED_DEBUG
>>>  /**************************************************************
>>>   * Scheduling class statistics methods:
>>>   */
>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>       check_schedstat_required();
>>>       update_stats_enqueue_fair(cfs_rq, se, flags);
>>>       check_spread(cfs_rq, se);
>>> -     if (!curr)
>>> +     if (!curr) {
>>>               __enqueue_entity(cfs_rq, se);
>>> +             __enqueue_latency(cfs_rq, se, flags);
>>> +     }
>>>       se->on_rq = 1;
>>>
>>>       if (cfs_rq->nr_running == 1) {
>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>
>>>       clear_buddies(cfs_rq, se);
>>>
>>> -     if (se != cfs_rq->curr)
>>> +     if (se != cfs_rq->curr) {
>>>               __dequeue_entity(cfs_rq, se);
>>> +             __dequeue_latency(cfs_rq, se);
>>> +     }
>>>       se->on_rq = 0;
>>>       account_entity_dequeue(cfs_rq, se);
>>>
>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>                */
>>>               update_stats_wait_end_fair(cfs_rq, se);
>>>               __dequeue_entity(cfs_rq, se);
>>> +             __dequeue_latency(cfs_rq, se);
>>>               update_load_avg(cfs_rq, se, UPDATE_TG);
>>>       }
>>>
>>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
>>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>  {
>>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
>>> -     struct sched_entity *se;
>>> +     struct sched_entity *latency, *se;
>>>
>>>       /*
>>>        * If curr is set we have to see if its left of the leftmost entity
>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>               se = cfs_rq->last;
>>>       }
>>>
>>> +     /* Check for latency sensitive entity waiting for running */
>>> +     latency = __pick_first_latency(cfs_rq);
>>> +     if (latency && (latency != se) &&
>>> +         wakeup_preempt_entity(latency, se) < 1)
>>> +             se = latency;
>>> +
>>>       return se;
>>>  }
>>>
>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
>>>               update_stats_wait_start_fair(cfs_rq, prev);
>>>               /* Put 'current' back into the tree. */
>>>               __enqueue_entity(cfs_rq, prev);
>>> +             __enqueue_latency(cfs_rq, prev, 0);
>>>               /* in !on_rq case, update occurred at dequeue */
>>>               update_load_avg(cfs_rq, prev, 0);
>>>       }
>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
>>>  {
>>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
>>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
>>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
>>>  #ifdef CONFIG_SMP
>>>       raw_spin_lock_init(&cfs_rq->removed.lock);
>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
>>>       se->my_q = cfs_rq;
>>>
>>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
>>> +     RB_CLEAR_NODE(&se->latency_node);
>>>
>>>       /* guarantee group entities always have weight */
>>>       update_load_set(&se->load, NICE_0_LOAD);
>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
>>>
>>>       for_each_possible_cpu(i) {
>>>               struct sched_entity *se = tg->se[i];
>>> +             struct rq *rq = cpu_rq(i);
>>> +             struct rq_flags rf;
>>> +             bool queued;
>>> +
>>> +             rq_lock_irqsave(rq, &rf);
>>>
>>> +             queued = __dequeue_latency(se->cfs_rq, se);
>>>               WRITE_ONCE(se->latency_offset, latency_offset);
>>> +             if (queued)
>>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
>>> +
>>> +
>>> +             rq_unlock_irqrestore(rq, &rf);
>>>       }
>>>
>>>       mutex_unlock(&shares_mutex);
>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
>>> index 9a2e71231083..21dd309e98a9 100644
>>> --- a/kernel/sched/sched.h
>>> +++ b/kernel/sched/sched.h
>>> @@ -570,6 +570,7 @@ struct cfs_rq {
>>>  #endif
>>>
>>>       struct rb_root_cached   tasks_timeline;
>>> +     struct rb_root_cached   latency_timeline;
>>>
>>>       /*
>>>        * 'curr' points to currently running entity on this cfs_rq.

Re: [PATCH v12 5/8] sched/fair: Take into account latency priority at wakeup

[Shrikanth Hegde]

On 3/2/23 1:13 PM, Vincent Guittot wrote:
> On Wed, 1 Mar 2023 at 20:29, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>>
>>
>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
>>> Take into account the latency priority of a thread when deciding to
>>> preempt the current running thread. We don't want to provide more CPU
>>> bandwidth to a thread but reorder the scheduling to run latency sensitive
>>> task first whenever possible.
>>>
>>> As long as a thread didn't use its bandwidth, it will be able to preempt
>>> the current thread.
>>>
>>> At the opposite, a thread with a low latency priority will preempt current
>>> thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will
>>> wait for the tick to get its sched slice.
>>>
>>>                                    curr vruntime
>>>                                        |
>>>                       sysctl_sched_wakeup_granularity
>>>                                    <-->
>>> ----------------------------------|----|-----------------------|---------------
>>>                                   |    |<--------------------->
>>>                                   |    .  sysctl_sched_latency
>>>                                   |    .
>>> default/current latency entity    |    .
>>>                                   |    .
>>> 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-
>>> se preempts curr at wakeup ------>|<- se doesn't preempt curr -----------------
>>>                                   |    .
>>>                                   |    .
>>>                                   |    .
>>> low latency entity                |    .
>>>                                    ---------------------->|
>>>                                % of sysctl_sched_latency  |
>>> 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-
>>> preempt ------------------------------------------------->|<- do not preempt --
>>>                                   |    .
>>>                                   |    .
>>>                                   |    .
>>> high latency entity               |    .
>>>          |<-----------------------|----.
>>>          | % of sysctl_sched_latency   .
>>> 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1
>>> preempt->|<- se doesn't preempt curr ------------------------------------------
>>>
>>> Tests results of nice latency impact on heavy load like hackbench:
>>>
>>> hackbench -l (2560 / group) -g group
>>> group        latency 0             latency 19
>>> 1            1.378(+/-  1%)      1.337(+/- 1%) + 3%
>>> 4            1.393(+/-  3%)      1.312(+/- 3%) + 6%
>>> 8            1.308(+/-  2%)      1.279(+/- 1%) + 2%
>>> 16           1.347(+/-  1%)      1.317(+/- 1%) + 2%
>>>
>>> hackbench -p -l (2560 / group) -g group
>>> group
>>> 1            1.836(+/- 17%)      1.148(+/- 5%) +37%
>>> 4            1.586(+/-  6%)      1.109(+/- 8%) +30%
>>> 8            1.209(+/-  4%)      0.780(+/- 4%) +35%
>>> 16           0.805(+/-  5%)      0.728(+/- 4%) +10%
>>>
>>> By deacreasing the latency prio, we reduce the number of preemption at
>>> wakeup and help hackbench making progress.
>>>
>>> Test results of nice latency impact on short live load like cyclictest
>>> while competing with heavy load like hackbench:
>>>
>>> hackbench -l 10000 -g $group &
>>> cyclictest --policy other -D 5 -q -n
>>>         latency 0           latency -20
>>> group   min  avg    max     min  avg    max
>>> 0       16    19     29      17   18     29
>>> 1       43   299   7359      63   84   3422
>>> 4       56   449  14806      45   83    284
>>> 8       63   820  51123      63   83    283
>>> 16      64  1326  70684      41  157  26852
>>>
>>> group = 0 means that hackbench is not running.
>>>
>>> The avg is significantly improved with nice latency -20 especially with
>>> large number of groups but min and max remain quite similar. If we add the
>>> histogram parameter to get details of latency, we have :
>>>
>>> hackbench -l 10000 -g 16 &
>>> cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
>>>               latency 0    latency -20
>>> Min Latencies:    64           62
>>> Avg Latencies:  1170          107
>>> Max Latencies: 88069        10417
>>> 50% latencies:   122           86
>>> 75% latencies:   614           91
>>> 85% latencies:   961           94
>>> 90% latencies:  1225           97
>>> 95% latencies:  6120          102
>>> 99% latencies: 18328          159
>>>
>>> With percentile details, we see the benefit of nice latency -20 as
>>> only 1% of the latencies are above 159us whereas the default latency
>>> has got 15% around ~1ms or above and 5% over the 6ms.
>>>
>>> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
>>> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
>>> ---
>>>  include/linux/sched.h      |  4 +++-
>>>  include/linux/sched/prio.h |  9 +++++++++
>>>  init/init_task.c           |  2 +-
>>>  kernel/sched/core.c        | 19 ++++++++++++++-----
>>>  kernel/sched/debug.c       |  2 +-
>>>  kernel/sched/fair.c        | 32 +++++++++++++++++++++++++++-----
>>>  kernel/sched/sched.h       | 11 +++++++++++
>>>  7 files changed, 66 insertions(+), 13 deletions(-)
>>>
>>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>>> index 6c61bde49152..38decae3e156 100644
>>> --- a/include/linux/sched.h
>>> +++ b/include/linux/sched.h
>>> @@ -568,6 +568,8 @@ struct sched_entity {
>>>       /* cached value of my_q->h_nr_running */
>>>       unsigned long                   runnable_weight;
>>>  #endif
>>> +     /* preemption offset in ns */
>>> +     long                            latency_offset;
>>>
>>>  #ifdef CONFIG_SMP
>>>       /*
>>> @@ -784,7 +786,7 @@ struct task_struct {
>>>       int                             static_prio;
>>>       int                             normal_prio;
>>>       unsigned int                    rt_priority;
>>> -     int                             latency_nice;
>>> +     int                             latency_prio;
>>>
>>>       struct sched_entity             se;
>>>       struct sched_rt_entity          rt;
>>> diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h
>>> index bfcd7f1d1e11..be79503d86af 100644
>>> --- a/include/linux/sched/prio.h
>>> +++ b/include/linux/sched/prio.h
>>> @@ -59,5 +59,14 @@ static inline long rlimit_to_nice(long prio)
>>>   * Default tasks should be treated as a task with latency_nice = 0.
>>>   */
>>>  #define DEFAULT_LATENCY_NICE 0
>>> +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2)
>>> +
>>> +/*
>>> + * Convert user-nice values [ -20 ... 0 ... 19 ]
>>> + * to static latency [ 0..39 ],
>>> + * and back.
>>> + */
>>> +#define NICE_TO_LATENCY(nice)        ((nice) + DEFAULT_LATENCY_PRIO)
>>> +#define LATENCY_TO_NICE(prio)        ((prio) - DEFAULT_LATENCY_PRIO)
>>>
>>>  #endif /* _LINUX_SCHED_PRIO_H */
>>> diff --git a/init/init_task.c b/init/init_task.c
>>> index 7dd71dd2d261..071deff8dbd1 100644
>>> --- a/init/init_task.c
>>> +++ b/init/init_task.c
>>> @@ -78,7 +78,7 @@ struct task_struct init_task
>>>       .prio           = MAX_PRIO - 20,
>>>       .static_prio    = MAX_PRIO - 20,
>>>       .normal_prio    = MAX_PRIO - 20,
>>> -     .latency_nice   = DEFAULT_LATENCY_NICE,
>>> +     .latency_prio   = DEFAULT_LATENCY_PRIO,
>>>       .policy         = SCHED_NORMAL,
>>>       .cpus_ptr       = &init_task.cpus_mask,
>>>       .user_cpus_ptr  = NULL,
>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>> index d327614c70b0..d5b7e237d79b 100644
>>> --- a/kernel/sched/core.c
>>> +++ b/kernel/sched/core.c
>>> @@ -1285,6 +1285,11 @@ static void set_load_weight(struct task_struct *p, bool update_load)
>>>       }
>>>  }
>>>
>>> +static void set_latency_offset(struct task_struct *p)
>>> +{
>>> +     p->se.latency_offset = calc_latency_offset(p->latency_prio);
>>> +}
>>> +
>>>  #ifdef CONFIG_UCLAMP_TASK
>>>  /*
>>>   * Serializes updates of utilization clamp values
>>> @@ -4681,7 +4686,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
>>>               p->prio = p->normal_prio = p->static_prio;
>>>               set_load_weight(p, false);
>>>
>>> -             p->latency_nice = DEFAULT_LATENCY_NICE;
>>> +             p->latency_prio = NICE_TO_LATENCY(0);
>>> +             set_latency_offset(p);
>>> +
>>>               /*
>>>                * We don't need the reset flag anymore after the fork. It has
>>>                * fulfilled its duty:
>>> @@ -7449,8 +7456,10 @@ static void __setscheduler_params(struct task_struct *p,
>>>  static void __setscheduler_latency(struct task_struct *p,
>>>               const struct sched_attr *attr)
>>>  {
>>> -     if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
>>> -             p->latency_nice = attr->sched_latency_nice;
>>> +     if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
>>> +             p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice);
>>> +             set_latency_offset(p);
>>> +     }
>>>  }
>>>
>>>  /*
>>> @@ -7635,7 +7644,7 @@ static int __sched_setscheduler(struct task_struct *p,
>>>               if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
>>>                       goto change;
>>>               if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
>>> -                 attr->sched_latency_nice != p->latency_nice)
>>> +                 attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio))
>>>                       goto change;
>>>
>>>               p->sched_reset_on_fork = reset_on_fork;
>>> @@ -8176,7 +8185,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
>>>       get_params(p, &kattr);
>>>       kattr.sched_flags &= SCHED_FLAG_ALL;
>>>
>>> -     kattr.sched_latency_nice = p->latency_nice;
>>> +     kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio);
>>>
>>>  #ifdef CONFIG_UCLAMP_TASK
>>>       /*
>>> diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
>>> index 68be7a3e42a3..b3922184af91 100644
>>> --- a/kernel/sched/debug.c
>>> +++ b/kernel/sched/debug.c
>>> @@ -1043,7 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
>>>  #endif
>>>       P(policy);
>>>       P(prio);
>>> -     P(latency_nice);
>>> +     P(latency_prio);
>> /proc/<pid>/sched  should update if the latency values are updated
>> for the cgroup right? That doesn't seem to happen.
> No It's not. The cgroup latency_nice value applies the the
> sched_entity of the group in which the task are scheduled


If this isnt per task, what is the use case of printing latency_prio 
/proc/<pid>/sched?

>
>> #cd /sys/fs/cgroup/cpu
>> # echo -20 >  task1/cpu.latency.nice
>> # cat task1/cgroup.procs
>> 1897
>> 1998
>> 1999
>> # cat /proc/1999/sched | grep latency
>> latency_prio                                 :                   20
>> # echo 0 >  task1/cpu.latency.nice
>> # cat /proc/1999/sched | grep latency
>> latency_prio                                 :                   20
>> # echo 19 >  task1/cpu.latency.nice
>> # cat /proc/1999/sched | grep latency
>> latency_prio                                 :                   20
>>
>>
>>>       if (task_has_dl_policy(p)) {
>>>               P(dl.runtime);
>>>               P(dl.deadline);
>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>> index 81bef11eb660..414b6243208b 100644
>>> --- a/kernel/sched/fair.c
>>> +++ b/kernel/sched/fair.c
>>> @@ -4877,6 +4877,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>               update_idle_cfs_rq_clock_pelt(cfs_rq);
>>>  }
>>>
>>> +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se);
>>> +
>>>  /*
>>>   * Preempt the current task with a newly woken task if needed:
>>>   */
>>> @@ -4885,7 +4887,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>  {
>>>       unsigned long ideal_runtime, delta_exec;
>>>       struct sched_entity *se;
>>> -     s64 delta;
>>> +     s64 delta, offset;
>>>
>>>       /*
>>>        * When many tasks blow up the sched_period; it is possible that
>>> @@ -4916,10 +4918,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>       se = __pick_first_entity(cfs_rq);
>>>       delta = curr->vruntime - se->vruntime;
>>>
>>> -     if (delta < 0)
>>> +     offset = wakeup_latency_gran(curr, se);
>>> +     if (delta < offset)
>>>               return;
>>>
>>> -     if (delta > ideal_runtime)
>>> +     if ((delta > ideal_runtime) ||
>>> +         (delta > get_latency_max()))
>>>               resched_curr(rq_of(cfs_rq));
>>>  }
>>>
>>> @@ -7662,6 +7666,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
>>>  }
>>>  #endif /* CONFIG_SMP */
>>>
>>> +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se)
>>> +{
>>> +     long latency_offset = se->latency_offset;
>>> +
>>> +     /*
>>> +      * A negative latency offset means that the sched_entity has latency
>>> +      * requirement that needs to be evaluated versus other entity.
>>> +      * Otherwise, use the latency weight to evaluate how much scheduling
>>> +      * delay is acceptable by se.
>>> +      */
>>> +     if ((latency_offset < 0) || (curr->latency_offset < 0))
>>> +             latency_offset -= curr->latency_offset;
>>> +     latency_offset = min_t(long, latency_offset, get_latency_max());
>>> +
>>> +     return latency_offset;
>>> +}
>>> +
>>>  static unsigned long wakeup_gran(struct sched_entity *se)
>>>  {
>>>       unsigned long gran = sysctl_sched_wakeup_granularity;
>>> @@ -7700,11 +7721,12 @@ static int
>>>  wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>  {
>>>       s64 gran, vdiff = curr->vruntime - se->vruntime;
>>> +     s64 offset = wakeup_latency_gran(curr, se);
>>>
>>> -     if (vdiff <= 0)
>>> +     if (vdiff < offset)
>>>               return -1;
>>>
>>> -     gran = wakeup_gran(se);
>>> +     gran = offset + wakeup_gran(se);
>>>
>>>       /*
>>>        * At wake up, the vruntime of a task is capped to not be older than
>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
>>> index 51ba0af7fb27..3f42f86105d4 100644
>>> --- a/kernel/sched/sched.h
>>> +++ b/kernel/sched/sched.h
>>> @@ -2494,6 +2494,17 @@ static inline unsigned long get_sleep_latency(bool idle)
>>>       return thresh;
>>>  }
>>>
>>> +/*
>>> + * Calculate the latency offset for a priority level.
>>> + * We use a linear mapping of the priority in the range:
>>> + *     [-sysctl_sched_latency:sysctl_sched_latency]
>>> + */
>>> +static inline long calc_latency_offset(int prio)
>>> +{
>>> +     return (long)get_sleep_latency(false) * LATENCY_TO_NICE(prio) /
>>> +                     (LATENCY_NICE_WIDTH/2);
>>> +}
>>> +
>>>  static inline unsigned long get_latency_max(void)
>>>  {
>>>       unsigned long thresh = get_sleep_latency(false);

Re: [PATCH v12 5/8] sched/fair: Take into account latency priority at wakeup

[Vincent Guittot]

On Thu, 2 Mar 2023 at 12:04, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>
>
>
> On 3/2/23 1:13 PM, Vincent Guittot wrote:
> > On Wed, 1 Mar 2023 at 20:29, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>
> >>
> >> On 2/24/23 3:04 PM, Vincent Guittot wrote:
> >>> Take into account the latency priority of a thread when deciding to
> >>> preempt the current running thread. We don't want to provide more CPU
> >>> bandwidth to a thread but reorder the scheduling to run latency sensitive
> >>> task first whenever possible.
> >>>
> >>> As long as a thread didn't use its bandwidth, it will be able to preempt
> >>> the current thread.
> >>>
> >>> At the opposite, a thread with a low latency priority will preempt current
> >>> thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will
> >>> wait for the tick to get its sched slice.
> >>>
> >>>                                    curr vruntime
> >>>                                        |
> >>>                       sysctl_sched_wakeup_granularity
> >>>                                    <-->
> >>> ----------------------------------|----|-----------------------|---------------
> >>>                                   |    |<--------------------->
> >>>                                   |    .  sysctl_sched_latency
> >>>                                   |    .
> >>> default/current latency entity    |    .
> >>>                                   |    .
> >>> 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-
> >>> se preempts curr at wakeup ------>|<- se doesn't preempt curr -----------------
> >>>                                   |    .
> >>>                                   |    .
> >>>                                   |    .
> >>> low latency entity                |    .
> >>>                                    ---------------------->|
> >>>                                % of sysctl_sched_latency  |
> >>> 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-
> >>> preempt ------------------------------------------------->|<- do not preempt --
> >>>                                   |    .
> >>>                                   |    .
> >>>                                   |    .
> >>> high latency entity               |    .
> >>>          |<-----------------------|----.
> >>>          | % of sysctl_sched_latency   .
> >>> 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1
> >>> preempt->|<- se doesn't preempt curr ------------------------------------------
> >>>
> >>> Tests results of nice latency impact on heavy load like hackbench:
> >>>
> >>> hackbench -l (2560 / group) -g group
> >>> group        latency 0             latency 19
> >>> 1            1.378(+/-  1%)      1.337(+/- 1%) + 3%
> >>> 4            1.393(+/-  3%)      1.312(+/- 3%) + 6%
> >>> 8            1.308(+/-  2%)      1.279(+/- 1%) + 2%
> >>> 16           1.347(+/-  1%)      1.317(+/- 1%) + 2%
> >>>
> >>> hackbench -p -l (2560 / group) -g group
> >>> group
> >>> 1            1.836(+/- 17%)      1.148(+/- 5%) +37%
> >>> 4            1.586(+/-  6%)      1.109(+/- 8%) +30%
> >>> 8            1.209(+/-  4%)      0.780(+/- 4%) +35%
> >>> 16           0.805(+/-  5%)      0.728(+/- 4%) +10%
> >>>
> >>> By deacreasing the latency prio, we reduce the number of preemption at
> >>> wakeup and help hackbench making progress.
> >>>
> >>> Test results of nice latency impact on short live load like cyclictest
> >>> while competing with heavy load like hackbench:
> >>>
> >>> hackbench -l 10000 -g $group &
> >>> cyclictest --policy other -D 5 -q -n
> >>>         latency 0           latency -20
> >>> group   min  avg    max     min  avg    max
> >>> 0       16    19     29      17   18     29
> >>> 1       43   299   7359      63   84   3422
> >>> 4       56   449  14806      45   83    284
> >>> 8       63   820  51123      63   83    283
> >>> 16      64  1326  70684      41  157  26852
> >>>
> >>> group = 0 means that hackbench is not running.
> >>>
> >>> The avg is significantly improved with nice latency -20 especially with
> >>> large number of groups but min and max remain quite similar. If we add the
> >>> histogram parameter to get details of latency, we have :
> >>>
> >>> hackbench -l 10000 -g 16 &
> >>> cyclictest --policy other -D 5 -q -n  -H 20000 --histfile data.txt
> >>>               latency 0    latency -20
> >>> Min Latencies:    64           62
> >>> Avg Latencies:  1170          107
> >>> Max Latencies: 88069        10417
> >>> 50% latencies:   122           86
> >>> 75% latencies:   614           91
> >>> 85% latencies:   961           94
> >>> 90% latencies:  1225           97
> >>> 95% latencies:  6120          102
> >>> 99% latencies: 18328          159
> >>>
> >>> With percentile details, we see the benefit of nice latency -20 as
> >>> only 1% of the latencies are above 159us whereas the default latency
> >>> has got 15% around ~1ms or above and 5% over the 6ms.
> >>>
> >>> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> >>> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> >>> ---
> >>>  include/linux/sched.h      |  4 +++-
> >>>  include/linux/sched/prio.h |  9 +++++++++
> >>>  init/init_task.c           |  2 +-
> >>>  kernel/sched/core.c        | 19 ++++++++++++++-----
> >>>  kernel/sched/debug.c       |  2 +-
> >>>  kernel/sched/fair.c        | 32 +++++++++++++++++++++++++++-----
> >>>  kernel/sched/sched.h       | 11 +++++++++++
> >>>  7 files changed, 66 insertions(+), 13 deletions(-)
> >>>
> >>> diff --git a/include/linux/sched.h b/include/linux/sched.h
> >>> index 6c61bde49152..38decae3e156 100644
> >>> --- a/include/linux/sched.h
> >>> +++ b/include/linux/sched.h
> >>> @@ -568,6 +568,8 @@ struct sched_entity {
> >>>       /* cached value of my_q->h_nr_running */
> >>>       unsigned long                   runnable_weight;
> >>>  #endif
> >>> +     /* preemption offset in ns */
> >>> +     long                            latency_offset;
> >>>
> >>>  #ifdef CONFIG_SMP
> >>>       /*
> >>> @@ -784,7 +786,7 @@ struct task_struct {
> >>>       int                             static_prio;
> >>>       int                             normal_prio;
> >>>       unsigned int                    rt_priority;
> >>> -     int                             latency_nice;
> >>> +     int                             latency_prio;
> >>>
> >>>       struct sched_entity             se;
> >>>       struct sched_rt_entity          rt;
> >>> diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h
> >>> index bfcd7f1d1e11..be79503d86af 100644
> >>> --- a/include/linux/sched/prio.h
> >>> +++ b/include/linux/sched/prio.h
> >>> @@ -59,5 +59,14 @@ static inline long rlimit_to_nice(long prio)
> >>>   * Default tasks should be treated as a task with latency_nice = 0.
> >>>   */
> >>>  #define DEFAULT_LATENCY_NICE 0
> >>> +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2)
> >>> +
> >>> +/*
> >>> + * Convert user-nice values [ -20 ... 0 ... 19 ]
> >>> + * to static latency [ 0..39 ],
> >>> + * and back.
> >>> + */
> >>> +#define NICE_TO_LATENCY(nice)        ((nice) + DEFAULT_LATENCY_PRIO)
> >>> +#define LATENCY_TO_NICE(prio)        ((prio) - DEFAULT_LATENCY_PRIO)
> >>>
> >>>  #endif /* _LINUX_SCHED_PRIO_H */
> >>> diff --git a/init/init_task.c b/init/init_task.c
> >>> index 7dd71dd2d261..071deff8dbd1 100644
> >>> --- a/init/init_task.c
> >>> +++ b/init/init_task.c
> >>> @@ -78,7 +78,7 @@ struct task_struct init_task
> >>>       .prio           = MAX_PRIO - 20,
> >>>       .static_prio    = MAX_PRIO - 20,
> >>>       .normal_prio    = MAX_PRIO - 20,
> >>> -     .latency_nice   = DEFAULT_LATENCY_NICE,
> >>> +     .latency_prio   = DEFAULT_LATENCY_PRIO,
> >>>       .policy         = SCHED_NORMAL,
> >>>       .cpus_ptr       = &init_task.cpus_mask,
> >>>       .user_cpus_ptr  = NULL,
> >>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >>> index d327614c70b0..d5b7e237d79b 100644
> >>> --- a/kernel/sched/core.c
> >>> +++ b/kernel/sched/core.c
> >>> @@ -1285,6 +1285,11 @@ static void set_load_weight(struct task_struct *p, bool update_load)
> >>>       }
> >>>  }
> >>>
> >>> +static void set_latency_offset(struct task_struct *p)
> >>> +{
> >>> +     p->se.latency_offset = calc_latency_offset(p->latency_prio);
> >>> +}
> >>> +
> >>>  #ifdef CONFIG_UCLAMP_TASK
> >>>  /*
> >>>   * Serializes updates of utilization clamp values
> >>> @@ -4681,7 +4686,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
> >>>               p->prio = p->normal_prio = p->static_prio;
> >>>               set_load_weight(p, false);
> >>>
> >>> -             p->latency_nice = DEFAULT_LATENCY_NICE;
> >>> +             p->latency_prio = NICE_TO_LATENCY(0);
> >>> +             set_latency_offset(p);
> >>> +
> >>>               /*
> >>>                * We don't need the reset flag anymore after the fork. It has
> >>>                * fulfilled its duty:
> >>> @@ -7449,8 +7456,10 @@ static void __setscheduler_params(struct task_struct *p,
> >>>  static void __setscheduler_latency(struct task_struct *p,
> >>>               const struct sched_attr *attr)
> >>>  {
> >>> -     if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE)
> >>> -             p->latency_nice = attr->sched_latency_nice;
> >>> +     if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) {
> >>> +             p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice);
> >>> +             set_latency_offset(p);
> >>> +     }
> >>>  }
> >>>
> >>>  /*
> >>> @@ -7635,7 +7644,7 @@ static int __sched_setscheduler(struct task_struct *p,
> >>>               if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)
> >>>                       goto change;
> >>>               if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE &&
> >>> -                 attr->sched_latency_nice != p->latency_nice)
> >>> +                 attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio))
> >>>                       goto change;
> >>>
> >>>               p->sched_reset_on_fork = reset_on_fork;
> >>> @@ -8176,7 +8185,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr,
> >>>       get_params(p, &kattr);
> >>>       kattr.sched_flags &= SCHED_FLAG_ALL;
> >>>
> >>> -     kattr.sched_latency_nice = p->latency_nice;
> >>> +     kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio);
> >>>
> >>>  #ifdef CONFIG_UCLAMP_TASK
> >>>       /*
> >>> diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
> >>> index 68be7a3e42a3..b3922184af91 100644
> >>> --- a/kernel/sched/debug.c
> >>> +++ b/kernel/sched/debug.c
> >>> @@ -1043,7 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
> >>>  #endif
> >>>       P(policy);
> >>>       P(prio);
> >>> -     P(latency_nice);
> >>> +     P(latency_prio);
> >> /proc/<pid>/sched  should update if the latency values are updated
> >> for the cgroup right? That doesn't seem to happen.
> > No It's not. The cgroup latency_nice value applies the the
> > sched_entity of the group in which the task are scheduled
>
>
> If this isnt per task, what is the use case of printing latency_prio
> /proc/<pid>/sched?


/proc/<pid>/sched shows the properties of task <pid>  and in
particular its latency_prio

The cgroup interface updates the properties of the cgroup that is then
used by its group_entity that is scheduled in the parent cfs

If you want to set the latency_prio of a task, you need to use the
syscall sched_setattr()

>
> >
> >> #cd /sys/fs/cgroup/cpu
> >> # echo -20 >  task1/cpu.latency.nice
> >> # cat task1/cgroup.procs
> >> 1897
> >> 1998
> >> 1999
> >> # cat /proc/1999/sched | grep latency
> >> latency_prio                                 :                   20
> >> # echo 0 >  task1/cpu.latency.nice
> >> # cat /proc/1999/sched | grep latency
> >> latency_prio                                 :                   20
> >> # echo 19 >  task1/cpu.latency.nice
> >> # cat /proc/1999/sched | grep latency
> >> latency_prio                                 :                   20
> >>
> >>
> >>>       if (task_has_dl_policy(p)) {
> >>>               P(dl.runtime);
> >>>               P(dl.deadline);
> >>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >>> index 81bef11eb660..414b6243208b 100644
> >>> --- a/kernel/sched/fair.c
> >>> +++ b/kernel/sched/fair.c
> >>> @@ -4877,6 +4877,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>               update_idle_cfs_rq_clock_pelt(cfs_rq);
> >>>  }
> >>>
> >>> +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se);
> >>> +
> >>>  /*
> >>>   * Preempt the current task with a newly woken task if needed:
> >>>   */
> >>> @@ -4885,7 +4887,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>  {
> >>>       unsigned long ideal_runtime, delta_exec;
> >>>       struct sched_entity *se;
> >>> -     s64 delta;
> >>> +     s64 delta, offset;
> >>>
> >>>       /*
> >>>        * When many tasks blow up the sched_period; it is possible that
> >>> @@ -4916,10 +4918,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>       se = __pick_first_entity(cfs_rq);
> >>>       delta = curr->vruntime - se->vruntime;
> >>>
> >>> -     if (delta < 0)
> >>> +     offset = wakeup_latency_gran(curr, se);
> >>> +     if (delta < offset)
> >>>               return;
> >>>
> >>> -     if (delta > ideal_runtime)
> >>> +     if ((delta > ideal_runtime) ||
> >>> +         (delta > get_latency_max()))
> >>>               resched_curr(rq_of(cfs_rq));
> >>>  }
> >>>
> >>> @@ -7662,6 +7666,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
> >>>  }
> >>>  #endif /* CONFIG_SMP */
> >>>
> >>> +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se)
> >>> +{
> >>> +     long latency_offset = se->latency_offset;
> >>> +
> >>> +     /*
> >>> +      * A negative latency offset means that the sched_entity has latency
> >>> +      * requirement that needs to be evaluated versus other entity.
> >>> +      * Otherwise, use the latency weight to evaluate how much scheduling
> >>> +      * delay is acceptable by se.
> >>> +      */
> >>> +     if ((latency_offset < 0) || (curr->latency_offset < 0))
> >>> +             latency_offset -= curr->latency_offset;
> >>> +     latency_offset = min_t(long, latency_offset, get_latency_max());
> >>> +
> >>> +     return latency_offset;
> >>> +}
> >>> +
> >>>  static unsigned long wakeup_gran(struct sched_entity *se)
> >>>  {
> >>>       unsigned long gran = sysctl_sched_wakeup_granularity;
> >>> @@ -7700,11 +7721,12 @@ static int
> >>>  wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >>>  {
> >>>       s64 gran, vdiff = curr->vruntime - se->vruntime;
> >>> +     s64 offset = wakeup_latency_gran(curr, se);
> >>>
> >>> -     if (vdiff <= 0)
> >>> +     if (vdiff < offset)
> >>>               return -1;
> >>>
> >>> -     gran = wakeup_gran(se);
> >>> +     gran = offset + wakeup_gran(se);
> >>>
> >>>       /*
> >>>        * At wake up, the vruntime of a task is capped to not be older than
> >>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> >>> index 51ba0af7fb27..3f42f86105d4 100644
> >>> --- a/kernel/sched/sched.h
> >>> +++ b/kernel/sched/sched.h
> >>> @@ -2494,6 +2494,17 @@ static inline unsigned long get_sleep_latency(bool idle)
> >>>       return thresh;
> >>>  }
> >>>
> >>> +/*
> >>> + * Calculate the latency offset for a priority level.
> >>> + * We use a linear mapping of the priority in the range:
> >>> + *     [-sysctl_sched_latency:sysctl_sched_latency]
> >>> + */
> >>> +static inline long calc_latency_offset(int prio)
> >>> +{
> >>> +     return (long)get_sleep_latency(false) * LATENCY_TO_NICE(prio) /
> >>> +                     (LATENCY_NICE_WIDTH/2);
> >>> +}
> >>> +
> >>>  static inline unsigned long get_latency_max(void)
> >>>  {
> >>>       unsigned long thresh = get_sleep_latency(false);
>

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Vincent Guittot]

On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>
>
> On 3/2/23 1:20 PM, Vincent Guittot wrote:
> > On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>
> >> On 2/24/23 3:04 PM, Vincent Guittot wrote:
> >>> Add a rb tree for latency sensitive entities so we can schedule the most
> >>> sensitive one first even when it failed to preempt current at wakeup or
> >>> when it got quickly preempted by another entity of higher priority.
> >>>
> >>> In order to keep fairness, the latency is used once at wakeup to get a
> >>> minimum slice and not during the following scheduling slice to prevent
> >>> long running entity to got more running time than allocated to his nice
> >>> priority.
> >>>
> >>> The rb tree enables to cover the last corner case where latency
> >>> sensitive entity can't got schedule quickly after the wakeup.
> >>>
> >>> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
> >>> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
> >>> ---
> >>>  include/linux/sched.h |   1 +
> >>>  kernel/sched/core.c   |   1 +
> >>>  kernel/sched/fair.c   | 109 ++++++++++++++++++++++++++++++++++++++++--
> >>>  kernel/sched/sched.h  |   1 +
> >>>  4 files changed, 109 insertions(+), 3 deletions(-)
> >>>
> >>> diff --git a/include/linux/sched.h b/include/linux/sched.h
> >>> index 38decae3e156..41bb92be5ecc 100644
> >>> --- a/include/linux/sched.h
> >>> +++ b/include/linux/sched.h
> >>> @@ -548,6 +548,7 @@ struct sched_entity {
> >>>       /* For load-balancing: */
> >>>       struct load_weight              load;
> >>>       struct rb_node                  run_node;
> >>> +     struct rb_node                  latency_node;
> >> Ran pahole to see if the frequently accessed fields change across cachelines.
> >> There is no change in cachelines of task_struct, whereas sched_entity differs
> >> due to latency_node.  Maybe the latency_node could be placed after
> >> runnable_weight as there is space available in that cacheline.
> > I will run some test on my system to confimr your results but we can
> > move latency_node field if it helps cache hit stats
> >
> >>
> >> 6.2
> >> #pahole sched_entity
> >> struct sched_entity {
> >>         struct load_weight         load;                 /*     0    16 */
> >>
> >>         /* XXX last struct has 4 bytes of padding */
> >>
> >>         struct rb_node             run_node;             /*    16    24 */
> >>         struct list_head           group_node;           /*    40    16 */
> >>         unsigned int               on_rq;                /*    56     4 */
> >>
> >>         /* XXX 4 bytes hole, try to pack */
> >>
> >>         u64                        exec_start;           /*    64     8 */
> >>         u64                        sum_exec_runtime;     /*    72     8 */
> >>         u64                        vruntime;             /*    80     8 */
> >>         u64                        prev_sum_exec_runtime; /*    88     8 */
> >>         u64                        nr_migrations;        /*    96     8 */
> >>         int                        depth;                /*   104     4 */
> >>
> >>         /* XXX 4 bytes hole, try to pack */
> >>
> >>         struct sched_entity *      parent;               /*   112     8 */
> >>         struct cfs_rq *            cfs_rq;               /*   120     8 */
> >>         /* --- cacheline 1 boundary (128 bytes) --- */
> >>         struct cfs_rq *            my_q;                 /*   128     8 */
> >>         long unsigned int          runnable_weight;      /*   136     8 */
> >>
> >>         /* XXX 112 bytes hole, try to pack */
> >>
> >>         /* --- cacheline 2 boundary (256 bytes) --- */
> >>         struct sched_avg           avg;                  /*   256   128 */
> >>
> >>
> >> 6.2 + V12 patch
> >> #pahole sched_entity
> >> struct sched_entity {
> >>         struct load_weight         load;                 /*     0    16 */
> >>
> >>         /* XXX last struct has 4 bytes of padding */
> >>
> >>         struct rb_node             run_node;             /*    16    24 */
> >>         struct rb_node             latency_node;         /*    40    24 */
> >>         struct list_head           group_node;           /*    64    16 */
> >>         unsigned int               on_rq;                /*    80     4 */
> >>
> >>         /* XXX 4 bytes hole, try to pack */
> >>
> >>         u64                        exec_start;           /*    88     8 */
> >>         u64                        sum_exec_runtime;     /*    96     8 */
> >>         u64                        vruntime;             /*   104     8 */
> >>         u64                        prev_sum_exec_runtime; /*   112     8 */
> >>         u64                        nr_migrations;        /*   120     8 */
> >>         /* --- cacheline 1 boundary (128 bytes) --- */
> >>         int                        depth;                /*   128     4 */
> >>
> >>         /* XXX 4 bytes hole, try to pack */
> >>
> >>         struct sched_entity *      parent;               /*   136     8 */
> >>         struct cfs_rq *            cfs_rq;               /*   144     8 */
> >>         struct cfs_rq *            my_q;                 /*   152     8 */
> >>
> >>
> >> 6.2 + V12 patch + Re-shuffle of latency_node
> >> #pahole sched_entity
> >> struct sched_entity {
> >>         struct load_weight         load;                 /*     0    16 */
> >>
> >>         /* XXX last struct has 4 bytes of padding */
> >>
> >>         struct rb_node             run_node;             /*    16    24 */
> >>         struct list_head           group_node;           /*    40    16 */
> >>         unsigned int               on_rq;                /*    56     4 */
> >>
> >>         /* XXX 4 bytes hole, try to pack */
> >>
> >>         u64                        exec_start;           /*    64     8 */
> >>         u64                        sum_exec_runtime;     /*    72     8 */
> >>         u64                        vruntime;             /*    80     8 */
> >>         u64                        prev_sum_exec_runtime; /*    88     8 */
> >>         u64                        nr_migrations;        /*    96     8 */
> >>         int                        depth;                /*   104     4 */
> >>
> >>         /* XXX 4 bytes hole, try to pack */
> >>
> >>         struct sched_entity *      parent;               /*   112     8 */
> >>         struct cfs_rq *            cfs_rq;               /*   120     8 */
> >>         /* --- cacheline 1 boundary (128 bytes) --- */
> >>         struct cfs_rq *            my_q;                 /*   128     8 */
> >>         long unsigned int          runnable_weight;      /*   136     8 */
> >>         struct rb_node             latency_node;         /*   144    24 */
> >>         long int                   latency_offset;       /*   168     8 */
> >>
> >>         /* XXX 80 bytes hole, try to pack */
> >>
> >>         /* --- cacheline 2 boundary (256 bytes) --- */
> >>
> >>
> >>
> >> diff --git a/include/linux/sched.h b/include/linux/sched.h
> >> index a2b52cf5e1bb..1e93aaaeead2 100644
> >> --- a/include/linux/sched.h
> >> +++ b/include/linux/sched.h
> >> @@ -548,7 +548,6 @@ struct sched_entity {
> >>         /* For load-balancing: */
> >>         struct load_weight              load;
> >>         struct rb_node                  run_node;
> >> -       struct rb_node                  latency_node;
> >>         struct list_head                group_node;
> >>         unsigned int                    on_rq;
> >>
> >> @@ -569,6 +568,7 @@ struct sched_entity {
> >>         /* cached value of my_q->h_nr_running */
> >>         unsigned long                   runnable_weight;
> >>  #endif
> >> +       struct rb_node                  latency_node;
> >>         /* preemption offset in ns */
> >>         long                            latency_offset;
> >>
> >>
> >> Ran the schbench and hackbench with this patch series. Here comparison is
> >> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
> >> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
> >> and other is running these benchmarks (group2). Set the latency nice
> >> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
> >> Total of 96 CPU.
> >>
> >> schbench gets lower latency compared to stabletree. Whereas hackbench seems
> >> to regress under this case. Maybe i am doing something wrong. I will re-run
> >> and attach the numbers to series.
> >> Please suggest if any variation in the test i need to try.
> > hackbench takes advanatge of a latency nice 19 as it mainly wants to
> > run longer slice to move forward rather than preempting others all the
> > time
>
> hackbench still seems to regress in different latency nice values compared to
> baseline of 6.2 in this case. up to 50% in some cases.
>
> 12 core powerpc system  with SMT=8 i.e 96 CPU
> running 2 CPU cgroups. No quota assigned.
> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
> latency is not changed for this cgroup.
> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
> nice values of 0, -20 and 19.

According to your other emails, you are using the cgroup interface and
not the task's one. Do I get it right ?

I haven't run test such tests in a cgroup but at least the test with
latency_nice == 0 should not make any noticeable difference. Does this
include the re-arrange patch that you have proposed previously ?

Also, the tests that you did on v6, gave better result.
https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@linux.vnet.ibm.com/

Are you running same tests or you changed something in the mean time ?

>
> Numbers are average of 10 runs in each case. Time is in seconds
>
> type       groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
>                   |            | lat nice=0    | lat nice=-20| lat nice=+19
>                   |            |               |             |
> Process       10  |   0.36     |     0.41      |    0.43     |    0.42
> Process       20  |   0.62     |     0.76      |    0.75     |    0.75
> Process       30  |   0.87     |     1.05      |    1.04     |    1.06
> Process       40  |   1.13     |     1.34      |    1.33     |    1.33
> Process       50  |   1.38     |     1.62      |    1.66     |    1.63
> Process       60  |   1.64     |     1.91      |    1.97     |    1.90
> thread        10  |   0.35     |     0.41      |    0.44     |    0.42
> thread        20  |   0.64     |     0.78      |    0.77     |    0.79
> Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
> Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
> Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
> Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
> Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
> Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
> thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
> thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55
>
>
>
> >> Re-arrange seems to help the patch series by avoiding an cacheline miss.
> >>
> >> =========================
> >> schbench
> >> =========================
> >>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
> >> 1 Thread
> >>   50.0th:        9.00  |    9.00        |        9.50
> >>   75.0th:       10.50  |   10.00        |        9.50
> >>   90.0th:       11.00  |   11.00        |       10.50
> >>   95.0th:       11.00  |   11.00        |       11.00
> >>   99.0th:       11.50  |   11.50        |       11.50
> >>   99.5th:       12.50  |   12.00        |       12.00
> >>   99.9th:       14.50  |   13.50        |       12.00
> >> 2 Threads
> >>   50.0th:        9.50  |    9.50        |        8.50
> >>   75.0th:       11.00  |   10.50        |        9.50
> >>   90.0th:       13.50  |   11.50        |       10.50
> >>   95.0th:       14.00  |   12.00        |       11.00
> >>   99.0th:       15.50  |   13.50        |       12.00
> >>   99.5th:       16.00  |   14.00        |       12.00
> >>   99.9th:       17.00  |   16.00        |       16.50
> >> 4 Threads
> >>   50.0th:       11.50  |   11.50        |       10.50
> >>   75.0th:       13.50  |   12.50        |       12.50
> >>   90.0th:       15.50  |   14.50        |       14.00
> >>   95.0th:       16.50  |   15.50        |       14.50
> >>   99.0th:       20.00  |   17.50        |       16.50
> >>   99.5th:       20.50  |   18.50        |       17.00
> >>   99.9th:       22.50  |   21.00        |       19.00
> >> 8 Threads
> >>   50.0th:       14.00  |   14.00        |       14.00
> >>   75.0th:       16.00  |   16.00        |       16.00
> >>   90.0th:       18.00  |   18.00        |       17.50
> >>   95.0th:       18.50  |   18.50        |       18.50
> >>   99.0th:       20.00  |   20.00        |       20.00
> >>   99.5th:       20.50  |   21.50        |       21.00
> >>   99.9th:       22.50  |   23.50        |       23.00
> >> 16 Threads
> >>   50.0th:       19.00  |   18.50        |       19.00
> >>   75.0th:       23.00  |   22.50        |       23.00
> >>   90.0th:       25.00  |   25.50        |       25.00
> >>   95.0th:       26.50  |   26.50        |       26.00
> >>   99.0th:       28.50  |   29.00        |       28.50
> >>   99.5th:       31.00  |   30.00        |       30.00
> >>   99.9th:     5626.00  | 4761.50        |       32.50
> >> 32 Threads
> >>   50.0th:       27.00  |   27.50        |       29.00
> >>   75.0th:       35.50  |   36.50        |       38.50
> >>   90.0th:       42.00  |   44.00        |       50.50
> >>   95.0th:      447.50  | 2959.00        |     8544.00
> >>   99.0th:     7372.00  | 17032.00       |    19136.00
> >>   99.5th:    15360.00  | 19808.00       |    20704.00
> >>   99.9th:    20640.00  | 30048.00       |    30048.00
> >>
> >> ====================
> >> hackbench
> >> ====================
> >>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
> >>
> >> Process 10 Time:        0.35    |       0.42        |           0.41
> >> Process 20 Time:        0.61    |       0.76        |           0.76
> >> Process 30 Time:        0.87    |       1.06        |           1.05
> >> thread 10 Time:         0.35    |       0.43        |           0.42
> >> thread 20 Time:         0.66    |       0.79        |           0.78
> >> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
> >> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
> >> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
> >> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
> >> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
> >>
> >>
> >>>       struct list_head                group_node;
> >>>       unsigned int                    on_rq;
> >>>
> >>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >>> index 093cc1af73dc..752fd364216c 100644
> >>> --- a/kernel/sched/core.c
> >>> +++ b/kernel/sched/core.c
> >>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
> >>>       p->se.nr_migrations             = 0;
> >>>       p->se.vruntime                  = 0;
> >>>       INIT_LIST_HEAD(&p->se.group_node);
> >>> +     RB_CLEAR_NODE(&p->se.latency_node);
> >>>
> >>>  #ifdef CONFIG_FAIR_GROUP_SCHED
> >>>       p->se.cfs_rq                    = NULL;
> >>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >>> index 125a6ff53378..e2aeb4511686 100644
> >>> --- a/kernel/sched/fair.c
> >>> +++ b/kernel/sched/fair.c
> >>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> >>>
> >>>       return __node_2_se(last);
> >>>  }
> >>> +#endif
> >>>
> >>> +/**************************************************************
> >>> + * Scheduling class tree data structure manipulation methods:
> >>> + * for latency
> >>> + */
> >>> +
> >>> +static inline bool latency_before(struct sched_entity *a,
> >>> +                             struct sched_entity *b)
> >>> +{
> >>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> >>> +}
> >>> +
> >>> +#define __latency_node_2_se(node) \
> >>> +     rb_entry((node), struct sched_entity, latency_node)
> >>> +
> >>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> >>> +{
> >>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> >>> +}
> >>> +
> >>> +/*
> >>> + * Enqueue an entity into the latency rb-tree:
> >>> + */
> >>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>> +{
> >>> +
> >>> +     /* Only latency sensitive entity can be added to the list */
> >>> +     if (se->latency_offset >= 0)
> >>> +             return;
> >>> +
> >>> +     if (!RB_EMPTY_NODE(&se->latency_node))
> >>> +             return;
> >>> +
> >>> +     /*
> >>> +      * The entity is always added the latency list at wakeup.
> >>> +      * Then, a not waking up entity that is put back in the list after an
> >>> +      * execution time less than sysctl_sched_min_granularity, means that
> >>> +      * the entity has been preempted by a higher sched class or an entity
> >>> +      * with higher latency constraint. In thi case, the entity is also put
> >>> +      * back in the latency list so it gets a chance to run 1st during the
> >>> +      * next slice.
> >>> +      */
> >>> +     if (!(flags & ENQUEUE_WAKEUP)) {
> >>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> >>> +
> >>> +             if (delta_exec >= sysctl_sched_min_granularity)
> >>> +                     return;
> >>> +     }
> >>> +
> >>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> >>> +}
> >>> +
> >>> +/*
> >>> + * Dequeue an entity from the latency rb-tree and return true if it was really
> >>> + * part of the rb-tree:
> >>> + */
> >>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >>> +{
> >>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
> >>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> >>> +             RB_CLEAR_NODE(&se->latency_node);
> >>> +             return true;
> >>> +     }
> >>> +
> >>> +     return false;
> >>> +}
> >>> +
> >>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> >>> +{
> >>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> >>> +
> >>> +     if (!left)
> >>> +             return NULL;
> >>> +
> >>> +     return __latency_node_2_se(left);
> >>> +}
> >>> +
> >>> +#ifdef CONFIG_SCHED_DEBUG
> >>>  /**************************************************************
> >>>   * Scheduling class statistics methods:
> >>>   */
> >>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>       check_schedstat_required();
> >>>       update_stats_enqueue_fair(cfs_rq, se, flags);
> >>>       check_spread(cfs_rq, se);
> >>> -     if (!curr)
> >>> +     if (!curr) {
> >>>               __enqueue_entity(cfs_rq, se);
> >>> +             __enqueue_latency(cfs_rq, se, flags);
> >>> +     }
> >>>       se->on_rq = 1;
> >>>
> >>>       if (cfs_rq->nr_running == 1) {
> >>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>
> >>>       clear_buddies(cfs_rq, se);
> >>>
> >>> -     if (se != cfs_rq->curr)
> >>> +     if (se != cfs_rq->curr) {
> >>>               __dequeue_entity(cfs_rq, se);
> >>> +             __dequeue_latency(cfs_rq, se);
> >>> +     }
> >>>       se->on_rq = 0;
> >>>       account_entity_dequeue(cfs_rq, se);
> >>>
> >>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >>>                */
> >>>               update_stats_wait_end_fair(cfs_rq, se);
> >>>               __dequeue_entity(cfs_rq, se);
> >>> +             __dequeue_latency(cfs_rq, se);
> >>>               update_load_avg(cfs_rq, se, UPDATE_TG);
> >>>       }
> >>>
> >>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
> >>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>  {
> >>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
> >>> -     struct sched_entity *se;
> >>> +     struct sched_entity *latency, *se;
> >>>
> >>>       /*
> >>>        * If curr is set we have to see if its left of the leftmost entity
> >>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>               se = cfs_rq->last;
> >>>       }
> >>>
> >>> +     /* Check for latency sensitive entity waiting for running */
> >>> +     latency = __pick_first_latency(cfs_rq);
> >>> +     if (latency && (latency != se) &&
> >>> +         wakeup_preempt_entity(latency, se) < 1)
> >>> +             se = latency;
> >>> +
> >>>       return se;
> >>>  }
> >>>
> >>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> >>>               update_stats_wait_start_fair(cfs_rq, prev);
> >>>               /* Put 'current' back into the tree. */
> >>>               __enqueue_entity(cfs_rq, prev);
> >>> +             __enqueue_latency(cfs_rq, prev, 0);
> >>>               /* in !on_rq case, update occurred at dequeue */
> >>>               update_load_avg(cfs_rq, prev, 0);
> >>>       }
> >>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> >>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
> >>>  {
> >>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> >>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
> >>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> >>>  #ifdef CONFIG_SMP
> >>>       raw_spin_lock_init(&cfs_rq->removed.lock);
> >>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
> >>>       se->my_q = cfs_rq;
> >>>
> >>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
> >>> +     RB_CLEAR_NODE(&se->latency_node);
> >>>
> >>>       /* guarantee group entities always have weight */
> >>>       update_load_set(&se->load, NICE_0_LOAD);
> >>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
> >>>
> >>>       for_each_possible_cpu(i) {
> >>>               struct sched_entity *se = tg->se[i];
> >>> +             struct rq *rq = cpu_rq(i);
> >>> +             struct rq_flags rf;
> >>> +             bool queued;
> >>> +
> >>> +             rq_lock_irqsave(rq, &rf);
> >>>
> >>> +             queued = __dequeue_latency(se->cfs_rq, se);
> >>>               WRITE_ONCE(se->latency_offset, latency_offset);
> >>> +             if (queued)
> >>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
> >>> +
> >>> +
> >>> +             rq_unlock_irqrestore(rq, &rf);
> >>>       }
> >>>
> >>>       mutex_unlock(&shares_mutex);
> >>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> >>> index 9a2e71231083..21dd309e98a9 100644
> >>> --- a/kernel/sched/sched.h
> >>> +++ b/kernel/sched/sched.h
> >>> @@ -570,6 +570,7 @@ struct cfs_rq {
> >>>  #endif
> >>>
> >>>       struct rb_root_cached   tasks_timeline;
> >>> +     struct rb_root_cached   latency_timeline;
> >>>
> >>>       /*
> >>>        * 'curr' points to currently running entity on this cfs_rq.
>

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Shrikanth Hegde]

On 3/2/23 6:47 PM, Vincent Guittot wrote:
> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>
>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
>>>>> Add a rb tree for latency sensitive entities so we can schedule the most
>>>>> sensitive one first even when it failed to preempt current at wakeup or
>>>>> when it got quickly preempted by another entity of higher priority.
>>>>>
>>>>> In order to keep fairness, the latency is used once at wakeup to get a
>>>>> minimum slice and not during the following scheduling slice to prevent
>>>>> long running entity to got more running time than allocated to his nice
>>>>> priority.
>>>>>
>>>>> The rb tree enables to cover the last corner case where latency
>>>>> sensitive entity can't got schedule quickly after the wakeup.
>>>>>
>>>>> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
>>>>> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
>>>>> ---
>>>>>  include/linux/sched.h |   1 +
>>>>>  kernel/sched/core.c   |   1 +
>>>>>  kernel/sched/fair.c   | 109 ++++++++++++++++++++++++++++++++++++++++--
>>>>>  kernel/sched/sched.h  |   1 +
>>>>>  4 files changed, 109 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>>>>> index 38decae3e156..41bb92be5ecc 100644
>>>>> --- a/include/linux/sched.h
>>>>> +++ b/include/linux/sched.h
>>>>> @@ -548,6 +548,7 @@ struct sched_entity {
>>>>>       /* For load-balancing: */
>>>>>       struct load_weight              load;
>>>>>       struct rb_node                  run_node;
>>>>> +     struct rb_node                  latency_node;
>>>> Ran pahole to see if the frequently accessed fields change across cachelines.
>>>> There is no change in cachelines of task_struct, whereas sched_entity differs
>>>> due to latency_node.  Maybe the latency_node could be placed after
>>>> runnable_weight as there is space available in that cacheline.
>>> I will run some test on my system to confimr your results but we can
>>> move latency_node field if it helps cache hit stats
>>>
>>>> 6.2
>>>> #pahole sched_entity
>>>> struct sched_entity {
>>>>         struct load_weight         load;                 /*     0    16 */
>>>>
>>>>         /* XXX last struct has 4 bytes of padding */
>>>>
>>>>         struct rb_node             run_node;             /*    16    24 */
>>>>         struct list_head           group_node;           /*    40    16 */
>>>>         unsigned int               on_rq;                /*    56     4 */
>>>>
>>>>         /* XXX 4 bytes hole, try to pack */
>>>>
>>>>         u64                        exec_start;           /*    64     8 */
>>>>         u64                        sum_exec_runtime;     /*    72     8 */
>>>>         u64                        vruntime;             /*    80     8 */
>>>>         u64                        prev_sum_exec_runtime; /*    88     8 */
>>>>         u64                        nr_migrations;        /*    96     8 */
>>>>         int                        depth;                /*   104     4 */
>>>>
>>>>         /* XXX 4 bytes hole, try to pack */
>>>>
>>>>         struct sched_entity *      parent;               /*   112     8 */
>>>>         struct cfs_rq *            cfs_rq;               /*   120     8 */
>>>>         /* --- cacheline 1 boundary (128 bytes) --- */
>>>>         struct cfs_rq *            my_q;                 /*   128     8 */
>>>>         long unsigned int          runnable_weight;      /*   136     8 */
>>>>
>>>>         /* XXX 112 bytes hole, try to pack */
>>>>
>>>>         /* --- cacheline 2 boundary (256 bytes) --- */
>>>>         struct sched_avg           avg;                  /*   256   128 */
>>>>
>>>>
>>>> 6.2 + V12 patch
>>>> #pahole sched_entity
>>>> struct sched_entity {
>>>>         struct load_weight         load;                 /*     0    16 */
>>>>
>>>>         /* XXX last struct has 4 bytes of padding */
>>>>
>>>>         struct rb_node             run_node;             /*    16    24 */
>>>>         struct rb_node             latency_node;         /*    40    24 */
>>>>         struct list_head           group_node;           /*    64    16 */
>>>>         unsigned int               on_rq;                /*    80     4 */
>>>>
>>>>         /* XXX 4 bytes hole, try to pack */
>>>>
>>>>         u64                        exec_start;           /*    88     8 */
>>>>         u64                        sum_exec_runtime;     /*    96     8 */
>>>>         u64                        vruntime;             /*   104     8 */
>>>>         u64                        prev_sum_exec_runtime; /*   112     8 */
>>>>         u64                        nr_migrations;        /*   120     8 */
>>>>         /* --- cacheline 1 boundary (128 bytes) --- */
>>>>         int                        depth;                /*   128     4 */
>>>>
>>>>         /* XXX 4 bytes hole, try to pack */
>>>>
>>>>         struct sched_entity *      parent;               /*   136     8 */
>>>>         struct cfs_rq *            cfs_rq;               /*   144     8 */
>>>>         struct cfs_rq *            my_q;                 /*   152     8 */
>>>>
>>>>
>>>> 6.2 + V12 patch + Re-shuffle of latency_node
>>>> #pahole sched_entity
>>>> struct sched_entity {
>>>>         struct load_weight         load;                 /*     0    16 */
>>>>
>>>>         /* XXX last struct has 4 bytes of padding */
>>>>
>>>>         struct rb_node             run_node;             /*    16    24 */
>>>>         struct list_head           group_node;           /*    40    16 */
>>>>         unsigned int               on_rq;                /*    56     4 */
>>>>
>>>>         /* XXX 4 bytes hole, try to pack */
>>>>
>>>>         u64                        exec_start;           /*    64     8 */
>>>>         u64                        sum_exec_runtime;     /*    72     8 */
>>>>         u64                        vruntime;             /*    80     8 */
>>>>         u64                        prev_sum_exec_runtime; /*    88     8 */
>>>>         u64                        nr_migrations;        /*    96     8 */
>>>>         int                        depth;                /*   104     4 */
>>>>
>>>>         /* XXX 4 bytes hole, try to pack */
>>>>
>>>>         struct sched_entity *      parent;               /*   112     8 */
>>>>         struct cfs_rq *            cfs_rq;               /*   120     8 */
>>>>         /* --- cacheline 1 boundary (128 bytes) --- */
>>>>         struct cfs_rq *            my_q;                 /*   128     8 */
>>>>         long unsigned int          runnable_weight;      /*   136     8 */
>>>>         struct rb_node             latency_node;         /*   144    24 */
>>>>         long int                   latency_offset;       /*   168     8 */
>>>>
>>>>         /* XXX 80 bytes hole, try to pack */
>>>>
>>>>         /* --- cacheline 2 boundary (256 bytes) --- */
>>>>
>>>>
>>>>
>>>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>>>> index a2b52cf5e1bb..1e93aaaeead2 100644
>>>> --- a/include/linux/sched.h
>>>> +++ b/include/linux/sched.h
>>>> @@ -548,7 +548,6 @@ struct sched_entity {
>>>>         /* For load-balancing: */
>>>>         struct load_weight              load;
>>>>         struct rb_node                  run_node;
>>>> -       struct rb_node                  latency_node;
>>>>         struct list_head                group_node;
>>>>         unsigned int                    on_rq;
>>>>
>>>> @@ -569,6 +568,7 @@ struct sched_entity {
>>>>         /* cached value of my_q->h_nr_running */
>>>>         unsigned long                   runnable_weight;
>>>>  #endif
>>>> +       struct rb_node                  latency_node;
>>>>         /* preemption offset in ns */
>>>>         long                            latency_offset;
>>>>
>>>>
>>>> Ran the schbench and hackbench with this patch series. Here comparison is
>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
>>>> and other is running these benchmarks (group2). Set the latency nice
>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
>>>> Total of 96 CPU.
>>>>
>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
>>>> and attach the numbers to series.
>>>> Please suggest if any variation in the test i need to try.
>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
>>> run longer slice to move forward rather than preempting others all the
>>> time
>> hackbench still seems to regress in different latency nice values compared to
>> baseline of 6.2 in this case. up to 50% in some cases.
>>
>> 12 core powerpc system  with SMT=8 i.e 96 CPU
>> running 2 CPU cgroups. No quota assigned.
>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
>> latency is not changed for this cgroup.
>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
>> nice values of 0, -20 and 19.
> According to your other emails, you are using the cgroup interface and
> not the task's one. Do I get it right ?

right. I create cgroup, attach bash command with echo $$, 
assign the latency nice to cgroup, and run hackbench from that bash prompt.

>
> I haven't run test such tests in a cgroup but at least the test with
> latency_nice == 0 should not make any noticeable difference. Does this
> include the re-arrange patch that you have proposed previously ?

No. This is only with V12 of the series.

>
> Also, the tests that you did on v6, gave better result.
> https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@linux.vnet.ibm.com/
>
> Are you running same tests or you changed something in the mean time ?

Test machine got changed. 
now i re-read my earlier mail. I see it was slightly different. 
I had created only one cgroup and stress-ng was run
without any cgroup. Let me try that scenario and get the numbers. 

>
>> Numbers are average of 10 runs in each case. Time is in seconds
>>
>> type       groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
>>                   |            | lat nice=0    | lat nice=-20| lat nice=+19
>>                   |            |               |             |
>> Process       10  |   0.36     |     0.41      |    0.43     |    0.42
>> Process       20  |   0.62     |     0.76      |    0.75     |    0.75
>> Process       30  |   0.87     |     1.05      |    1.04     |    1.06
>> Process       40  |   1.13     |     1.34      |    1.33     |    1.33
>> Process       50  |   1.38     |     1.62      |    1.66     |    1.63
>> Process       60  |   1.64     |     1.91      |    1.97     |    1.90
>> thread        10  |   0.35     |     0.41      |    0.44     |    0.42
>> thread        20  |   0.64     |     0.78      |    0.77     |    0.79
>> Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
>> Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
>> Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
>> Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
>> Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
>> Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
>> thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
>> thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55
>>
>>
>>
>>>> Re-arrange seems to help the patch series by avoiding an cacheline miss.
>>>>
>>>> =========================
>>>> schbench
>>>> =========================
>>>>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
>>>> 1 Thread
>>>>   50.0th:        9.00  |    9.00        |        9.50
>>>>   75.0th:       10.50  |   10.00        |        9.50
>>>>   90.0th:       11.00  |   11.00        |       10.50
>>>>   95.0th:       11.00  |   11.00        |       11.00
>>>>   99.0th:       11.50  |   11.50        |       11.50
>>>>   99.5th:       12.50  |   12.00        |       12.00
>>>>   99.9th:       14.50  |   13.50        |       12.00
>>>> 2 Threads
>>>>   50.0th:        9.50  |    9.50        |        8.50
>>>>   75.0th:       11.00  |   10.50        |        9.50
>>>>   90.0th:       13.50  |   11.50        |       10.50
>>>>   95.0th:       14.00  |   12.00        |       11.00
>>>>   99.0th:       15.50  |   13.50        |       12.00
>>>>   99.5th:       16.00  |   14.00        |       12.00
>>>>   99.9th:       17.00  |   16.00        |       16.50
>>>> 4 Threads
>>>>   50.0th:       11.50  |   11.50        |       10.50
>>>>   75.0th:       13.50  |   12.50        |       12.50
>>>>   90.0th:       15.50  |   14.50        |       14.00
>>>>   95.0th:       16.50  |   15.50        |       14.50
>>>>   99.0th:       20.00  |   17.50        |       16.50
>>>>   99.5th:       20.50  |   18.50        |       17.00
>>>>   99.9th:       22.50  |   21.00        |       19.00
>>>> 8 Threads
>>>>   50.0th:       14.00  |   14.00        |       14.00
>>>>   75.0th:       16.00  |   16.00        |       16.00
>>>>   90.0th:       18.00  |   18.00        |       17.50
>>>>   95.0th:       18.50  |   18.50        |       18.50
>>>>   99.0th:       20.00  |   20.00        |       20.00
>>>>   99.5th:       20.50  |   21.50        |       21.00
>>>>   99.9th:       22.50  |   23.50        |       23.00
>>>> 16 Threads
>>>>   50.0th:       19.00  |   18.50        |       19.00
>>>>   75.0th:       23.00  |   22.50        |       23.00
>>>>   90.0th:       25.00  |   25.50        |       25.00
>>>>   95.0th:       26.50  |   26.50        |       26.00
>>>>   99.0th:       28.50  |   29.00        |       28.50
>>>>   99.5th:       31.00  |   30.00        |       30.00
>>>>   99.9th:     5626.00  | 4761.50        |       32.50
>>>> 32 Threads
>>>>   50.0th:       27.00  |   27.50        |       29.00
>>>>   75.0th:       35.50  |   36.50        |       38.50
>>>>   90.0th:       42.00  |   44.00        |       50.50
>>>>   95.0th:      447.50  | 2959.00        |     8544.00
>>>>   99.0th:     7372.00  | 17032.00       |    19136.00
>>>>   99.5th:    15360.00  | 19808.00       |    20704.00
>>>>   99.9th:    20640.00  | 30048.00       |    30048.00
>>>>
>>>> ====================
>>>> hackbench
>>>> ====================
>>>>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
>>>>
>>>> Process 10 Time:        0.35    |       0.42        |           0.41
>>>> Process 20 Time:        0.61    |       0.76        |           0.76
>>>> Process 30 Time:        0.87    |       1.06        |           1.05
>>>> thread 10 Time:         0.35    |       0.43        |           0.42
>>>> thread 20 Time:         0.66    |       0.79        |           0.78
>>>> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
>>>> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
>>>> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
>>>> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
>>>> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
>>>>
>>>>
>>>>>       struct list_head                group_node;
>>>>>       unsigned int                    on_rq;
>>>>>
>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>>> index 093cc1af73dc..752fd364216c 100644
>>>>> --- a/kernel/sched/core.c
>>>>> +++ b/kernel/sched/core.c
>>>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
>>>>>       p->se.nr_migrations             = 0;
>>>>>       p->se.vruntime                  = 0;
>>>>>       INIT_LIST_HEAD(&p->se.group_node);
>>>>> +     RB_CLEAR_NODE(&p->se.latency_node);
>>>>>
>>>>>  #ifdef CONFIG_FAIR_GROUP_SCHED
>>>>>       p->se.cfs_rq                    = NULL;
>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>>>> index 125a6ff53378..e2aeb4511686 100644
>>>>> --- a/kernel/sched/fair.c
>>>>> +++ b/kernel/sched/fair.c
>>>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
>>>>>
>>>>>       return __node_2_se(last);
>>>>>  }
>>>>> +#endif
>>>>>
>>>>> +/**************************************************************
>>>>> + * Scheduling class tree data structure manipulation methods:
>>>>> + * for latency
>>>>> + */
>>>>> +
>>>>> +static inline bool latency_before(struct sched_entity *a,
>>>>> +                             struct sched_entity *b)
>>>>> +{
>>>>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
>>>>> +}
>>>>> +
>>>>> +#define __latency_node_2_se(node) \
>>>>> +     rb_entry((node), struct sched_entity, latency_node)
>>>>> +
>>>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
>>>>> +{
>>>>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Enqueue an entity into the latency rb-tree:
>>>>> + */
>>>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>> +{
>>>>> +
>>>>> +     /* Only latency sensitive entity can be added to the list */
>>>>> +     if (se->latency_offset >= 0)
>>>>> +             return;
>>>>> +
>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node))
>>>>> +             return;
>>>>> +
>>>>> +     /*
>>>>> +      * The entity is always added the latency list at wakeup.
>>>>> +      * Then, a not waking up entity that is put back in the list after an
>>>>> +      * execution time less than sysctl_sched_min_granularity, means that
>>>>> +      * the entity has been preempted by a higher sched class or an entity
>>>>> +      * with higher latency constraint. In thi case, the entity is also put
>>>>> +      * back in the latency list so it gets a chance to run 1st during the
>>>>> +      * next slice.
>>>>> +      */
>>>>> +     if (!(flags & ENQUEUE_WAKEUP)) {
>>>>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
>>>>> +
>>>>> +             if (delta_exec >= sysctl_sched_min_granularity)
>>>>> +                     return;
>>>>> +     }
>>>>> +
>>>>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * Dequeue an entity from the latency rb-tree and return true if it was really
>>>>> + * part of the rb-tree:
>>>>> + */
>>>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>> +{
>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
>>>>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
>>>>> +             RB_CLEAR_NODE(&se->latency_node);
>>>>> +             return true;
>>>>> +     }
>>>>> +
>>>>> +     return false;
>>>>> +}
>>>>> +
>>>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
>>>>> +{
>>>>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
>>>>> +
>>>>> +     if (!left)
>>>>> +             return NULL;
>>>>> +
>>>>> +     return __latency_node_2_se(left);
>>>>> +}
>>>>> +
>>>>> +#ifdef CONFIG_SCHED_DEBUG
>>>>>  /**************************************************************
>>>>>   * Scheduling class statistics methods:
>>>>>   */
>>>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>       check_schedstat_required();
>>>>>       update_stats_enqueue_fair(cfs_rq, se, flags);
>>>>>       check_spread(cfs_rq, se);
>>>>> -     if (!curr)
>>>>> +     if (!curr) {
>>>>>               __enqueue_entity(cfs_rq, se);
>>>>> +             __enqueue_latency(cfs_rq, se, flags);
>>>>> +     }
>>>>>       se->on_rq = 1;
>>>>>
>>>>>       if (cfs_rq->nr_running == 1) {
>>>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>
>>>>>       clear_buddies(cfs_rq, se);
>>>>>
>>>>> -     if (se != cfs_rq->curr)
>>>>> +     if (se != cfs_rq->curr) {
>>>>>               __dequeue_entity(cfs_rq, se);
>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>> +     }
>>>>>       se->on_rq = 0;
>>>>>       account_entity_dequeue(cfs_rq, se);
>>>>>
>>>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>                */
>>>>>               update_stats_wait_end_fair(cfs_rq, se);
>>>>>               __dequeue_entity(cfs_rq, se);
>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>>               update_load_avg(cfs_rq, se, UPDATE_TG);
>>>>>       }
>>>>>
>>>>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
>>>>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>  {
>>>>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
>>>>> -     struct sched_entity *se;
>>>>> +     struct sched_entity *latency, *se;
>>>>>
>>>>>       /*
>>>>>        * If curr is set we have to see if its left of the leftmost entity
>>>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>               se = cfs_rq->last;
>>>>>       }
>>>>>
>>>>> +     /* Check for latency sensitive entity waiting for running */
>>>>> +     latency = __pick_first_latency(cfs_rq);
>>>>> +     if (latency && (latency != se) &&
>>>>> +         wakeup_preempt_entity(latency, se) < 1)
>>>>> +             se = latency;
>>>>> +
>>>>>       return se;
>>>>>  }
>>>>>
>>>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
>>>>>               update_stats_wait_start_fair(cfs_rq, prev);
>>>>>               /* Put 'current' back into the tree. */
>>>>>               __enqueue_entity(cfs_rq, prev);
>>>>> +             __enqueue_latency(cfs_rq, prev, 0);
>>>>>               /* in !on_rq case, update occurred at dequeue */
>>>>>               update_load_avg(cfs_rq, prev, 0);
>>>>>       }
>>>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>>>>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
>>>>>  {
>>>>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
>>>>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
>>>>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
>>>>>  #ifdef CONFIG_SMP
>>>>>       raw_spin_lock_init(&cfs_rq->removed.lock);
>>>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
>>>>>       se->my_q = cfs_rq;
>>>>>
>>>>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
>>>>> +     RB_CLEAR_NODE(&se->latency_node);
>>>>>
>>>>>       /* guarantee group entities always have weight */
>>>>>       update_load_set(&se->load, NICE_0_LOAD);
>>>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
>>>>>
>>>>>       for_each_possible_cpu(i) {
>>>>>               struct sched_entity *se = tg->se[i];
>>>>> +             struct rq *rq = cpu_rq(i);
>>>>> +             struct rq_flags rf;
>>>>> +             bool queued;
>>>>> +
>>>>> +             rq_lock_irqsave(rq, &rf);
>>>>>
>>>>> +             queued = __dequeue_latency(se->cfs_rq, se);
>>>>>               WRITE_ONCE(se->latency_offset, latency_offset);
>>>>> +             if (queued)
>>>>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
>>>>> +
>>>>> +
>>>>> +             rq_unlock_irqrestore(rq, &rf);
>>>>>       }
>>>>>
>>>>>       mutex_unlock(&shares_mutex);
>>>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
>>>>> index 9a2e71231083..21dd309e98a9 100644
>>>>> --- a/kernel/sched/sched.h
>>>>> +++ b/kernel/sched/sched.h
>>>>> @@ -570,6 +570,7 @@ struct cfs_rq {
>>>>>  #endif
>>>>>
>>>>>       struct rb_root_cached   tasks_timeline;
>>>>> +     struct rb_root_cached   latency_timeline;
>>>>>
>>>>>       /*
>>>>>        * 'curr' points to currently running entity on this cfs_rq.

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Shrikanth Hegde]

On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
>
> On 3/2/23 6:47 PM, Vincent Guittot wrote:
>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
>>>>>> Add a rb tree for latency sensitive entities so we can schedule the most
>>>>>> sensitive one first even when it failed to preempt current at wakeup or
>>>>>> when it got quickly preempted by another entity of higher priority.
>>>>>>
>>>>>> In order to keep fairness, the latency is used once at wakeup to get a
>>>>>> minimum slice and not during the following scheduling slice to prevent
>>>>>> long running entity to got more running time than allocated to his nice
>>>>>> priority.
>>>>>>
>>>>>> The rb tree enables to cover the last corner case where latency
>>>>>> sensitive entity can't got schedule quickly after the wakeup.
>>>>>>
>>>>>> Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
>>>>>> Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>
>>>>>> ---
>>>>>>  include/linux/sched.h |   1 +
>>>>>>  kernel/sched/core.c   |   1 +
>>>>>>  kernel/sched/fair.c   | 109 ++++++++++++++++++++++++++++++++++++++++--
>>>>>>  kernel/sched/sched.h  |   1 +
>>>>>>  4 files changed, 109 insertions(+), 3 deletions(-)
>>>>>>
>>>>>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>>>>>> index 38decae3e156..41bb92be5ecc 100644
>>>>>> --- a/include/linux/sched.h
>>>>>> +++ b/include/linux/sched.h
>>>>>> @@ -548,6 +548,7 @@ struct sched_entity {
>>>>>>       /* For load-balancing: */
>>>>>>       struct load_weight              load;
>>>>>>       struct rb_node                  run_node;
>>>>>> +     struct rb_node                  latency_node;
>>>>> Ran pahole to see if the frequently accessed fields change across cachelines.
>>>>> There is no change in cachelines of task_struct, whereas sched_entity differs
>>>>> due to latency_node.  Maybe the latency_node could be placed after
>>>>> runnable_weight as there is space available in that cacheline.
>>>> I will run some test on my system to confimr your results but we can
>>>> move latency_node field if it helps cache hit stats
>>>>
>>>>> 6.2
>>>>> #pahole sched_entity
>>>>> struct sched_entity {
>>>>>         struct load_weight         load;                 /*     0    16 */
>>>>>
>>>>>         /* XXX last struct has 4 bytes of padding */
>>>>>
>>>>>         struct rb_node             run_node;             /*    16    24 */
>>>>>         struct list_head           group_node;           /*    40    16 */
>>>>>         unsigned int               on_rq;                /*    56     4 */
>>>>>
>>>>>         /* XXX 4 bytes hole, try to pack */
>>>>>
>>>>>         u64                        exec_start;           /*    64     8 */
>>>>>         u64                        sum_exec_runtime;     /*    72     8 */
>>>>>         u64                        vruntime;             /*    80     8 */
>>>>>         u64                        prev_sum_exec_runtime; /*    88     8 */
>>>>>         u64                        nr_migrations;        /*    96     8 */
>>>>>         int                        depth;                /*   104     4 */
>>>>>
>>>>>         /* XXX 4 bytes hole, try to pack */
>>>>>
>>>>>         struct sched_entity *      parent;               /*   112     8 */
>>>>>         struct cfs_rq *            cfs_rq;               /*   120     8 */
>>>>>         /* --- cacheline 1 boundary (128 bytes) --- */
>>>>>         struct cfs_rq *            my_q;                 /*   128     8 */
>>>>>         long unsigned int          runnable_weight;      /*   136     8 */
>>>>>
>>>>>         /* XXX 112 bytes hole, try to pack */
>>>>>
>>>>>         /* --- cacheline 2 boundary (256 bytes) --- */
>>>>>         struct sched_avg           avg;                  /*   256   128 */
>>>>>
>>>>>
>>>>> 6.2 + V12 patch
>>>>> #pahole sched_entity
>>>>> struct sched_entity {
>>>>>         struct load_weight         load;                 /*     0    16 */
>>>>>
>>>>>         /* XXX last struct has 4 bytes of padding */
>>>>>
>>>>>         struct rb_node             run_node;             /*    16    24 */
>>>>>         struct rb_node             latency_node;         /*    40    24 */
>>>>>         struct list_head           group_node;           /*    64    16 */
>>>>>         unsigned int               on_rq;                /*    80     4 */
>>>>>
>>>>>         /* XXX 4 bytes hole, try to pack */
>>>>>
>>>>>         u64                        exec_start;           /*    88     8 */
>>>>>         u64                        sum_exec_runtime;     /*    96     8 */
>>>>>         u64                        vruntime;             /*   104     8 */
>>>>>         u64                        prev_sum_exec_runtime; /*   112     8 */
>>>>>         u64                        nr_migrations;        /*   120     8 */
>>>>>         /* --- cacheline 1 boundary (128 bytes) --- */
>>>>>         int                        depth;                /*   128     4 */
>>>>>
>>>>>         /* XXX 4 bytes hole, try to pack */
>>>>>
>>>>>         struct sched_entity *      parent;               /*   136     8 */
>>>>>         struct cfs_rq *            cfs_rq;               /*   144     8 */
>>>>>         struct cfs_rq *            my_q;                 /*   152     8 */
>>>>>
>>>>>
>>>>> 6.2 + V12 patch + Re-shuffle of latency_node
>>>>> #pahole sched_entity
>>>>> struct sched_entity {
>>>>>         struct load_weight         load;                 /*     0    16 */
>>>>>
>>>>>         /* XXX last struct has 4 bytes of padding */
>>>>>
>>>>>         struct rb_node             run_node;             /*    16    24 */
>>>>>         struct list_head           group_node;           /*    40    16 */
>>>>>         unsigned int               on_rq;                /*    56     4 */
>>>>>
>>>>>         /* XXX 4 bytes hole, try to pack */
>>>>>
>>>>>         u64                        exec_start;           /*    64     8 */
>>>>>         u64                        sum_exec_runtime;     /*    72     8 */
>>>>>         u64                        vruntime;             /*    80     8 */
>>>>>         u64                        prev_sum_exec_runtime; /*    88     8 */
>>>>>         u64                        nr_migrations;        /*    96     8 */
>>>>>         int                        depth;                /*   104     4 */
>>>>>
>>>>>         /* XXX 4 bytes hole, try to pack */
>>>>>
>>>>>         struct sched_entity *      parent;               /*   112     8 */
>>>>>         struct cfs_rq *            cfs_rq;               /*   120     8 */
>>>>>         /* --- cacheline 1 boundary (128 bytes) --- */
>>>>>         struct cfs_rq *            my_q;                 /*   128     8 */
>>>>>         long unsigned int          runnable_weight;      /*   136     8 */
>>>>>         struct rb_node             latency_node;         /*   144    24 */
>>>>>         long int                   latency_offset;       /*   168     8 */
>>>>>
>>>>>         /* XXX 80 bytes hole, try to pack */
>>>>>
>>>>>         /* --- cacheline 2 boundary (256 bytes) --- */
>>>>>
>>>>>
>>>>>
>>>>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>>>>> index a2b52cf5e1bb..1e93aaaeead2 100644
>>>>> --- a/include/linux/sched.h
>>>>> +++ b/include/linux/sched.h
>>>>> @@ -548,7 +548,6 @@ struct sched_entity {
>>>>>         /* For load-balancing: */
>>>>>         struct load_weight              load;
>>>>>         struct rb_node                  run_node;
>>>>> -       struct rb_node                  latency_node;
>>>>>         struct list_head                group_node;
>>>>>         unsigned int                    on_rq;
>>>>>
>>>>> @@ -569,6 +568,7 @@ struct sched_entity {
>>>>>         /* cached value of my_q->h_nr_running */
>>>>>         unsigned long                   runnable_weight;
>>>>>  #endif
>>>>> +       struct rb_node                  latency_node;
>>>>>         /* preemption offset in ns */
>>>>>         long                            latency_offset;
>>>>>
>>>>>
>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
>>>>> and other is running these benchmarks (group2). Set the latency nice
>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
>>>>> Total of 96 CPU.
>>>>>
>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
>>>>> and attach the numbers to series.
>>>>> Please suggest if any variation in the test i need to try.
>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
>>>> run longer slice to move forward rather than preempting others all the
>>>> time
>>> hackbench still seems to regress in different latency nice values compared to
>>> baseline of 6.2 in this case. up to 50% in some cases.
>>>
>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
>>> running 2 CPU cgroups. No quota assigned.
>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
>>> latency is not changed for this cgroup.
>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
>>> nice values of 0, -20 and 19.
>> According to your other emails, you are using the cgroup interface and
>> not the task's one. Do I get it right ?
> right. I create cgroup, attach bash command with echo $$, 
> assign the latency nice to cgroup, and run hackbench from that bash prompt.
>
>> I haven't run test such tests in a cgroup but at least the test with
>> latency_nice == 0 should not make any noticeable difference. Does this
>> include the re-arrange patch that you have proposed previously ?
> No. This is only with V12 of the series.
>
>> Also, the tests that you did on v6, gave better result.
>> https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@linux.vnet.ibm.com/
>>
>> Are you running same tests or you changed something in the mean time ?
> Test machine got changed. 
> now i re-read my earlier mail. I see it was slightly different. 
> I had created only one cgroup and stress-ng was run
> without any cgroup. Let me try that scenario and get the numbers. 


Tried the same method of testing i had done on V7 of the series. on this
machine hackbench still regress's both on V12 as well as V7 of the series.

Created one cpu cgroup called cgroup1. created two bash prompts. 
assigned "bash $$" to cgroup1 and on other bash prompt running,
stress-ng --cpu=96 -l 50. Ran hackbench from cgroup1 prompt. 
assigned latency values to the cgroup1.

I will try to run with only task's set with latency_nice=0 as well. 

type	   groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
		  |	        |lat nice=0| lat nice=-20| lat nice=+19

Process	      10  |	0.33    |   0.37   |   0.38     |   0.37
Process       20  |	0.61    |   0.67   |   0.68     |   0.67
Process	      30  |	0.85    |   0.95   |   0.95     |   0.96
Process	      40  |	1.10    |   1.20   |   1.20     |   1.21
Process	      50  |	1.34    |   1.47   |   1.44     |   1.45
Process	      60  |	1.57    |   1.70   |   1.71     |   1.70
thread	      10  |	0.36    |   0.40   |   0.39     |   0.39
thread	      20  |	0.65    |   0.72   |   0.71     |   0.71
Process(Pipe) 10  |	0.18	|   0.31   |   0.31	|   0.33
Process(Pipe) 20  |	0.32	|   0.51   |   0.50	|   0.50
Process(Pipe) 30  |	0.43	|   0.65   |   0.67	|   0.67
Process(Pipe) 40  |	0.57	|   0.82   |   0.83	|   0.83
Process(Pipe) 50  |	0.67	|   1.00   |   0.97	|   0.98
Process(Pipe) 60  |	0.81	|   1.13   |   1.11	|   1.12
thread(Pipe)  10  |	0.19	|   0.33   |   0.33	|   0.33
thread(Pipe)  20  |	0.34	|   0.53   |   0.51	|   0.52



type	   groups |   v6.2	|v6.2 + V7 | v6.2 + V7  | v6.2 + V7
		  |		|lat nice=0|lat nice=-20| lat nice=+19
Process	      10  |	0.33    |   0.37   |   0.37     |   0.37
Process	      20  |	0.61    |   0.67   |   0.67     |   0.67
Process	      30  |	0.85    |   0.96   |   0.94     |   0.95
Process	      40  |	1.10    |   1.20   |   1.20     |   1.20
Process	      50  |	1.34    |   1.45   |   1.46     |   1.45
Process	      60  |	1.57    |   1.71   |   1.68     |   1.72
thread	      10  |	0.36    |   0.40   |   0.40     |   0.40
thread	      20  |	0.65    |   0.71   |   0.71     |   0.71
Process(Pipe) 10  |	0.18	|   0.30   |   0.30	|   0.31
Process(Pipe) 20  |	0.32	|   0.50   |   0.50	|   0.50
Process(Pipe) 30  |	0.43	|   0.67   |   0.67	|   0.66
Process(Pipe) 40  |	0.57	|   0.86   |   0.84	|   0.84
Process(Pipe) 50  |	0.67	|   0.99   |   0.97	|   0.97
Process(Pipe) 60  |	0.81	|   1.10   |   1.13	|   1.13
thread(Pipe)  10  |	0.19	|   0.34   |   0.34	|   0.33
thread(Pipe)  20  |	0.34	|   0.55   |   0.53	|   0.54

>
>>> Numbers are average of 10 runs in each case. Time is in seconds
>>>
>>> type       groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
>>>                   |            | lat nice=0    | lat nice=-20| lat nice=+19
>>>                   |            |               |             |
>>> Process       10  |   0.36     |     0.41      |    0.43     |    0.42
>>> Process       20  |   0.62     |     0.76      |    0.75     |    0.75
>>> Process       30  |   0.87     |     1.05      |    1.04     |    1.06
>>> Process       40  |   1.13     |     1.34      |    1.33     |    1.33
>>> Process       50  |   1.38     |     1.62      |    1.66     |    1.63
>>> Process       60  |   1.64     |     1.91      |    1.97     |    1.90
>>> thread        10  |   0.35     |     0.41      |    0.44     |    0.42
>>> thread        20  |   0.64     |     0.78      |    0.77     |    0.79
>>> Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
>>> Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
>>> Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
>>> Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
>>> Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
>>> Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
>>> thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
>>> thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55
>>>
>>>
>>>
>>>>> Re-arrange seems to help the patch series by avoiding an cacheline miss.
>>>>>
>>>>> =========================
>>>>> schbench
>>>>> =========================
>>>>>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
>>>>> 1 Thread
>>>>>   50.0th:        9.00  |    9.00        |        9.50
>>>>>   75.0th:       10.50  |   10.00        |        9.50
>>>>>   90.0th:       11.00  |   11.00        |       10.50
>>>>>   95.0th:       11.00  |   11.00        |       11.00
>>>>>   99.0th:       11.50  |   11.50        |       11.50
>>>>>   99.5th:       12.50  |   12.00        |       12.00
>>>>>   99.9th:       14.50  |   13.50        |       12.00
>>>>> 2 Threads
>>>>>   50.0th:        9.50  |    9.50        |        8.50
>>>>>   75.0th:       11.00  |   10.50        |        9.50
>>>>>   90.0th:       13.50  |   11.50        |       10.50
>>>>>   95.0th:       14.00  |   12.00        |       11.00
>>>>>   99.0th:       15.50  |   13.50        |       12.00
>>>>>   99.5th:       16.00  |   14.00        |       12.00
>>>>>   99.9th:       17.00  |   16.00        |       16.50
>>>>> 4 Threads
>>>>>   50.0th:       11.50  |   11.50        |       10.50
>>>>>   75.0th:       13.50  |   12.50        |       12.50
>>>>>   90.0th:       15.50  |   14.50        |       14.00
>>>>>   95.0th:       16.50  |   15.50        |       14.50
>>>>>   99.0th:       20.00  |   17.50        |       16.50
>>>>>   99.5th:       20.50  |   18.50        |       17.00
>>>>>   99.9th:       22.50  |   21.00        |       19.00
>>>>> 8 Threads
>>>>>   50.0th:       14.00  |   14.00        |       14.00
>>>>>   75.0th:       16.00  |   16.00        |       16.00
>>>>>   90.0th:       18.00  |   18.00        |       17.50
>>>>>   95.0th:       18.50  |   18.50        |       18.50
>>>>>   99.0th:       20.00  |   20.00        |       20.00
>>>>>   99.5th:       20.50  |   21.50        |       21.00
>>>>>   99.9th:       22.50  |   23.50        |       23.00
>>>>> 16 Threads
>>>>>   50.0th:       19.00  |   18.50        |       19.00
>>>>>   75.0th:       23.00  |   22.50        |       23.00
>>>>>   90.0th:       25.00  |   25.50        |       25.00
>>>>>   95.0th:       26.50  |   26.50        |       26.00
>>>>>   99.0th:       28.50  |   29.00        |       28.50
>>>>>   99.5th:       31.00  |   30.00        |       30.00
>>>>>   99.9th:     5626.00  | 4761.50        |       32.50
>>>>> 32 Threads
>>>>>   50.0th:       27.00  |   27.50        |       29.00
>>>>>   75.0th:       35.50  |   36.50        |       38.50
>>>>>   90.0th:       42.00  |   44.00        |       50.50
>>>>>   95.0th:      447.50  | 2959.00        |     8544.00
>>>>>   99.0th:     7372.00  | 17032.00       |    19136.00
>>>>>   99.5th:    15360.00  | 19808.00       |    20704.00
>>>>>   99.9th:    20640.00  | 30048.00       |    30048.00
>>>>>
>>>>> ====================
>>>>> hackbench
>>>>> ====================
>>>>>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
>>>>>
>>>>> Process 10 Time:        0.35    |       0.42        |           0.41
>>>>> Process 20 Time:        0.61    |       0.76        |           0.76
>>>>> Process 30 Time:        0.87    |       1.06        |           1.05
>>>>> thread 10 Time:         0.35    |       0.43        |           0.42
>>>>> thread 20 Time:         0.66    |       0.79        |           0.78
>>>>> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
>>>>> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
>>>>> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
>>>>> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
>>>>> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
>>>>>
>>>>>
>>>>>>       struct list_head                group_node;
>>>>>>       unsigned int                    on_rq;
>>>>>>
>>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>>>> index 093cc1af73dc..752fd364216c 100644
>>>>>> --- a/kernel/sched/core.c
>>>>>> +++ b/kernel/sched/core.c
>>>>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
>>>>>>       p->se.nr_migrations             = 0;
>>>>>>       p->se.vruntime                  = 0;
>>>>>>       INIT_LIST_HEAD(&p->se.group_node);
>>>>>> +     RB_CLEAR_NODE(&p->se.latency_node);
>>>>>>
>>>>>>  #ifdef CONFIG_FAIR_GROUP_SCHED
>>>>>>       p->se.cfs_rq                    = NULL;
>>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>>>>> index 125a6ff53378..e2aeb4511686 100644
>>>>>> --- a/kernel/sched/fair.c
>>>>>> +++ b/kernel/sched/fair.c
>>>>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
>>>>>>
>>>>>>       return __node_2_se(last);
>>>>>>  }
>>>>>> +#endif
>>>>>>
>>>>>> +/**************************************************************
>>>>>> + * Scheduling class tree data structure manipulation methods:
>>>>>> + * for latency
>>>>>> + */
>>>>>> +
>>>>>> +static inline bool latency_before(struct sched_entity *a,
>>>>>> +                             struct sched_entity *b)
>>>>>> +{
>>>>>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
>>>>>> +}
>>>>>> +
>>>>>> +#define __latency_node_2_se(node) \
>>>>>> +     rb_entry((node), struct sched_entity, latency_node)
>>>>>> +
>>>>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
>>>>>> +{
>>>>>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
>>>>>> +}
>>>>>> +
>>>>>> +/*
>>>>>> + * Enqueue an entity into the latency rb-tree:
>>>>>> + */
>>>>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>> +{
>>>>>> +
>>>>>> +     /* Only latency sensitive entity can be added to the list */
>>>>>> +     if (se->latency_offset >= 0)
>>>>>> +             return;
>>>>>> +
>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node))
>>>>>> +             return;
>>>>>> +
>>>>>> +     /*
>>>>>> +      * The entity is always added the latency list at wakeup.
>>>>>> +      * Then, a not waking up entity that is put back in the list after an
>>>>>> +      * execution time less than sysctl_sched_min_granularity, means that
>>>>>> +      * the entity has been preempted by a higher sched class or an entity
>>>>>> +      * with higher latency constraint. In thi case, the entity is also put
>>>>>> +      * back in the latency list so it gets a chance to run 1st during the
>>>>>> +      * next slice.
>>>>>> +      */
>>>>>> +     if (!(flags & ENQUEUE_WAKEUP)) {
>>>>>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
>>>>>> +
>>>>>> +             if (delta_exec >= sysctl_sched_min_granularity)
>>>>>> +                     return;
>>>>>> +     }
>>>>>> +
>>>>>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
>>>>>> +}
>>>>>> +
>>>>>> +/*
>>>>>> + * Dequeue an entity from the latency rb-tree and return true if it was really
>>>>>> + * part of the rb-tree:
>>>>>> + */
>>>>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>> +{
>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
>>>>>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
>>>>>> +             RB_CLEAR_NODE(&se->latency_node);
>>>>>> +             return true;
>>>>>> +     }
>>>>>> +
>>>>>> +     return false;
>>>>>> +}
>>>>>> +
>>>>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
>>>>>> +{
>>>>>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
>>>>>> +
>>>>>> +     if (!left)
>>>>>> +             return NULL;
>>>>>> +
>>>>>> +     return __latency_node_2_se(left);
>>>>>> +}
>>>>>> +
>>>>>> +#ifdef CONFIG_SCHED_DEBUG
>>>>>>  /**************************************************************
>>>>>>   * Scheduling class statistics methods:
>>>>>>   */
>>>>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>       check_schedstat_required();
>>>>>>       update_stats_enqueue_fair(cfs_rq, se, flags);
>>>>>>       check_spread(cfs_rq, se);
>>>>>> -     if (!curr)
>>>>>> +     if (!curr) {
>>>>>>               __enqueue_entity(cfs_rq, se);
>>>>>> +             __enqueue_latency(cfs_rq, se, flags);
>>>>>> +     }
>>>>>>       se->on_rq = 1;
>>>>>>
>>>>>>       if (cfs_rq->nr_running == 1) {
>>>>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>
>>>>>>       clear_buddies(cfs_rq, se);
>>>>>>
>>>>>> -     if (se != cfs_rq->curr)
>>>>>> +     if (se != cfs_rq->curr) {
>>>>>>               __dequeue_entity(cfs_rq, se);
>>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>>> +     }
>>>>>>       se->on_rq = 0;
>>>>>>       account_entity_dequeue(cfs_rq, se);
>>>>>>
>>>>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>>                */
>>>>>>               update_stats_wait_end_fair(cfs_rq, se);
>>>>>>               __dequeue_entity(cfs_rq, se);
>>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>>>               update_load_avg(cfs_rq, se, UPDATE_TG);
>>>>>>       }
>>>>>>
>>>>>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
>>>>>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>>  {
>>>>>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
>>>>>> -     struct sched_entity *se;
>>>>>> +     struct sched_entity *latency, *se;
>>>>>>
>>>>>>       /*
>>>>>>        * If curr is set we have to see if its left of the leftmost entity
>>>>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>>               se = cfs_rq->last;
>>>>>>       }
>>>>>>
>>>>>> +     /* Check for latency sensitive entity waiting for running */
>>>>>> +     latency = __pick_first_latency(cfs_rq);
>>>>>> +     if (latency && (latency != se) &&
>>>>>> +         wakeup_preempt_entity(latency, se) < 1)
>>>>>> +             se = latency;
>>>>>> +
>>>>>>       return se;
>>>>>>  }
>>>>>>
>>>>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
>>>>>>               update_stats_wait_start_fair(cfs_rq, prev);
>>>>>>               /* Put 'current' back into the tree. */
>>>>>>               __enqueue_entity(cfs_rq, prev);
>>>>>> +             __enqueue_latency(cfs_rq, prev, 0);
>>>>>>               /* in !on_rq case, update occurred at dequeue */
>>>>>>               update_load_avg(cfs_rq, prev, 0);
>>>>>>       }
>>>>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>>>>>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
>>>>>>  {
>>>>>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
>>>>>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
>>>>>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
>>>>>>  #ifdef CONFIG_SMP
>>>>>>       raw_spin_lock_init(&cfs_rq->removed.lock);
>>>>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
>>>>>>       se->my_q = cfs_rq;
>>>>>>
>>>>>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
>>>>>> +     RB_CLEAR_NODE(&se->latency_node);
>>>>>>
>>>>>>       /* guarantee group entities always have weight */
>>>>>>       update_load_set(&se->load, NICE_0_LOAD);
>>>>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
>>>>>>
>>>>>>       for_each_possible_cpu(i) {
>>>>>>               struct sched_entity *se = tg->se[i];
>>>>>> +             struct rq *rq = cpu_rq(i);
>>>>>> +             struct rq_flags rf;
>>>>>> +             bool queued;
>>>>>> +
>>>>>> +             rq_lock_irqsave(rq, &rf);
>>>>>>
>>>>>> +             queued = __dequeue_latency(se->cfs_rq, se);
>>>>>>               WRITE_ONCE(se->latency_offset, latency_offset);
>>>>>> +             if (queued)
>>>>>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
>>>>>> +
>>>>>> +
>>>>>> +             rq_unlock_irqrestore(rq, &rf);
>>>>>>       }
>>>>>>
>>>>>>       mutex_unlock(&shares_mutex);
>>>>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
>>>>>> index 9a2e71231083..21dd309e98a9 100644
>>>>>> --- a/kernel/sched/sched.h
>>>>>> +++ b/kernel/sched/sched.h
>>>>>> @@ -570,6 +570,7 @@ struct cfs_rq {
>>>>>>  #endif
>>>>>>
>>>>>>       struct rb_root_cached   tasks_timeline;
>>>>>> +     struct rb_root_cached   latency_timeline;
>>>>>>
>>>>>>       /*
>>>>>>        * 'curr' points to currently running entity on this cfs_rq.

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Vincent Guittot]

Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
> 
> 
> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
> >
> > On 3/2/23 6:47 PM, Vincent Guittot wrote:
> >> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
> >>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:

[...]

> >>>>> Ran the schbench and hackbench with this patch series. Here comparison is
> >>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
> >>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
> >>>>> and other is running these benchmarks (group2). Set the latency nice
> >>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
> >>>>> Total of 96 CPU.
> >>>>>
> >>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
> >>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
> >>>>> and attach the numbers to series.
> >>>>> Please suggest if any variation in the test i need to try.
> >>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
> >>>> run longer slice to move forward rather than preempting others all the
> >>>> time
> >>> hackbench still seems to regress in different latency nice values compared to
> >>> baseline of 6.2 in this case. up to 50% in some cases.
> >>>
> >>> 12 core powerpc system  with SMT=8 i.e 96 CPU
> >>> running 2 CPU cgroups. No quota assigned.
> >>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
> >>> latency is not changed for this cgroup.
> >>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
> >>> nice values of 0, -20 and 19.
> >> According to your other emails, you are using the cgroup interface and
> >> not the task's one. Do I get it right ?
> > right. I create cgroup, attach bash command with echo $$, 
> > assign the latency nice to cgroup, and run hackbench from that bash prompt.
> >
> >> I haven't run test such tests in a cgroup but at least the test with
> >> latency_nice == 0 should not make any noticeable difference. Does this
> >> include the re-arrange patch that you have proposed previously ?
> > No. This is only with V12 of the series.
> >
> >> Also, the tests that you did on v6, gave better result.
> >> https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@linux.vnet.ibm.com/
> >>
> >> Are you running same tests or you changed something in the mean time ?
> > Test machine got changed. 
> > now i re-read my earlier mail. I see it was slightly different. 
> > I had created only one cgroup and stress-ng was run
> > without any cgroup. Let me try that scenario and get the numbers. 
> 
> 
> Tried the same method of testing i had done on V7 of the series. on this
> machine hackbench still regress's both on V12 as well as V7 of the series.
> 
> Created one cpu cgroup called cgroup1. created two bash prompts. 
> assigned "bash $$" to cgroup1 and on other bash prompt running,
> stress-ng --cpu=96 -l 50. Ran hackbench from cgroup1 prompt. 
> assigned latency values to the cgroup1.

I have tried to reproduce your results on some of my systems but I can't see
the impacts that you are reporting below.
The fact that your other platform was not impacted as well could imply that
it's specific to this platform.
In particular, the lat nice=0 case should not show any real impact as it
should be similar to a nop. At least that what I can see in the tests on my
platforms and Prateek on his.

Nevertheless, could you try to run your tests with the changes below ?
These are the only places which could have an impact even with lat nice = 0

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8137bca80572..979571a98b28 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4991,8 +4991,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
        if (delta < offset)
                return;

-       if ((delta > ideal_runtime) ||
-           (delta > get_latency_max()))
+       if (delta > ideal_runtime)
                resched_curr(rq_of(cfs_rq));
 }

@@ -7574,9 +7573,10 @@ static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *
         * Otherwise, use the latency weight to evaluate how much scheduling
         * delay is acceptable by se.
         */
-       if ((latency_offset < 0) || (curr->latency_offset < 0))
+       if ((latency_offset < 0) || (curr->latency_offset < 0)) {
                latency_offset -= curr->latency_offset;
-       latency_offset = min_t(long, latency_offset, get_latency_max());
+               latency_offset = min_t(long, latency_offset, get_latency_max());
+       }

        return latency_offset;
 }
@@ -7635,7 +7635,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
         * for low priority task. Make sure that long sleeping task will get a
         * chance to preempt current.
         */
-       gran = min_t(s64, gran, get_latency_max());

        if (vdiff > gran)
                return 1;


> 
> I will try to run with only task's set with latency_nice=0 as well. 
> 
> type	   groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
> 		  |	        |lat nice=0| lat nice=-20| lat nice=+19
> 
> Process	      10  |	0.33    |   0.37   |   0.38     |   0.37
> Process       20  |	0.61    |   0.67   |   0.68     |   0.67
> Process	      30  |	0.85    |   0.95   |   0.95     |   0.96
> Process	      40  |	1.10    |   1.20   |   1.20     |   1.21
> Process	      50  |	1.34    |   1.47   |   1.44     |   1.45
> Process	      60  |	1.57    |   1.70   |   1.71     |   1.70
> thread	      10  |	0.36    |   0.40   |   0.39     |   0.39
> thread	      20  |	0.65    |   0.72   |   0.71     |   0.71
> Process(Pipe) 10  |	0.18	|   0.31   |   0.31	|   0.33
> Process(Pipe) 20  |	0.32	|   0.51   |   0.50	|   0.50
> Process(Pipe) 30  |	0.43	|   0.65   |   0.67	|   0.67
> Process(Pipe) 40  |	0.57	|   0.82   |   0.83	|   0.83
> Process(Pipe) 50  |	0.67	|   1.00   |   0.97	|   0.98
> Process(Pipe) 60  |	0.81	|   1.13   |   1.11	|   1.12
> thread(Pipe)  10  |	0.19	|   0.33   |   0.33	|   0.33
> thread(Pipe)  20  |	0.34	|   0.53   |   0.51	|   0.52
> 
> 
> 
> type	   groups |   v6.2	|v6.2 + V7 | v6.2 + V7  | v6.2 + V7
> 		  |		|lat nice=0|lat nice=-20| lat nice=+19
> Process	      10  |	0.33    |   0.37   |   0.37     |   0.37
> Process	      20  |	0.61    |   0.67   |   0.67     |   0.67
> Process	      30  |	0.85    |   0.96   |   0.94     |   0.95
> Process	      40  |	1.10    |   1.20   |   1.20     |   1.20
> Process	      50  |	1.34    |   1.45   |   1.46     |   1.45
> Process	      60  |	1.57    |   1.71   |   1.68     |   1.72
> thread	      10  |	0.36    |   0.40   |   0.40     |   0.40
> thread	      20  |	0.65    |   0.71   |   0.71     |   0.71
> Process(Pipe) 10  |	0.18	|   0.30   |   0.30	|   0.31
> Process(Pipe) 20  |	0.32	|   0.50   |   0.50	|   0.50
> Process(Pipe) 30  |	0.43	|   0.67   |   0.67	|   0.66
> Process(Pipe) 40  |	0.57	|   0.86   |   0.84	|   0.84
> Process(Pipe) 50  |	0.67	|   0.99   |   0.97	|   0.97
> Process(Pipe) 60  |	0.81	|   1.10   |   1.13	|   1.13
> thread(Pipe)  10  |	0.19	|   0.34   |   0.34	|   0.33
> thread(Pipe)  20  |	0.34	|   0.55   |   0.53	|   0.54
> 
> >
> >>> Numbers are average of 10 runs in each case. Time is in seconds
> >>>
> >>> type       groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
> >>>                   |            | lat nice=0    | lat nice=-20| lat nice=+19
> >>>                   |            |               |             |
> >>> Process       10  |   0.36     |     0.41      |    0.43     |    0.42
> >>> Process       20  |   0.62     |     0.76      |    0.75     |    0.75
> >>> Process       30  |   0.87     |     1.05      |    1.04     |    1.06
> >>> Process       40  |   1.13     |     1.34      |    1.33     |    1.33
> >>> Process       50  |   1.38     |     1.62      |    1.66     |    1.63
> >>> Process       60  |   1.64     |     1.91      |    1.97     |    1.90
> >>> thread        10  |   0.35     |     0.41      |    0.44     |    0.42
> >>> thread        20  |   0.64     |     0.78      |    0.77     |    0.79
> >>> Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
> >>> Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
> >>> Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
> >>> Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
> >>> Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
> >>> Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
> >>> thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
> >>> thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55
> >>>
> >>>
> >>>
> >>>>> Re-arrange seems to help the patch series by avoiding an cacheline miss.
> >>>>>
> >>>>> =========================
> >>>>> schbench
> >>>>> =========================
> >>>>>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
> >>>>> 1 Thread
> >>>>>   50.0th:        9.00  |    9.00        |        9.50
> >>>>>   75.0th:       10.50  |   10.00        |        9.50
> >>>>>   90.0th:       11.00  |   11.00        |       10.50
> >>>>>   95.0th:       11.00  |   11.00        |       11.00
> >>>>>   99.0th:       11.50  |   11.50        |       11.50
> >>>>>   99.5th:       12.50  |   12.00        |       12.00
> >>>>>   99.9th:       14.50  |   13.50        |       12.00
> >>>>> 2 Threads
> >>>>>   50.0th:        9.50  |    9.50        |        8.50
> >>>>>   75.0th:       11.00  |   10.50        |        9.50
> >>>>>   90.0th:       13.50  |   11.50        |       10.50
> >>>>>   95.0th:       14.00  |   12.00        |       11.00
> >>>>>   99.0th:       15.50  |   13.50        |       12.00
> >>>>>   99.5th:       16.00  |   14.00        |       12.00
> >>>>>   99.9th:       17.00  |   16.00        |       16.50
> >>>>> 4 Threads
> >>>>>   50.0th:       11.50  |   11.50        |       10.50
> >>>>>   75.0th:       13.50  |   12.50        |       12.50
> >>>>>   90.0th:       15.50  |   14.50        |       14.00
> >>>>>   95.0th:       16.50  |   15.50        |       14.50
> >>>>>   99.0th:       20.00  |   17.50        |       16.50
> >>>>>   99.5th:       20.50  |   18.50        |       17.00
> >>>>>   99.9th:       22.50  |   21.00        |       19.00
> >>>>> 8 Threads
> >>>>>   50.0th:       14.00  |   14.00        |       14.00
> >>>>>   75.0th:       16.00  |   16.00        |       16.00
> >>>>>   90.0th:       18.00  |   18.00        |       17.50
> >>>>>   95.0th:       18.50  |   18.50        |       18.50
> >>>>>   99.0th:       20.00  |   20.00        |       20.00
> >>>>>   99.5th:       20.50  |   21.50        |       21.00
> >>>>>   99.9th:       22.50  |   23.50        |       23.00
> >>>>> 16 Threads
> >>>>>   50.0th:       19.00  |   18.50        |       19.00
> >>>>>   75.0th:       23.00  |   22.50        |       23.00
> >>>>>   90.0th:       25.00  |   25.50        |       25.00
> >>>>>   95.0th:       26.50  |   26.50        |       26.00
> >>>>>   99.0th:       28.50  |   29.00        |       28.50
> >>>>>   99.5th:       31.00  |   30.00        |       30.00
> >>>>>   99.9th:     5626.00  | 4761.50        |       32.50
> >>>>> 32 Threads
> >>>>>   50.0th:       27.00  |   27.50        |       29.00
> >>>>>   75.0th:       35.50  |   36.50        |       38.50
> >>>>>   90.0th:       42.00  |   44.00        |       50.50
> >>>>>   95.0th:      447.50  | 2959.00        |     8544.00
> >>>>>   99.0th:     7372.00  | 17032.00       |    19136.00
> >>>>>   99.5th:    15360.00  | 19808.00       |    20704.00
> >>>>>   99.9th:    20640.00  | 30048.00       |    30048.00
> >>>>>
> >>>>> ====================
> >>>>> hackbench
> >>>>> ====================
> >>>>>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
> >>>>>
> >>>>> Process 10 Time:        0.35    |       0.42        |           0.41
> >>>>> Process 20 Time:        0.61    |       0.76        |           0.76
> >>>>> Process 30 Time:        0.87    |       1.06        |           1.05
> >>>>> thread 10 Time:         0.35    |       0.43        |           0.42
> >>>>> thread 20 Time:         0.66    |       0.79        |           0.78
> >>>>> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
> >>>>> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
> >>>>> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
> >>>>> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
> >>>>> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
> >>>>>
> >>>>>
> >>>>>>       struct list_head                group_node;
> >>>>>>       unsigned int                    on_rq;
> >>>>>>
> >>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >>>>>> index 093cc1af73dc..752fd364216c 100644
> >>>>>> --- a/kernel/sched/core.c
> >>>>>> +++ b/kernel/sched/core.c
> >>>>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
> >>>>>>       p->se.nr_migrations             = 0;
> >>>>>>       p->se.vruntime                  = 0;
> >>>>>>       INIT_LIST_HEAD(&p->se.group_node);
> >>>>>> +     RB_CLEAR_NODE(&p->se.latency_node);
> >>>>>>
> >>>>>>  #ifdef CONFIG_FAIR_GROUP_SCHED
> >>>>>>       p->se.cfs_rq                    = NULL;
> >>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >>>>>> index 125a6ff53378..e2aeb4511686 100644
> >>>>>> --- a/kernel/sched/fair.c
> >>>>>> +++ b/kernel/sched/fair.c
> >>>>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> >>>>>>
> >>>>>>       return __node_2_se(last);
> >>>>>>  }
> >>>>>> +#endif
> >>>>>>
> >>>>>> +/**************************************************************
> >>>>>> + * Scheduling class tree data structure manipulation methods:
> >>>>>> + * for latency
> >>>>>> + */
> >>>>>> +
> >>>>>> +static inline bool latency_before(struct sched_entity *a,
> >>>>>> +                             struct sched_entity *b)
> >>>>>> +{
> >>>>>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +#define __latency_node_2_se(node) \
> >>>>>> +     rb_entry((node), struct sched_entity, latency_node)
> >>>>>> +
> >>>>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> >>>>>> +{
> >>>>>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> >>>>>> +}
> >>>>>> +
> >>>>>> +/*
> >>>>>> + * Enqueue an entity into the latency rb-tree:
> >>>>>> + */
> >>>>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>>>> +{
> >>>>>> +
> >>>>>> +     /* Only latency sensitive entity can be added to the list */
> >>>>>> +     if (se->latency_offset >= 0)
> >>>>>> +             return;
> >>>>>> +
> >>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node))
> >>>>>> +             return;
> >>>>>> +
> >>>>>> +     /*
> >>>>>> +      * The entity is always added the latency list at wakeup.
> >>>>>> +      * Then, a not waking up entity that is put back in the list after an
> >>>>>> +      * execution time less than sysctl_sched_min_granularity, means that
> >>>>>> +      * the entity has been preempted by a higher sched class or an entity
> >>>>>> +      * with higher latency constraint. In thi case, the entity is also put
> >>>>>> +      * back in the latency list so it gets a chance to run 1st during the
> >>>>>> +      * next slice.
> >>>>>> +      */
> >>>>>> +     if (!(flags & ENQUEUE_WAKEUP)) {
> >>>>>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> >>>>>> +
> >>>>>> +             if (delta_exec >= sysctl_sched_min_granularity)
> >>>>>> +                     return;
> >>>>>> +     }
> >>>>>> +
> >>>>>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> >>>>>> +}
> >>>>>> +
> >>>>>> +/*
> >>>>>> + * Dequeue an entity from the latency rb-tree and return true if it was really
> >>>>>> + * part of the rb-tree:
> >>>>>> + */
> >>>>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >>>>>> +{
> >>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
> >>>>>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> >>>>>> +             RB_CLEAR_NODE(&se->latency_node);
> >>>>>> +             return true;
> >>>>>> +     }
> >>>>>> +
> >>>>>> +     return false;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> >>>>>> +{
> >>>>>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> >>>>>> +
> >>>>>> +     if (!left)
> >>>>>> +             return NULL;
> >>>>>> +
> >>>>>> +     return __latency_node_2_se(left);
> >>>>>> +}
> >>>>>> +
> >>>>>> +#ifdef CONFIG_SCHED_DEBUG
> >>>>>>  /**************************************************************
> >>>>>>   * Scheduling class statistics methods:
> >>>>>>   */
> >>>>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>>>>       check_schedstat_required();
> >>>>>>       update_stats_enqueue_fair(cfs_rq, se, flags);
> >>>>>>       check_spread(cfs_rq, se);
> >>>>>> -     if (!curr)
> >>>>>> +     if (!curr) {
> >>>>>>               __enqueue_entity(cfs_rq, se);
> >>>>>> +             __enqueue_latency(cfs_rq, se, flags);
> >>>>>> +     }
> >>>>>>       se->on_rq = 1;
> >>>>>>
> >>>>>>       if (cfs_rq->nr_running == 1) {
> >>>>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>>>>
> >>>>>>       clear_buddies(cfs_rq, se);
> >>>>>>
> >>>>>> -     if (se != cfs_rq->curr)
> >>>>>> +     if (se != cfs_rq->curr) {
> >>>>>>               __dequeue_entity(cfs_rq, se);
> >>>>>> +             __dequeue_latency(cfs_rq, se);
> >>>>>> +     }
> >>>>>>       se->on_rq = 0;
> >>>>>>       account_entity_dequeue(cfs_rq, se);
> >>>>>>
> >>>>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >>>>>>                */
> >>>>>>               update_stats_wait_end_fair(cfs_rq, se);
> >>>>>>               __dequeue_entity(cfs_rq, se);
> >>>>>> +             __dequeue_latency(cfs_rq, se);
> >>>>>>               update_load_avg(cfs_rq, se, UPDATE_TG);
> >>>>>>       }
> >>>>>>
> >>>>>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
> >>>>>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>>>>  {
> >>>>>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
> >>>>>> -     struct sched_entity *se;
> >>>>>> +     struct sched_entity *latency, *se;
> >>>>>>
> >>>>>>       /*
> >>>>>>        * If curr is set we have to see if its left of the leftmost entity
> >>>>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>>>>               se = cfs_rq->last;
> >>>>>>       }
> >>>>>>
> >>>>>> +     /* Check for latency sensitive entity waiting for running */
> >>>>>> +     latency = __pick_first_latency(cfs_rq);
> >>>>>> +     if (latency && (latency != se) &&
> >>>>>> +         wakeup_preempt_entity(latency, se) < 1)
> >>>>>> +             se = latency;
> >>>>>> +
> >>>>>>       return se;
> >>>>>>  }
> >>>>>>
> >>>>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> >>>>>>               update_stats_wait_start_fair(cfs_rq, prev);
> >>>>>>               /* Put 'current' back into the tree. */
> >>>>>>               __enqueue_entity(cfs_rq, prev);
> >>>>>> +             __enqueue_latency(cfs_rq, prev, 0);
> >>>>>>               /* in !on_rq case, update occurred at dequeue */
> >>>>>>               update_load_avg(cfs_rq, prev, 0);
> >>>>>>       }
> >>>>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> >>>>>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
> >>>>>>  {
> >>>>>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> >>>>>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
> >>>>>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> >>>>>>  #ifdef CONFIG_SMP
> >>>>>>       raw_spin_lock_init(&cfs_rq->removed.lock);
> >>>>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
> >>>>>>       se->my_q = cfs_rq;
> >>>>>>
> >>>>>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
> >>>>>> +     RB_CLEAR_NODE(&se->latency_node);
> >>>>>>
> >>>>>>       /* guarantee group entities always have weight */
> >>>>>>       update_load_set(&se->load, NICE_0_LOAD);
> >>>>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
> >>>>>>
> >>>>>>       for_each_possible_cpu(i) {
> >>>>>>               struct sched_entity *se = tg->se[i];
> >>>>>> +             struct rq *rq = cpu_rq(i);
> >>>>>> +             struct rq_flags rf;
> >>>>>> +             bool queued;
> >>>>>> +
> >>>>>> +             rq_lock_irqsave(rq, &rf);
> >>>>>>
> >>>>>> +             queued = __dequeue_latency(se->cfs_rq, se);
> >>>>>>               WRITE_ONCE(se->latency_offset, latency_offset);
> >>>>>> +             if (queued)
> >>>>>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
> >>>>>> +
> >>>>>> +
> >>>>>> +             rq_unlock_irqrestore(rq, &rf);
> >>>>>>       }
> >>>>>>
> >>>>>>       mutex_unlock(&shares_mutex);
> >>>>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> >>>>>> index 9a2e71231083..21dd309e98a9 100644
> >>>>>> --- a/kernel/sched/sched.h
> >>>>>> +++ b/kernel/sched/sched.h
> >>>>>> @@ -570,6 +570,7 @@ struct cfs_rq {
> >>>>>>  #endif
> >>>>>>
> >>>>>>       struct rb_root_cached   tasks_timeline;
> >>>>>> +     struct rb_root_cached   latency_timeline;
> >>>>>>
> >>>>>>       /*
> >>>>>>        * 'curr' points to currently running entity on this cfs_rq.
> 

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Shrikanth Hegde]

On 3/3/23 10:01 PM, Vincent Guittot wrote:
> Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
>>
>> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
>>> On 3/2/23 6:47 PM, Vincent Guittot wrote:
>>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
>>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
> [...]
>
>>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
>>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
>>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
>>>>>>> and other is running these benchmarks (group2). Set the latency nice
>>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
>>>>>>> Total of 96 CPU.
>>>>>>>
>>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
>>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
>>>>>>> and attach the numbers to series.
>>>>>>> Please suggest if any variation in the test i need to try.
>>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
>>>>>> run longer slice to move forward rather than preempting others all the
>>>>>> time
>>>>> hackbench still seems to regress in different latency nice values compared to
>>>>> baseline of 6.2 in this case. up to 50% in some cases.
>>>>>
>>>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
>>>>> running 2 CPU cgroups. No quota assigned.
>>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
>>>>> latency is not changed for this cgroup.
>>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
>>>>> nice values of 0, -20 and 19.
>>>> According to your other emails, you are using the cgroup interface and
>>>> not the task's one. Do I get it right ?
>>> right. I create cgroup, attach bash command with echo $$, 
>>> assign the latency nice to cgroup, and run hackbench from that bash prompt.
>>>
>>>> I haven't run test such tests in a cgroup but at least the test with
>>>> latency_nice == 0 should not make any noticeable difference. Does this
>>>> include the re-arrange patch that you have proposed previously ?
>>> No. This is only with V12 of the series.
>>>
>>>> Also, the tests that you did on v6, gave better result.
>>>> https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@linux.vnet.ibm.com/
>>>>
>>>> Are you running same tests or you changed something in the mean time ?
>>> Test machine got changed. 
>>> now i re-read my earlier mail. I see it was slightly different. 
>>> I had created only one cgroup and stress-ng was run
>>> without any cgroup. Let me try that scenario and get the numbers. 
>>
>> Tried the same method of testing i had done on V7 of the series. on this
>> machine hackbench still regress's both on V12 as well as V7 of the series.
>>
>> Created one cpu cgroup called cgroup1. created two bash prompts. 
>> assigned "bash $$" to cgroup1 and on other bash prompt running,
>> stress-ng --cpu=96 -l 50. Ran hackbench from cgroup1 prompt. 
>> assigned latency values to the cgroup1.
> I have tried to reproduce your results on some of my systems but I can't see
> the impacts that you are reporting below.
> The fact that your other platform was not impacted as well could imply that
> it's specific to this platform.
> In particular, the lat nice=0 case should not show any real impact as it
> should be similar to a nop. At least that what I can see in the tests on my
> platforms and Prateek on his.
>
> Nevertheless, could you try to run your tests with the changes below ?
> These are the only places which could have an impact even with lat nice = 0
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 8137bca80572..979571a98b28 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4991,8 +4991,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>         if (delta < offset)
>                 return;
>
> -       if ((delta > ideal_runtime) ||
> -           (delta > get_latency_max()))
> +       if (delta > ideal_runtime)
>                 resched_curr(rq_of(cfs_rq));
>  }
>
> @@ -7574,9 +7573,10 @@ static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *
>          * Otherwise, use the latency weight to evaluate how much scheduling
>          * delay is acceptable by se.
>          */
> -       if ((latency_offset < 0) || (curr->latency_offset < 0))
> +       if ((latency_offset < 0) || (curr->latency_offset < 0)) {
>                 latency_offset -= curr->latency_offset;
> -       latency_offset = min_t(long, latency_offset, get_latency_max());
> +               latency_offset = min_t(long, latency_offset, get_latency_max());
> +       }
>
>         return latency_offset;
>  }
> @@ -7635,7 +7635,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>          * for low priority task. Make sure that long sleeping task will get a
>          * chance to preempt current.
>          */
> -       gran = min_t(s64, gran, get_latency_max());
>
>         if (vdiff > gran)
>                 return 1;
>

Above patch helps. thank you.
Numbers are comparable to 6.2 and there is slight improvement. Much better than V12 numbers. 

type	   groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
		  |	        |lat nice=0| lat nice=-20| lat nice=+19

Process	      10  |	0.33    |   0.37   |   0.38     |   0.37
Process       20  |	0.61    |   0.67   |   0.68     |   0.67
Process	      30  |	0.85    |   0.95   |   0.95     |   0.96
Process	      40  |	1.10    |   1.20   |   1.20     |   1.21
Process	      50  |	1.34    |   1.47   |   1.44     |   1.45
Process	      60  |	1.57    |   1.70   |   1.71     |   1.70
thread	      10  |	0.36    |   0.40   |   0.39     |   0.39
thread	      20  |	0.65    |   0.72   |   0.71     |   0.71
Process(Pipe) 10  |	0.18	|   0.31   |   0.31	|   0.33
Process(Pipe) 20  |	0.32	|   0.51   |   0.50	|   0.50
Process(Pipe) 30  |	0.43	|   0.65   |   0.67	|   0.67
Process(Pipe) 40  |	0.57	|   0.82   |   0.83	|   0.83
Process(Pipe) 50  |	0.67	|   1.00   |   0.97	|   0.98
Process(Pipe) 60  |	0.81	|   1.13   |   1.11	|   1.12
thread(Pipe)  10  |	0.19	|   0.33   |   0.33	|   0.33
thread(Pipe)  20  |	0.34	|   0.53   |   0.51	|   0.52



type	   groups |   v6.2	|v6.2+ V12+ | v6.2 + V12+| v6.2 + V12
                  |             |above patch|above patch | above patch
		  |		|lat nice=0 |lat nice=-20| lat nice=+19

Process       10  |	0.36    |   0.33    |   0.34     |   0.34
Process       20  |	0.62    |   0.60    |   0.61     |   0.61
Process       30  |	0.87    |   0.84    |   0.85     |   0.84
Process       40  |	1.13    |   1.09    |   1.10     |   1.09
Process       50  |	1.38    |   1.33    |   1.33     |   1.34
Process       60  |	1.64    |   1.56    |   1.58     |   1.56
thread        10  |	0.35    |   0.35    |   0.35     |   0.35
thread        20  |	0.64    |   0.63    |   0.63     |   0.63
Process(Pipe) 10  |	0.20    |   0.18    |   0.18     |   0.18
Process(Pipe) 20  |	0.32    |   0.31    |   0.31     |   0.32
Process(Pipe) 30  |	0.44    |   0.43    |   0.43     |   0.43
Process(Pipe) 40  |	0.56    |   0.57    |   0.56     |   0.55
Process(Pipe) 50  |	0.70    |   0.67    |   0.67     |   0.67
Process(Pipe) 60  |	0.83    |   0.79    |   0.81     |   0.80
thread(Pipe)  10  |	0.21    |   0.19    |   0.19     |   0.19
thread(Pipe)  20  |	0.35    |   0.33    |   0.34     |   0.33


Do you want me to try any other experiment on this further?

>> I will try to run with only task's set with latency_nice=0 as well. 
>>
>> type	   groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
>> 		  |	        |lat nice=0| lat nice=-20| lat nice=+19
>>
>> Process	      10  |	0.33    |   0.37   |   0.38     |   0.37
>> Process       20  |	0.61    |   0.67   |   0.68     |   0.67
>> Process	      30  |	0.85    |   0.95   |   0.95     |   0.96
>> Process	      40  |	1.10    |   1.20   |   1.20     |   1.21
>> Process	      50  |	1.34    |   1.47   |   1.44     |   1.45
>> Process	      60  |	1.57    |   1.70   |   1.71     |   1.70
>> thread	      10  |	0.36    |   0.40   |   0.39     |   0.39
>> thread	      20  |	0.65    |   0.72   |   0.71     |   0.71
>> Process(Pipe) 10  |	0.18	|   0.31   |   0.31	|   0.33
>> Process(Pipe) 20  |	0.32	|   0.51   |   0.50	|   0.50
>> Process(Pipe) 30  |	0.43	|   0.65   |   0.67	|   0.67
>> Process(Pipe) 40  |	0.57	|   0.82   |   0.83	|   0.83
>> Process(Pipe) 50  |	0.67	|   1.00   |   0.97	|   0.98
>> Process(Pipe) 60  |	0.81	|   1.13   |   1.11	|   1.12
>> thread(Pipe)  10  |	0.19	|   0.33   |   0.33	|   0.33
>> thread(Pipe)  20  |	0.34	|   0.53   |   0.51	|   0.52
>>
>>
>>
>> type	   groups |   v6.2	|v6.2 + V7 | v6.2 + V7  | v6.2 + V7
>> 		  |		|lat nice=0|lat nice=-20| lat nice=+19
>> Process	      10  |	0.33    |   0.37   |   0.37     |   0.37
>> Process	      20  |	0.61    |   0.67   |   0.67     |   0.67
>> Process	      30  |	0.85    |   0.96   |   0.94     |   0.95
>> Process	      40  |	1.10    |   1.20   |   1.20     |   1.20
>> Process	      50  |	1.34    |   1.45   |   1.46     |   1.45
>> Process	      60  |	1.57    |   1.71   |   1.68     |   1.72
>> thread	      10  |	0.36    |   0.40   |   0.40     |   0.40
>> thread	      20  |	0.65    |   0.71   |   0.71     |   0.71
>> Process(Pipe) 10  |	0.18	|   0.30   |   0.30	|   0.31
>> Process(Pipe) 20  |	0.32	|   0.50   |   0.50	|   0.50
>> Process(Pipe) 30  |	0.43	|   0.67   |   0.67	|   0.66
>> Process(Pipe) 40  |	0.57	|   0.86   |   0.84	|   0.84
>> Process(Pipe) 50  |	0.67	|   0.99   |   0.97	|   0.97
>> Process(Pipe) 60  |	0.81	|   1.10   |   1.13	|   1.13
>> thread(Pipe)  10  |	0.19	|   0.34   |   0.34	|   0.33
>> thread(Pipe)  20  |	0.34	|   0.55   |   0.53	|   0.54
>>
>>>>> Numbers are average of 10 runs in each case. Time is in seconds
>>>>>
>>>>> type       groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
>>>>>                   |            | lat nice=0    | lat nice=-20| lat nice=+19
>>>>>                   |            |               |             |
>>>>> Process       10  |   0.36     |     0.41      |    0.43     |    0.42
>>>>> Process       20  |   0.62     |     0.76      |    0.75     |    0.75
>>>>> Process       30  |   0.87     |     1.05      |    1.04     |    1.06
>>>>> Process       40  |   1.13     |     1.34      |    1.33     |    1.33
>>>>> Process       50  |   1.38     |     1.62      |    1.66     |    1.63
>>>>> Process       60  |   1.64     |     1.91      |    1.97     |    1.90
>>>>> thread        10  |   0.35     |     0.41      |    0.44     |    0.42
>>>>> thread        20  |   0.64     |     0.78      |    0.77     |    0.79
>>>>> Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
>>>>> Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
>>>>> Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
>>>>> Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
>>>>> Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
>>>>> Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
>>>>> thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
>>>>> thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55
>>>>>
>>>>>
>>>>>
>>>>>>> Re-arrange seems to help the patch series by avoiding an cacheline miss.
>>>>>>>
>>>>>>> =========================
>>>>>>> schbench
>>>>>>> =========================
>>>>>>>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
>>>>>>> 1 Thread
>>>>>>>   50.0th:        9.00  |    9.00        |        9.50
>>>>>>>   75.0th:       10.50  |   10.00        |        9.50
>>>>>>>   90.0th:       11.00  |   11.00        |       10.50
>>>>>>>   95.0th:       11.00  |   11.00        |       11.00
>>>>>>>   99.0th:       11.50  |   11.50        |       11.50
>>>>>>>   99.5th:       12.50  |   12.00        |       12.00
>>>>>>>   99.9th:       14.50  |   13.50        |       12.00
>>>>>>> 2 Threads
>>>>>>>   50.0th:        9.50  |    9.50        |        8.50
>>>>>>>   75.0th:       11.00  |   10.50        |        9.50
>>>>>>>   90.0th:       13.50  |   11.50        |       10.50
>>>>>>>   95.0th:       14.00  |   12.00        |       11.00
>>>>>>>   99.0th:       15.50  |   13.50        |       12.00
>>>>>>>   99.5th:       16.00  |   14.00        |       12.00
>>>>>>>   99.9th:       17.00  |   16.00        |       16.50
>>>>>>> 4 Threads
>>>>>>>   50.0th:       11.50  |   11.50        |       10.50
>>>>>>>   75.0th:       13.50  |   12.50        |       12.50
>>>>>>>   90.0th:       15.50  |   14.50        |       14.00
>>>>>>>   95.0th:       16.50  |   15.50        |       14.50
>>>>>>>   99.0th:       20.00  |   17.50        |       16.50
>>>>>>>   99.5th:       20.50  |   18.50        |       17.00
>>>>>>>   99.9th:       22.50  |   21.00        |       19.00
>>>>>>> 8 Threads
>>>>>>>   50.0th:       14.00  |   14.00        |       14.00
>>>>>>>   75.0th:       16.00  |   16.00        |       16.00
>>>>>>>   90.0th:       18.00  |   18.00        |       17.50
>>>>>>>   95.0th:       18.50  |   18.50        |       18.50
>>>>>>>   99.0th:       20.00  |   20.00        |       20.00
>>>>>>>   99.5th:       20.50  |   21.50        |       21.00
>>>>>>>   99.9th:       22.50  |   23.50        |       23.00
>>>>>>> 16 Threads
>>>>>>>   50.0th:       19.00  |   18.50        |       19.00
>>>>>>>   75.0th:       23.00  |   22.50        |       23.00
>>>>>>>   90.0th:       25.00  |   25.50        |       25.00
>>>>>>>   95.0th:       26.50  |   26.50        |       26.00
>>>>>>>   99.0th:       28.50  |   29.00        |       28.50
>>>>>>>   99.5th:       31.00  |   30.00        |       30.00
>>>>>>>   99.9th:     5626.00  | 4761.50        |       32.50
>>>>>>> 32 Threads
>>>>>>>   50.0th:       27.00  |   27.50        |       29.00
>>>>>>>   75.0th:       35.50  |   36.50        |       38.50
>>>>>>>   90.0th:       42.00  |   44.00        |       50.50
>>>>>>>   95.0th:      447.50  | 2959.00        |     8544.00
>>>>>>>   99.0th:     7372.00  | 17032.00       |    19136.00
>>>>>>>   99.5th:    15360.00  | 19808.00       |    20704.00
>>>>>>>   99.9th:    20640.00  | 30048.00       |    30048.00
>>>>>>>
>>>>>>> ====================
>>>>>>> hackbench
>>>>>>> ====================
>>>>>>>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
>>>>>>>
>>>>>>> Process 10 Time:        0.35    |       0.42        |           0.41
>>>>>>> Process 20 Time:        0.61    |       0.76        |           0.76
>>>>>>> Process 30 Time:        0.87    |       1.06        |           1.05
>>>>>>> thread 10 Time:         0.35    |       0.43        |           0.42
>>>>>>> thread 20 Time:         0.66    |       0.79        |           0.78
>>>>>>> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
>>>>>>> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
>>>>>>> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
>>>>>>> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
>>>>>>> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
>>>>>>>
>>>>>>>
>>>>>>>>       struct list_head                group_node;
>>>>>>>>       unsigned int                    on_rq;
>>>>>>>>
>>>>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>>>>>> index 093cc1af73dc..752fd364216c 100644
>>>>>>>> --- a/kernel/sched/core.c
>>>>>>>> +++ b/kernel/sched/core.c
>>>>>>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
>>>>>>>>       p->se.nr_migrations             = 0;
>>>>>>>>       p->se.vruntime                  = 0;
>>>>>>>>       INIT_LIST_HEAD(&p->se.group_node);
>>>>>>>> +     RB_CLEAR_NODE(&p->se.latency_node);
>>>>>>>>
>>>>>>>>  #ifdef CONFIG_FAIR_GROUP_SCHED
>>>>>>>>       p->se.cfs_rq                    = NULL;
>>>>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>>>>>>> index 125a6ff53378..e2aeb4511686 100644
>>>>>>>> --- a/kernel/sched/fair.c
>>>>>>>> +++ b/kernel/sched/fair.c
>>>>>>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
>>>>>>>>
>>>>>>>>       return __node_2_se(last);
>>>>>>>>  }
>>>>>>>> +#endif
>>>>>>>>
>>>>>>>> +/**************************************************************
>>>>>>>> + * Scheduling class tree data structure manipulation methods:
>>>>>>>> + * for latency
>>>>>>>> + */
>>>>>>>> +
>>>>>>>> +static inline bool latency_before(struct sched_entity *a,
>>>>>>>> +                             struct sched_entity *b)
>>>>>>>> +{
>>>>>>>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +#define __latency_node_2_se(node) \
>>>>>>>> +     rb_entry((node), struct sched_entity, latency_node)
>>>>>>>> +
>>>>>>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
>>>>>>>> +{
>>>>>>>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +/*
>>>>>>>> + * Enqueue an entity into the latency rb-tree:
>>>>>>>> + */
>>>>>>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>>> +{
>>>>>>>> +
>>>>>>>> +     /* Only latency sensitive entity can be added to the list */
>>>>>>>> +     if (se->latency_offset >= 0)
>>>>>>>> +             return;
>>>>>>>> +
>>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node))
>>>>>>>> +             return;
>>>>>>>> +
>>>>>>>> +     /*
>>>>>>>> +      * The entity is always added the latency list at wakeup.
>>>>>>>> +      * Then, a not waking up entity that is put back in the list after an
>>>>>>>> +      * execution time less than sysctl_sched_min_granularity, means that
>>>>>>>> +      * the entity has been preempted by a higher sched class or an entity
>>>>>>>> +      * with higher latency constraint. In thi case, the entity is also put
>>>>>>>> +      * back in the latency list so it gets a chance to run 1st during the
>>>>>>>> +      * next slice.
>>>>>>>> +      */
>>>>>>>> +     if (!(flags & ENQUEUE_WAKEUP)) {
>>>>>>>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
>>>>>>>> +
>>>>>>>> +             if (delta_exec >= sysctl_sched_min_granularity)
>>>>>>>> +                     return;
>>>>>>>> +     }
>>>>>>>> +
>>>>>>>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +/*
>>>>>>>> + * Dequeue an entity from the latency rb-tree and return true if it was really
>>>>>>>> + * part of the rb-tree:
>>>>>>>> + */
>>>>>>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>>>> +{
>>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
>>>>>>>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
>>>>>>>> +             RB_CLEAR_NODE(&se->latency_node);
>>>>>>>> +             return true;
>>>>>>>> +     }
>>>>>>>> +
>>>>>>>> +     return false;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
>>>>>>>> +{
>>>>>>>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
>>>>>>>> +
>>>>>>>> +     if (!left)
>>>>>>>> +             return NULL;
>>>>>>>> +
>>>>>>>> +     return __latency_node_2_se(left);
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +#ifdef CONFIG_SCHED_DEBUG
>>>>>>>>  /**************************************************************
>>>>>>>>   * Scheduling class statistics methods:
>>>>>>>>   */
>>>>>>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>>>       check_schedstat_required();
>>>>>>>>       update_stats_enqueue_fair(cfs_rq, se, flags);
>>>>>>>>       check_spread(cfs_rq, se);
>>>>>>>> -     if (!curr)
>>>>>>>> +     if (!curr) {
>>>>>>>>               __enqueue_entity(cfs_rq, se);
>>>>>>>> +             __enqueue_latency(cfs_rq, se, flags);
>>>>>>>> +     }
>>>>>>>>       se->on_rq = 1;
>>>>>>>>
>>>>>>>>       if (cfs_rq->nr_running == 1) {
>>>>>>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>>>
>>>>>>>>       clear_buddies(cfs_rq, se);
>>>>>>>>
>>>>>>>> -     if (se != cfs_rq->curr)
>>>>>>>> +     if (se != cfs_rq->curr) {
>>>>>>>>               __dequeue_entity(cfs_rq, se);
>>>>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>>>>> +     }
>>>>>>>>       se->on_rq = 0;
>>>>>>>>       account_entity_dequeue(cfs_rq, se);
>>>>>>>>
>>>>>>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>>>>                */
>>>>>>>>               update_stats_wait_end_fair(cfs_rq, se);
>>>>>>>>               __dequeue_entity(cfs_rq, se);
>>>>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>>>>>               update_load_avg(cfs_rq, se, UPDATE_TG);
>>>>>>>>       }
>>>>>>>>
>>>>>>>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
>>>>>>>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>>>>  {
>>>>>>>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
>>>>>>>> -     struct sched_entity *se;
>>>>>>>> +     struct sched_entity *latency, *se;
>>>>>>>>
>>>>>>>>       /*
>>>>>>>>        * If curr is set we have to see if its left of the leftmost entity
>>>>>>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>>>>               se = cfs_rq->last;
>>>>>>>>       }
>>>>>>>>
>>>>>>>> +     /* Check for latency sensitive entity waiting for running */
>>>>>>>> +     latency = __pick_first_latency(cfs_rq);
>>>>>>>> +     if (latency && (latency != se) &&
>>>>>>>> +         wakeup_preempt_entity(latency, se) < 1)
>>>>>>>> +             se = latency;
>>>>>>>> +
>>>>>>>>       return se;
>>>>>>>>  }
>>>>>>>>
>>>>>>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
>>>>>>>>               update_stats_wait_start_fair(cfs_rq, prev);
>>>>>>>>               /* Put 'current' back into the tree. */
>>>>>>>>               __enqueue_entity(cfs_rq, prev);
>>>>>>>> +             __enqueue_latency(cfs_rq, prev, 0);
>>>>>>>>               /* in !on_rq case, update occurred at dequeue */
>>>>>>>>               update_load_avg(cfs_rq, prev, 0);
>>>>>>>>       }
>>>>>>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>>>>>>>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
>>>>>>>>  {
>>>>>>>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
>>>>>>>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
>>>>>>>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
>>>>>>>>  #ifdef CONFIG_SMP
>>>>>>>>       raw_spin_lock_init(&cfs_rq->removed.lock);
>>>>>>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
>>>>>>>>       se->my_q = cfs_rq;
>>>>>>>>
>>>>>>>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
>>>>>>>> +     RB_CLEAR_NODE(&se->latency_node);
>>>>>>>>
>>>>>>>>       /* guarantee group entities always have weight */
>>>>>>>>       update_load_set(&se->load, NICE_0_LOAD);
>>>>>>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
>>>>>>>>
>>>>>>>>       for_each_possible_cpu(i) {
>>>>>>>>               struct sched_entity *se = tg->se[i];
>>>>>>>> +             struct rq *rq = cpu_rq(i);
>>>>>>>> +             struct rq_flags rf;
>>>>>>>> +             bool queued;
>>>>>>>> +
>>>>>>>> +             rq_lock_irqsave(rq, &rf);
>>>>>>>>
>>>>>>>> +             queued = __dequeue_latency(se->cfs_rq, se);
>>>>>>>>               WRITE_ONCE(se->latency_offset, latency_offset);
>>>>>>>> +             if (queued)
>>>>>>>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +             rq_unlock_irqrestore(rq, &rf);
>>>>>>>>       }
>>>>>>>>
>>>>>>>>       mutex_unlock(&shares_mutex);
>>>>>>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
>>>>>>>> index 9a2e71231083..21dd309e98a9 100644
>>>>>>>> --- a/kernel/sched/sched.h
>>>>>>>> +++ b/kernel/sched/sched.h
>>>>>>>> @@ -570,6 +570,7 @@ struct cfs_rq {
>>>>>>>>  #endif
>>>>>>>>
>>>>>>>>       struct rb_root_cached   tasks_timeline;
>>>>>>>> +     struct rb_root_cached   latency_timeline;
>>>>>>>>
>>>>>>>>       /*
>>>>>>>>        * 'curr' points to currently running entity on this cfs_rq.

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Vincent Guittot]

On Sat, 4 Mar 2023 at 16:13, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>
>
>
> On 3/3/23 10:01 PM, Vincent Guittot wrote:
> > Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
> >>
> >> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
> >>> On 3/2/23 6:47 PM, Vincent Guittot wrote:
> >>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
> >>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
> > [...]
> >
> >>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
> >>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
> >>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
> >>>>>>> and other is running these benchmarks (group2). Set the latency nice
> >>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
> >>>>>>> Total of 96 CPU.
> >>>>>>>
> >>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
> >>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
> >>>>>>> and attach the numbers to series.
> >>>>>>> Please suggest if any variation in the test i need to try.
> >>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
> >>>>>> run longer slice to move forward rather than preempting others all the
> >>>>>> time
> >>>>> hackbench still seems to regress in different latency nice values compared to
> >>>>> baseline of 6.2 in this case. up to 50% in some cases.
> >>>>>
> >>>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
> >>>>> running 2 CPU cgroups. No quota assigned.
> >>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
> >>>>> latency is not changed for this cgroup.
> >>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
> >>>>> nice values of 0, -20 and 19.
> >>>> According to your other emails, you are using the cgroup interface and
> >>>> not the task's one. Do I get it right ?
> >>> right. I create cgroup, attach bash command with echo $$,
> >>> assign the latency nice to cgroup, and run hackbench from that bash prompt.
> >>>
> >>>> I haven't run test such tests in a cgroup but at least the test with
> >>>> latency_nice == 0 should not make any noticeable difference. Does this
> >>>> include the re-arrange patch that you have proposed previously ?
> >>> No. This is only with V12 of the series.
> >>>
> >>>> Also, the tests that you did on v6, gave better result.
> >>>> https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@linux.vnet.ibm.com/
> >>>>
> >>>> Are you running same tests or you changed something in the mean time ?
> >>> Test machine got changed.
> >>> now i re-read my earlier mail. I see it was slightly different.
> >>> I had created only one cgroup and stress-ng was run
> >>> without any cgroup. Let me try that scenario and get the numbers.
> >>
> >> Tried the same method of testing i had done on V7 of the series. on this
> >> machine hackbench still regress's both on V12 as well as V7 of the series.
> >>
> >> Created one cpu cgroup called cgroup1. created two bash prompts.
> >> assigned "bash $$" to cgroup1 and on other bash prompt running,
> >> stress-ng --cpu=96 -l 50. Ran hackbench from cgroup1 prompt.
> >> assigned latency values to the cgroup1.
> > I have tried to reproduce your results on some of my systems but I can't see
> > the impacts that you are reporting below.
> > The fact that your other platform was not impacted as well could imply that
> > it's specific to this platform.
> > In particular, the lat nice=0 case should not show any real impact as it
> > should be similar to a nop. At least that what I can see in the tests on my
> > platforms and Prateek on his.
> >
> > Nevertheless, could you try to run your tests with the changes below ?
> > These are the only places which could have an impact even with lat nice = 0
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 8137bca80572..979571a98b28 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -4991,8 +4991,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >         if (delta < offset)
> >                 return;
> >
> > -       if ((delta > ideal_runtime) ||
> > -           (delta > get_latency_max()))
> > +       if (delta > ideal_runtime)
> >                 resched_curr(rq_of(cfs_rq));
> >  }
> >
> > @@ -7574,9 +7573,10 @@ static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *
> >          * Otherwise, use the latency weight to evaluate how much scheduling
> >          * delay is acceptable by se.
> >          */
> > -       if ((latency_offset < 0) || (curr->latency_offset < 0))
> > +       if ((latency_offset < 0) || (curr->latency_offset < 0)) {
> >                 latency_offset -= curr->latency_offset;
> > -       latency_offset = min_t(long, latency_offset, get_latency_max());
> > +               latency_offset = min_t(long, latency_offset, get_latency_max());
> > +       }
> >
> >         return latency_offset;
> >  }
> > @@ -7635,7 +7635,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >          * for low priority task. Make sure that long sleeping task will get a
> >          * chance to preempt current.
> >          */
> > -       gran = min_t(s64, gran, get_latency_max());
> >
> >         if (vdiff > gran)
> >                 return 1;
> >
>
> Above patch helps. thank you.

Great. At least we have narrow the problem to one f the 3 changes.

> Numbers are comparable to 6.2 and there is slight improvement. Much better than V12 numbers.
>
> type       groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
>                   |             |lat nice=0| lat nice=-20| lat nice=+19
>
> Process       10  |     0.33    |   0.37   |   0.38     |   0.37
> Process       20  |     0.61    |   0.67   |   0.68     |   0.67
> Process       30  |     0.85    |   0.95   |   0.95     |   0.96
> Process       40  |     1.10    |   1.20   |   1.20     |   1.21
> Process       50  |     1.34    |   1.47   |   1.44     |   1.45
> Process       60  |     1.57    |   1.70   |   1.71     |   1.70
> thread        10  |     0.36    |   0.40   |   0.39     |   0.39
> thread        20  |     0.65    |   0.72   |   0.71     |   0.71
> Process(Pipe) 10  |     0.18    |   0.31   |   0.31     |   0.33
> Process(Pipe) 20  |     0.32    |   0.51   |   0.50     |   0.50
> Process(Pipe) 30  |     0.43    |   0.65   |   0.67     |   0.67
> Process(Pipe) 40  |     0.57    |   0.82   |   0.83     |   0.83
> Process(Pipe) 50  |     0.67    |   1.00   |   0.97     |   0.98
> Process(Pipe) 60  |     0.81    |   1.13   |   1.11     |   1.12
> thread(Pipe)  10  |     0.19    |   0.33   |   0.33     |   0.33
> thread(Pipe)  20  |     0.34    |   0.53   |   0.51     |   0.52
>
>
>
> type       groups |   v6.2      |v6.2+ V12+ | v6.2 + V12+| v6.2 + V12
>                   |             |above patch|above patch | above patch
>                   |             |lat nice=0 |lat nice=-20| lat nice=+19
>
> Process       10  |     0.36    |   0.33    |   0.34     |   0.34
> Process       20  |     0.62    |   0.60    |   0.61     |   0.61
> Process       30  |     0.87    |   0.84    |   0.85     |   0.84
> Process       40  |     1.13    |   1.09    |   1.10     |   1.09
> Process       50  |     1.38    |   1.33    |   1.33     |   1.34
> Process       60  |     1.64    |   1.56    |   1.58     |   1.56
> thread        10  |     0.35    |   0.35    |   0.35     |   0.35
> thread        20  |     0.64    |   0.63    |   0.63     |   0.63
> Process(Pipe) 10  |     0.20    |   0.18    |   0.18     |   0.18
> Process(Pipe) 20  |     0.32    |   0.31    |   0.31     |   0.32
> Process(Pipe) 30  |     0.44    |   0.43    |   0.43     |   0.43
> Process(Pipe) 40  |     0.56    |   0.57    |   0.56     |   0.55
> Process(Pipe) 50  |     0.70    |   0.67    |   0.67     |   0.67
> Process(Pipe) 60  |     0.83    |   0.79    |   0.81     |   0.80
> thread(Pipe)  10  |     0.21    |   0.19    |   0.19     |   0.19
> thread(Pipe)  20  |     0.35    |   0.33    |   0.34     |   0.33
>
>
> Do you want me to try any other experiment on this further?

Yes, would be good to know which of the 3 changes in the patch create
the regression

I suspect the 1st change to be the root cause of your problem but It
would be good if you can confirm my assumption with some tests

Thanks

>
> >> I will try to run with only task's set with latency_nice=0 as well.
> >>
> >> type    groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
> >>                |             |lat nice=0| lat nice=-20| lat nice=+19
> >>
> >> Process            10  |     0.33    |   0.37   |   0.38     |   0.37
> >> Process       20  |  0.61    |   0.67   |   0.68     |   0.67
> >> Process            30  |     0.85    |   0.95   |   0.95     |   0.96
> >> Process            40  |     1.10    |   1.20   |   1.20     |   1.21
> >> Process            50  |     1.34    |   1.47   |   1.44     |   1.45
> >> Process            60  |     1.57    |   1.70   |   1.71     |   1.70
> >> thread             10  |     0.36    |   0.40   |   0.39     |   0.39
> >> thread             20  |     0.65    |   0.72   |   0.71     |   0.71
> >> Process(Pipe) 10  |  0.18    |   0.31   |   0.31     |   0.33
> >> Process(Pipe) 20  |  0.32    |   0.51   |   0.50     |   0.50
> >> Process(Pipe) 30  |  0.43    |   0.65   |   0.67     |   0.67
> >> Process(Pipe) 40  |  0.57    |   0.82   |   0.83     |   0.83
> >> Process(Pipe) 50  |  0.67    |   1.00   |   0.97     |   0.98
> >> Process(Pipe) 60  |  0.81    |   1.13   |   1.11     |   1.12
> >> thread(Pipe)  10  |  0.19    |   0.33   |   0.33     |   0.33
> >> thread(Pipe)  20  |  0.34    |   0.53   |   0.51     |   0.52
> >>
> >>
> >>
> >> type    groups |   v6.2      |v6.2 + V7 | v6.2 + V7  | v6.2 + V7
> >>                |             |lat nice=0|lat nice=-20| lat nice=+19
> >> Process            10  |     0.33    |   0.37   |   0.37     |   0.37
> >> Process            20  |     0.61    |   0.67   |   0.67     |   0.67
> >> Process            30  |     0.85    |   0.96   |   0.94     |   0.95
> >> Process            40  |     1.10    |   1.20   |   1.20     |   1.20
> >> Process            50  |     1.34    |   1.45   |   1.46     |   1.45
> >> Process            60  |     1.57    |   1.71   |   1.68     |   1.72
> >> thread             10  |     0.36    |   0.40   |   0.40     |   0.40
> >> thread             20  |     0.65    |   0.71   |   0.71     |   0.71
> >> Process(Pipe) 10  |  0.18    |   0.30   |   0.30     |   0.31
> >> Process(Pipe) 20  |  0.32    |   0.50   |   0.50     |   0.50
> >> Process(Pipe) 30  |  0.43    |   0.67   |   0.67     |   0.66
> >> Process(Pipe) 40  |  0.57    |   0.86   |   0.84     |   0.84
> >> Process(Pipe) 50  |  0.67    |   0.99   |   0.97     |   0.97
> >> Process(Pipe) 60  |  0.81    |   1.10   |   1.13     |   1.13
> >> thread(Pipe)  10  |  0.19    |   0.34   |   0.34     |   0.33
> >> thread(Pipe)  20  |  0.34    |   0.55   |   0.53     |   0.54
> >>
> >>>>> Numbers are average of 10 runs in each case. Time is in seconds
> >>>>>
> >>>>> type       groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
> >>>>>                   |            | lat nice=0    | lat nice=-20| lat nice=+19
> >>>>>                   |            |               |             |
> >>>>> Process       10  |   0.36     |     0.41      |    0.43     |    0.42
> >>>>> Process       20  |   0.62     |     0.76      |    0.75     |    0.75
> >>>>> Process       30  |   0.87     |     1.05      |    1.04     |    1.06
> >>>>> Process       40  |   1.13     |     1.34      |    1.33     |    1.33
> >>>>> Process       50  |   1.38     |     1.62      |    1.66     |    1.63
> >>>>> Process       60  |   1.64     |     1.91      |    1.97     |    1.90
> >>>>> thread        10  |   0.35     |     0.41      |    0.44     |    0.42
> >>>>> thread        20  |   0.64     |     0.78      |    0.77     |    0.79
> >>>>> Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
> >>>>> Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
> >>>>> Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
> >>>>> Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
> >>>>> Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
> >>>>> Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
> >>>>> thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
> >>>>> thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55
> >>>>>
> >>>>>
> >>>>>
> >>>>>>> Re-arrange seems to help the patch series by avoiding an cacheline miss.
> >>>>>>>
> >>>>>>> =========================
> >>>>>>> schbench
> >>>>>>> =========================
> >>>>>>>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
> >>>>>>> 1 Thread
> >>>>>>>   50.0th:        9.00  |    9.00        |        9.50
> >>>>>>>   75.0th:       10.50  |   10.00        |        9.50
> >>>>>>>   90.0th:       11.00  |   11.00        |       10.50
> >>>>>>>   95.0th:       11.00  |   11.00        |       11.00
> >>>>>>>   99.0th:       11.50  |   11.50        |       11.50
> >>>>>>>   99.5th:       12.50  |   12.00        |       12.00
> >>>>>>>   99.9th:       14.50  |   13.50        |       12.00
> >>>>>>> 2 Threads
> >>>>>>>   50.0th:        9.50  |    9.50        |        8.50
> >>>>>>>   75.0th:       11.00  |   10.50        |        9.50
> >>>>>>>   90.0th:       13.50  |   11.50        |       10.50
> >>>>>>>   95.0th:       14.00  |   12.00        |       11.00
> >>>>>>>   99.0th:       15.50  |   13.50        |       12.00
> >>>>>>>   99.5th:       16.00  |   14.00        |       12.00
> >>>>>>>   99.9th:       17.00  |   16.00        |       16.50
> >>>>>>> 4 Threads
> >>>>>>>   50.0th:       11.50  |   11.50        |       10.50
> >>>>>>>   75.0th:       13.50  |   12.50        |       12.50
> >>>>>>>   90.0th:       15.50  |   14.50        |       14.00
> >>>>>>>   95.0th:       16.50  |   15.50        |       14.50
> >>>>>>>   99.0th:       20.00  |   17.50        |       16.50
> >>>>>>>   99.5th:       20.50  |   18.50        |       17.00
> >>>>>>>   99.9th:       22.50  |   21.00        |       19.00
> >>>>>>> 8 Threads
> >>>>>>>   50.0th:       14.00  |   14.00        |       14.00
> >>>>>>>   75.0th:       16.00  |   16.00        |       16.00
> >>>>>>>   90.0th:       18.00  |   18.00        |       17.50
> >>>>>>>   95.0th:       18.50  |   18.50        |       18.50
> >>>>>>>   99.0th:       20.00  |   20.00        |       20.00
> >>>>>>>   99.5th:       20.50  |   21.50        |       21.00
> >>>>>>>   99.9th:       22.50  |   23.50        |       23.00
> >>>>>>> 16 Threads
> >>>>>>>   50.0th:       19.00  |   18.50        |       19.00
> >>>>>>>   75.0th:       23.00  |   22.50        |       23.00
> >>>>>>>   90.0th:       25.00  |   25.50        |       25.00
> >>>>>>>   95.0th:       26.50  |   26.50        |       26.00
> >>>>>>>   99.0th:       28.50  |   29.00        |       28.50
> >>>>>>>   99.5th:       31.00  |   30.00        |       30.00
> >>>>>>>   99.9th:     5626.00  | 4761.50        |       32.50
> >>>>>>> 32 Threads
> >>>>>>>   50.0th:       27.00  |   27.50        |       29.00
> >>>>>>>   75.0th:       35.50  |   36.50        |       38.50
> >>>>>>>   90.0th:       42.00  |   44.00        |       50.50
> >>>>>>>   95.0th:      447.50  | 2959.00        |     8544.00
> >>>>>>>   99.0th:     7372.00  | 17032.00       |    19136.00
> >>>>>>>   99.5th:    15360.00  | 19808.00       |    20704.00
> >>>>>>>   99.9th:    20640.00  | 30048.00       |    30048.00
> >>>>>>>
> >>>>>>> ====================
> >>>>>>> hackbench
> >>>>>>> ====================
> >>>>>>>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
> >>>>>>>
> >>>>>>> Process 10 Time:        0.35    |       0.42        |           0.41
> >>>>>>> Process 20 Time:        0.61    |       0.76        |           0.76
> >>>>>>> Process 30 Time:        0.87    |       1.06        |           1.05
> >>>>>>> thread 10 Time:         0.35    |       0.43        |           0.42
> >>>>>>> thread 20 Time:         0.66    |       0.79        |           0.78
> >>>>>>> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
> >>>>>>> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
> >>>>>>> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
> >>>>>>> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
> >>>>>>> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
> >>>>>>>
> >>>>>>>
> >>>>>>>>       struct list_head                group_node;
> >>>>>>>>       unsigned int                    on_rq;
> >>>>>>>>
> >>>>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >>>>>>>> index 093cc1af73dc..752fd364216c 100644
> >>>>>>>> --- a/kernel/sched/core.c
> >>>>>>>> +++ b/kernel/sched/core.c
> >>>>>>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
> >>>>>>>>       p->se.nr_migrations             = 0;
> >>>>>>>>       p->se.vruntime                  = 0;
> >>>>>>>>       INIT_LIST_HEAD(&p->se.group_node);
> >>>>>>>> +     RB_CLEAR_NODE(&p->se.latency_node);
> >>>>>>>>
> >>>>>>>>  #ifdef CONFIG_FAIR_GROUP_SCHED
> >>>>>>>>       p->se.cfs_rq                    = NULL;
> >>>>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >>>>>>>> index 125a6ff53378..e2aeb4511686 100644
> >>>>>>>> --- a/kernel/sched/fair.c
> >>>>>>>> +++ b/kernel/sched/fair.c
> >>>>>>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> >>>>>>>>
> >>>>>>>>       return __node_2_se(last);
> >>>>>>>>  }
> >>>>>>>> +#endif
> >>>>>>>>
> >>>>>>>> +/**************************************************************
> >>>>>>>> + * Scheduling class tree data structure manipulation methods:
> >>>>>>>> + * for latency
> >>>>>>>> + */
> >>>>>>>> +
> >>>>>>>> +static inline bool latency_before(struct sched_entity *a,
> >>>>>>>> +                             struct sched_entity *b)
> >>>>>>>> +{
> >>>>>>>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> >>>>>>>> +}
> >>>>>>>> +
> >>>>>>>> +#define __latency_node_2_se(node) \
> >>>>>>>> +     rb_entry((node), struct sched_entity, latency_node)
> >>>>>>>> +
> >>>>>>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> >>>>>>>> +{
> >>>>>>>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> >>>>>>>> +}
> >>>>>>>> +
> >>>>>>>> +/*
> >>>>>>>> + * Enqueue an entity into the latency rb-tree:
> >>>>>>>> + */
> >>>>>>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>>>>>> +{
> >>>>>>>> +
> >>>>>>>> +     /* Only latency sensitive entity can be added to the list */
> >>>>>>>> +     if (se->latency_offset >= 0)
> >>>>>>>> +             return;
> >>>>>>>> +
> >>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node))
> >>>>>>>> +             return;
> >>>>>>>> +
> >>>>>>>> +     /*
> >>>>>>>> +      * The entity is always added the latency list at wakeup.
> >>>>>>>> +      * Then, a not waking up entity that is put back in the list after an
> >>>>>>>> +      * execution time less than sysctl_sched_min_granularity, means that
> >>>>>>>> +      * the entity has been preempted by a higher sched class or an entity
> >>>>>>>> +      * with higher latency constraint. In thi case, the entity is also put
> >>>>>>>> +      * back in the latency list so it gets a chance to run 1st during the
> >>>>>>>> +      * next slice.
> >>>>>>>> +      */
> >>>>>>>> +     if (!(flags & ENQUEUE_WAKEUP)) {
> >>>>>>>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> >>>>>>>> +
> >>>>>>>> +             if (delta_exec >= sysctl_sched_min_granularity)
> >>>>>>>> +                     return;
> >>>>>>>> +     }
> >>>>>>>> +
> >>>>>>>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> >>>>>>>> +}
> >>>>>>>> +
> >>>>>>>> +/*
> >>>>>>>> + * Dequeue an entity from the latency rb-tree and return true if it was really
> >>>>>>>> + * part of the rb-tree:
> >>>>>>>> + */
> >>>>>>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >>>>>>>> +{
> >>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
> >>>>>>>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> >>>>>>>> +             RB_CLEAR_NODE(&se->latency_node);
> >>>>>>>> +             return true;
> >>>>>>>> +     }
> >>>>>>>> +
> >>>>>>>> +     return false;
> >>>>>>>> +}
> >>>>>>>> +
> >>>>>>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> >>>>>>>> +{
> >>>>>>>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> >>>>>>>> +
> >>>>>>>> +     if (!left)
> >>>>>>>> +             return NULL;
> >>>>>>>> +
> >>>>>>>> +     return __latency_node_2_se(left);
> >>>>>>>> +}
> >>>>>>>> +
> >>>>>>>> +#ifdef CONFIG_SCHED_DEBUG
> >>>>>>>>  /**************************************************************
> >>>>>>>>   * Scheduling class statistics methods:
> >>>>>>>>   */
> >>>>>>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>>>>>>       check_schedstat_required();
> >>>>>>>>       update_stats_enqueue_fair(cfs_rq, se, flags);
> >>>>>>>>       check_spread(cfs_rq, se);
> >>>>>>>> -     if (!curr)
> >>>>>>>> +     if (!curr) {
> >>>>>>>>               __enqueue_entity(cfs_rq, se);
> >>>>>>>> +             __enqueue_latency(cfs_rq, se, flags);
> >>>>>>>> +     }
> >>>>>>>>       se->on_rq = 1;
> >>>>>>>>
> >>>>>>>>       if (cfs_rq->nr_running == 1) {
> >>>>>>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>>>>>>
> >>>>>>>>       clear_buddies(cfs_rq, se);
> >>>>>>>>
> >>>>>>>> -     if (se != cfs_rq->curr)
> >>>>>>>> +     if (se != cfs_rq->curr) {
> >>>>>>>>               __dequeue_entity(cfs_rq, se);
> >>>>>>>> +             __dequeue_latency(cfs_rq, se);
> >>>>>>>> +     }
> >>>>>>>>       se->on_rq = 0;
> >>>>>>>>       account_entity_dequeue(cfs_rq, se);
> >>>>>>>>
> >>>>>>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >>>>>>>>                */
> >>>>>>>>               update_stats_wait_end_fair(cfs_rq, se);
> >>>>>>>>               __dequeue_entity(cfs_rq, se);
> >>>>>>>> +             __dequeue_latency(cfs_rq, se);
> >>>>>>>>               update_load_avg(cfs_rq, se, UPDATE_TG);
> >>>>>>>>       }
> >>>>>>>>
> >>>>>>>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
> >>>>>>>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>>>>>>  {
> >>>>>>>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
> >>>>>>>> -     struct sched_entity *se;
> >>>>>>>> +     struct sched_entity *latency, *se;
> >>>>>>>>
> >>>>>>>>       /*
> >>>>>>>>        * If curr is set we have to see if its left of the leftmost entity
> >>>>>>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>>>>>>               se = cfs_rq->last;
> >>>>>>>>       }
> >>>>>>>>
> >>>>>>>> +     /* Check for latency sensitive entity waiting for running */
> >>>>>>>> +     latency = __pick_first_latency(cfs_rq);
> >>>>>>>> +     if (latency && (latency != se) &&
> >>>>>>>> +         wakeup_preempt_entity(latency, se) < 1)
> >>>>>>>> +             se = latency;
> >>>>>>>> +
> >>>>>>>>       return se;
> >>>>>>>>  }
> >>>>>>>>
> >>>>>>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> >>>>>>>>               update_stats_wait_start_fair(cfs_rq, prev);
> >>>>>>>>               /* Put 'current' back into the tree. */
> >>>>>>>>               __enqueue_entity(cfs_rq, prev);
> >>>>>>>> +             __enqueue_latency(cfs_rq, prev, 0);
> >>>>>>>>               /* in !on_rq case, update occurred at dequeue */
> >>>>>>>>               update_load_avg(cfs_rq, prev, 0);
> >>>>>>>>       }
> >>>>>>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> >>>>>>>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
> >>>>>>>>  {
> >>>>>>>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> >>>>>>>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
> >>>>>>>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> >>>>>>>>  #ifdef CONFIG_SMP
> >>>>>>>>       raw_spin_lock_init(&cfs_rq->removed.lock);
> >>>>>>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
> >>>>>>>>       se->my_q = cfs_rq;
> >>>>>>>>
> >>>>>>>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
> >>>>>>>> +     RB_CLEAR_NODE(&se->latency_node);
> >>>>>>>>
> >>>>>>>>       /* guarantee group entities always have weight */
> >>>>>>>>       update_load_set(&se->load, NICE_0_LOAD);
> >>>>>>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
> >>>>>>>>
> >>>>>>>>       for_each_possible_cpu(i) {
> >>>>>>>>               struct sched_entity *se = tg->se[i];
> >>>>>>>> +             struct rq *rq = cpu_rq(i);
> >>>>>>>> +             struct rq_flags rf;
> >>>>>>>> +             bool queued;
> >>>>>>>> +
> >>>>>>>> +             rq_lock_irqsave(rq, &rf);
> >>>>>>>>
> >>>>>>>> +             queued = __dequeue_latency(se->cfs_rq, se);
> >>>>>>>>               WRITE_ONCE(se->latency_offset, latency_offset);
> >>>>>>>> +             if (queued)
> >>>>>>>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
> >>>>>>>> +
> >>>>>>>> +
> >>>>>>>> +             rq_unlock_irqrestore(rq, &rf);
> >>>>>>>>       }
> >>>>>>>>
> >>>>>>>>       mutex_unlock(&shares_mutex);
> >>>>>>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> >>>>>>>> index 9a2e71231083..21dd309e98a9 100644
> >>>>>>>> --- a/kernel/sched/sched.h
> >>>>>>>> +++ b/kernel/sched/sched.h
> >>>>>>>> @@ -570,6 +570,7 @@ struct cfs_rq {
> >>>>>>>>  #endif
> >>>>>>>>
> >>>>>>>>       struct rb_root_cached   tasks_timeline;
> >>>>>>>> +     struct rb_root_cached   latency_timeline;
> >>>>>>>>
> >>>>>>>>       /*
> >>>>>>>>        * 'curr' points to currently running entity on this cfs_rq.
>

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Shrikanth Hegde]

On 3/5/23 6:33 PM, Vincent Guittot wrote:
> On Sat, 4 Mar 2023 at 16:13, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>
>>
>> On 3/3/23 10:01 PM, Vincent Guittot wrote:
>>> Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
>>>> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
>>>>> On 3/2/23 6:47 PM, Vincent Guittot wrote:
>>>>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
>>>>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
>>> [...]
>>>
>>>>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
>>>>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
>>>>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
>>>>>>>>> and other is running these benchmarks (group2). Set the latency nice
>>>>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
>>>>>>>>> Total of 96 CPU.
>>>>>>>>>
>>>>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
>>>>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
>>>>>>>>> and attach the numbers to series.
>>>>>>>>> Please suggest if any variation in the test i need to try.
>>>>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
>>>>>>>> run longer slice to move forward rather than preempting others all the
>>>>>>>> time
>>>>>>> hackbench still seems to regress in different latency nice values compared to
>>>>>>> baseline of 6.2 in this case. up to 50% in some cases.
>>>>>>>
>>>>>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
>>>>>>> running 2 CPU cgroups. No quota assigned.
>>>>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
>>>>>>> latency is not changed for this cgroup.
>>>>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
>>>>>>> nice values of 0, -20 and 19.
>>>>>> According to your other emails, you are using the cgroup interface and
>>>>>> not the task's one. Do I get it right ?
>>>>> right. I create cgroup, attach bash command with echo $$,
>>>>> assign the latency nice to cgroup, and run hackbench from that bash prompt.
>>>>>
>>>>>> I haven't run test such tests in a cgroup but at least the test with
>>>>>> latency_nice == 0 should not make any noticeable difference. Does this
>>>>>> include the re-arrange patch that you have proposed previously ?
>>>>> No. This is only with V12 of the series.
>>>>>
>>>>>> Also, the tests that you did on v6, gave better result.
>>>>>> https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@linux.vnet.ibm.com/
>>>>>>
>>>>>> Are you running same tests or you changed something in the mean time ?
>>>>> Test machine got changed.
>>>>> now i re-read my earlier mail. I see it was slightly different.
>>>>> I had created only one cgroup and stress-ng was run
>>>>> without any cgroup. Let me try that scenario and get the numbers.
>>>> Tried the same method of testing i had done on V7 of the series. on this
>>>> machine hackbench still regress's both on V12 as well as V7 of the series.
>>>>
>>>> Created one cpu cgroup called cgroup1. created two bash prompts.
>>>> assigned "bash $$" to cgroup1 and on other bash prompt running,
>>>> stress-ng --cpu=96 -l 50. Ran hackbench from cgroup1 prompt.
>>>> assigned latency values to the cgroup1.
>>> I have tried to reproduce your results on some of my systems but I can't see
>>> the impacts that you are reporting below.
>>> The fact that your other platform was not impacted as well could imply that
>>> it's specific to this platform.
>>> In particular, the lat nice=0 case should not show any real impact as it
>>> should be similar to a nop. At least that what I can see in the tests on my
>>> platforms and Prateek on his.
>>>
>>> Nevertheless, could you try to run your tests with the changes below ?
>>> These are the only places which could have an impact even with lat nice = 0
>>>
>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>> index 8137bca80572..979571a98b28 100644
>>> --- a/kernel/sched/fair.c
>>> +++ b/kernel/sched/fair.c
>>> @@ -4991,8 +4991,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>         if (delta < offset)
>>>                 return;
>>>
>>> -       if ((delta > ideal_runtime) ||
>>> -           (delta > get_latency_max()))
>>> +       if (delta > ideal_runtime)
>>>                 resched_curr(rq_of(cfs_rq));
>>>  }
>>>
>>> @@ -7574,9 +7573,10 @@ static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *
>>>          * Otherwise, use the latency weight to evaluate how much scheduling
>>>          * delay is acceptable by se.
>>>          */
>>> -       if ((latency_offset < 0) || (curr->latency_offset < 0))
>>> +       if ((latency_offset < 0) || (curr->latency_offset < 0)) {
>>>                 latency_offset -= curr->latency_offset;
>>> -       latency_offset = min_t(long, latency_offset, get_latency_max());
>>> +               latency_offset = min_t(long, latency_offset, get_latency_max());
>>> +       }
>>>
>>>         return latency_offset;
>>>  }
>>> @@ -7635,7 +7635,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>          * for low priority task. Make sure that long sleeping task will get a
>>>          * chance to preempt current.
>>>          */
>>> -       gran = min_t(s64, gran, get_latency_max());
>>>
>>>         if (vdiff > gran)
>>>                 return 1;
>>>
>> Above patch helps. thank you.
> Great. At least we have narrow the problem to one f the 3 changes.
>
>> Numbers are comparable to 6.2 and there is slight improvement. Much better than V12 numbers.
>>
>> type       groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
>>                   |             |lat nice=0| lat nice=-20| lat nice=+19
>>
>> Process       10  |     0.33    |   0.37   |   0.38     |   0.37
>> Process       20  |     0.61    |   0.67   |   0.68     |   0.67
>> Process       30  |     0.85    |   0.95   |   0.95     |   0.96
>> Process       40  |     1.10    |   1.20   |   1.20     |   1.21
>> Process       50  |     1.34    |   1.47   |   1.44     |   1.45
>> Process       60  |     1.57    |   1.70   |   1.71     |   1.70
>> thread        10  |     0.36    |   0.40   |   0.39     |   0.39
>> thread        20  |     0.65    |   0.72   |   0.71     |   0.71
>> Process(Pipe) 10  |     0.18    |   0.31   |   0.31     |   0.33
>> Process(Pipe) 20  |     0.32    |   0.51   |   0.50     |   0.50
>> Process(Pipe) 30  |     0.43    |   0.65   |   0.67     |   0.67
>> Process(Pipe) 40  |     0.57    |   0.82   |   0.83     |   0.83
>> Process(Pipe) 50  |     0.67    |   1.00   |   0.97     |   0.98
>> Process(Pipe) 60  |     0.81    |   1.13   |   1.11     |   1.12
>> thread(Pipe)  10  |     0.19    |   0.33   |   0.33     |   0.33
>> thread(Pipe)  20  |     0.34    |   0.53   |   0.51     |   0.52
>>
>>
>>
>> type       groups |   v6.2      |v6.2+ V12+ | v6.2 + V12+| v6.2 + V12
>>                   |             |above patch|above patch | above patch
>>                   |             |lat nice=0 |lat nice=-20| lat nice=+19
>>
>> Process       10  |     0.36    |   0.33    |   0.34     |   0.34
>> Process       20  |     0.62    |   0.60    |   0.61     |   0.61
>> Process       30  |     0.87    |   0.84    |   0.85     |   0.84
>> Process       40  |     1.13    |   1.09    |   1.10     |   1.09
>> Process       50  |     1.38    |   1.33    |   1.33     |   1.34
>> Process       60  |     1.64    |   1.56    |   1.58     |   1.56
>> thread        10  |     0.35    |   0.35    |   0.35     |   0.35
>> thread        20  |     0.64    |   0.63    |   0.63     |   0.63
>> Process(Pipe) 10  |     0.20    |   0.18    |   0.18     |   0.18
>> Process(Pipe) 20  |     0.32    |   0.31    |   0.31     |   0.32
>> Process(Pipe) 30  |     0.44    |   0.43    |   0.43     |   0.43
>> Process(Pipe) 40  |     0.56    |   0.57    |   0.56     |   0.55
>> Process(Pipe) 50  |     0.70    |   0.67    |   0.67     |   0.67
>> Process(Pipe) 60  |     0.83    |   0.79    |   0.81     |   0.80
>> thread(Pipe)  10  |     0.21    |   0.19    |   0.19     |   0.19
>> thread(Pipe)  20  |     0.35    |   0.33    |   0.34     |   0.33
>>
>>
>> Do you want me to try any other experiment on this further?
> Yes, would be good to know which of the 3 changes in the patch create
> the regression
>
> I suspect the 1st change to be the root cause of your problem but It
> would be good if you can confirm my assumption with some tests
>
> Thanks

Applied each change individually. 3rd change seems to cause the regression.
Kept only the 3rd change and numbers are same as stable 6.2 for latency nice
value of 0.

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cdcd991bbcf1..c89c201dd164 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7827,7 +7827,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
         * for low priority task. Make sure that long sleeping task will get a
         * chance to preempt current.
         */
-       gran = min_t(s64, gran, get_latency_max());

        if (vdiff > gran)
                return 1;



>>>> I will try to run with only task's set with latency_nice=0 as well.
>>>>
>>>> type    groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
>>>>                |             |lat nice=0| lat nice=-20| lat nice=+19
>>>>
>>>> Process            10  |     0.33    |   0.37   |   0.38     |   0.37
>>>> Process       20  |  0.61    |   0.67   |   0.68     |   0.67
>>>> Process            30  |     0.85    |   0.95   |   0.95     |   0.96
>>>> Process            40  |     1.10    |   1.20   |   1.20     |   1.21
>>>> Process            50  |     1.34    |   1.47   |   1.44     |   1.45
>>>> Process            60  |     1.57    |   1.70   |   1.71     |   1.70
>>>> thread             10  |     0.36    |   0.40   |   0.39     |   0.39
>>>> thread             20  |     0.65    |   0.72   |   0.71     |   0.71
>>>> Process(Pipe) 10  |  0.18    |   0.31   |   0.31     |   0.33
>>>> Process(Pipe) 20  |  0.32    |   0.51   |   0.50     |   0.50
>>>> Process(Pipe) 30  |  0.43    |   0.65   |   0.67     |   0.67
>>>> Process(Pipe) 40  |  0.57    |   0.82   |   0.83     |   0.83
>>>> Process(Pipe) 50  |  0.67    |   1.00   |   0.97     |   0.98
>>>> Process(Pipe) 60  |  0.81    |   1.13   |   1.11     |   1.12
>>>> thread(Pipe)  10  |  0.19    |   0.33   |   0.33     |   0.33
>>>> thread(Pipe)  20  |  0.34    |   0.53   |   0.51     |   0.52
>>>>
>>>>
>>>>
>>>> type    groups |   v6.2      |v6.2 + V7 | v6.2 + V7  | v6.2 + V7
>>>>                |             |lat nice=0|lat nice=-20| lat nice=+19
>>>> Process            10  |     0.33    |   0.37   |   0.37     |   0.37
>>>> Process            20  |     0.61    |   0.67   |   0.67     |   0.67
>>>> Process            30  |     0.85    |   0.96   |   0.94     |   0.95
>>>> Process            40  |     1.10    |   1.20   |   1.20     |   1.20
>>>> Process            50  |     1.34    |   1.45   |   1.46     |   1.45
>>>> Process            60  |     1.57    |   1.71   |   1.68     |   1.72
>>>> thread             10  |     0.36    |   0.40   |   0.40     |   0.40
>>>> thread             20  |     0.65    |   0.71   |   0.71     |   0.71
>>>> Process(Pipe) 10  |  0.18    |   0.30   |   0.30     |   0.31
>>>> Process(Pipe) 20  |  0.32    |   0.50   |   0.50     |   0.50
>>>> Process(Pipe) 30  |  0.43    |   0.67   |   0.67     |   0.66
>>>> Process(Pipe) 40  |  0.57    |   0.86   |   0.84     |   0.84
>>>> Process(Pipe) 50  |  0.67    |   0.99   |   0.97     |   0.97
>>>> Process(Pipe) 60  |  0.81    |   1.10   |   1.13     |   1.13
>>>> thread(Pipe)  10  |  0.19    |   0.34   |   0.34     |   0.33
>>>> thread(Pipe)  20  |  0.34    |   0.55   |   0.53     |   0.54
>>>>
>>>>>>> Numbers are average of 10 runs in each case. Time is in seconds
>>>>>>>
>>>>>>> type       groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
>>>>>>>                   |            | lat nice=0    | lat nice=-20| lat nice=+19
>>>>>>>                   |            |               |             |
>>>>>>> Process       10  |   0.36     |     0.41      |    0.43     |    0.42
>>>>>>> Process       20  |   0.62     |     0.76      |    0.75     |    0.75
>>>>>>> Process       30  |   0.87     |     1.05      |    1.04     |    1.06
>>>>>>> Process       40  |   1.13     |     1.34      |    1.33     |    1.33
>>>>>>> Process       50  |   1.38     |     1.62      |    1.66     |    1.63
>>>>>>> Process       60  |   1.64     |     1.91      |    1.97     |    1.90
>>>>>>> thread        10  |   0.35     |     0.41      |    0.44     |    0.42
>>>>>>> thread        20  |   0.64     |     0.78      |    0.77     |    0.79
>>>>>>> Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
>>>>>>> Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
>>>>>>> Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
>>>>>>> Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
>>>>>>> Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
>>>>>>> Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
>>>>>>> thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
>>>>>>> thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>> Re-arrange seems to help the patch series by avoiding an cacheline miss.
>>>>>>>>>
>>>>>>>>> =========================
>>>>>>>>> schbench
>>>>>>>>> =========================
>>>>>>>>>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
>>>>>>>>> 1 Thread
>>>>>>>>>   50.0th:        9.00  |    9.00        |        9.50
>>>>>>>>>   75.0th:       10.50  |   10.00        |        9.50
>>>>>>>>>   90.0th:       11.00  |   11.00        |       10.50
>>>>>>>>>   95.0th:       11.00  |   11.00        |       11.00
>>>>>>>>>   99.0th:       11.50  |   11.50        |       11.50
>>>>>>>>>   99.5th:       12.50  |   12.00        |       12.00
>>>>>>>>>   99.9th:       14.50  |   13.50        |       12.00
>>>>>>>>> 2 Threads
>>>>>>>>>   50.0th:        9.50  |    9.50        |        8.50
>>>>>>>>>   75.0th:       11.00  |   10.50        |        9.50
>>>>>>>>>   90.0th:       13.50  |   11.50        |       10.50
>>>>>>>>>   95.0th:       14.00  |   12.00        |       11.00
>>>>>>>>>   99.0th:       15.50  |   13.50        |       12.00
>>>>>>>>>   99.5th:       16.00  |   14.00        |       12.00
>>>>>>>>>   99.9th:       17.00  |   16.00        |       16.50
>>>>>>>>> 4 Threads
>>>>>>>>>   50.0th:       11.50  |   11.50        |       10.50
>>>>>>>>>   75.0th:       13.50  |   12.50        |       12.50
>>>>>>>>>   90.0th:       15.50  |   14.50        |       14.00
>>>>>>>>>   95.0th:       16.50  |   15.50        |       14.50
>>>>>>>>>   99.0th:       20.00  |   17.50        |       16.50
>>>>>>>>>   99.5th:       20.50  |   18.50        |       17.00
>>>>>>>>>   99.9th:       22.50  |   21.00        |       19.00
>>>>>>>>> 8 Threads
>>>>>>>>>   50.0th:       14.00  |   14.00        |       14.00
>>>>>>>>>   75.0th:       16.00  |   16.00        |       16.00
>>>>>>>>>   90.0th:       18.00  |   18.00        |       17.50
>>>>>>>>>   95.0th:       18.50  |   18.50        |       18.50
>>>>>>>>>   99.0th:       20.00  |   20.00        |       20.00
>>>>>>>>>   99.5th:       20.50  |   21.50        |       21.00
>>>>>>>>>   99.9th:       22.50  |   23.50        |       23.00
>>>>>>>>> 16 Threads
>>>>>>>>>   50.0th:       19.00  |   18.50        |       19.00
>>>>>>>>>   75.0th:       23.00  |   22.50        |       23.00
>>>>>>>>>   90.0th:       25.00  |   25.50        |       25.00
>>>>>>>>>   95.0th:       26.50  |   26.50        |       26.00
>>>>>>>>>   99.0th:       28.50  |   29.00        |       28.50
>>>>>>>>>   99.5th:       31.00  |   30.00        |       30.00
>>>>>>>>>   99.9th:     5626.00  | 4761.50        |       32.50
>>>>>>>>> 32 Threads
>>>>>>>>>   50.0th:       27.00  |   27.50        |       29.00
>>>>>>>>>   75.0th:       35.50  |   36.50        |       38.50
>>>>>>>>>   90.0th:       42.00  |   44.00        |       50.50
>>>>>>>>>   95.0th:      447.50  | 2959.00        |     8544.00
>>>>>>>>>   99.0th:     7372.00  | 17032.00       |    19136.00
>>>>>>>>>   99.5th:    15360.00  | 19808.00       |    20704.00
>>>>>>>>>   99.9th:    20640.00  | 30048.00       |    30048.00
>>>>>>>>>
>>>>>>>>> ====================
>>>>>>>>> hackbench
>>>>>>>>> ====================
>>>>>>>>>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
>>>>>>>>>
>>>>>>>>> Process 10 Time:        0.35    |       0.42        |           0.41
>>>>>>>>> Process 20 Time:        0.61    |       0.76        |           0.76
>>>>>>>>> Process 30 Time:        0.87    |       1.06        |           1.05
>>>>>>>>> thread 10 Time:         0.35    |       0.43        |           0.42
>>>>>>>>> thread 20 Time:         0.66    |       0.79        |           0.78
>>>>>>>>> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
>>>>>>>>> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
>>>>>>>>> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
>>>>>>>>> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
>>>>>>>>> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>       struct list_head                group_node;
>>>>>>>>>>       unsigned int                    on_rq;
>>>>>>>>>>
>>>>>>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>>>>>>>> index 093cc1af73dc..752fd364216c 100644
>>>>>>>>>> --- a/kernel/sched/core.c
>>>>>>>>>> +++ b/kernel/sched/core.c
>>>>>>>>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
>>>>>>>>>>       p->se.nr_migrations             = 0;
>>>>>>>>>>       p->se.vruntime                  = 0;
>>>>>>>>>>       INIT_LIST_HEAD(&p->se.group_node);
>>>>>>>>>> +     RB_CLEAR_NODE(&p->se.latency_node);
>>>>>>>>>>
>>>>>>>>>>  #ifdef CONFIG_FAIR_GROUP_SCHED
>>>>>>>>>>       p->se.cfs_rq                    = NULL;
>>>>>>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>>>>>>>>> index 125a6ff53378..e2aeb4511686 100644
>>>>>>>>>> --- a/kernel/sched/fair.c
>>>>>>>>>> +++ b/kernel/sched/fair.c
>>>>>>>>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
>>>>>>>>>>
>>>>>>>>>>       return __node_2_se(last);
>>>>>>>>>>  }
>>>>>>>>>> +#endif
>>>>>>>>>>
>>>>>>>>>> +/**************************************************************
>>>>>>>>>> + * Scheduling class tree data structure manipulation methods:
>>>>>>>>>> + * for latency
>>>>>>>>>> + */
>>>>>>>>>> +
>>>>>>>>>> +static inline bool latency_before(struct sched_entity *a,
>>>>>>>>>> +                             struct sched_entity *b)
>>>>>>>>>> +{
>>>>>>>>>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +#define __latency_node_2_se(node) \
>>>>>>>>>> +     rb_entry((node), struct sched_entity, latency_node)
>>>>>>>>>> +
>>>>>>>>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
>>>>>>>>>> +{
>>>>>>>>>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +/*
>>>>>>>>>> + * Enqueue an entity into the latency rb-tree:
>>>>>>>>>> + */
>>>>>>>>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>>>>> +{
>>>>>>>>>> +
>>>>>>>>>> +     /* Only latency sensitive entity can be added to the list */
>>>>>>>>>> +     if (se->latency_offset >= 0)
>>>>>>>>>> +             return;
>>>>>>>>>> +
>>>>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node))
>>>>>>>>>> +             return;
>>>>>>>>>> +
>>>>>>>>>> +     /*
>>>>>>>>>> +      * The entity is always added the latency list at wakeup.
>>>>>>>>>> +      * Then, a not waking up entity that is put back in the list after an
>>>>>>>>>> +      * execution time less than sysctl_sched_min_granularity, means that
>>>>>>>>>> +      * the entity has been preempted by a higher sched class or an entity
>>>>>>>>>> +      * with higher latency constraint. In thi case, the entity is also put
>>>>>>>>>> +      * back in the latency list so it gets a chance to run 1st during the
>>>>>>>>>> +      * next slice.
>>>>>>>>>> +      */
>>>>>>>>>> +     if (!(flags & ENQUEUE_WAKEUP)) {
>>>>>>>>>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
>>>>>>>>>> +
>>>>>>>>>> +             if (delta_exec >= sysctl_sched_min_granularity)
>>>>>>>>>> +                     return;
>>>>>>>>>> +     }
>>>>>>>>>> +
>>>>>>>>>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +/*
>>>>>>>>>> + * Dequeue an entity from the latency rb-tree and return true if it was really
>>>>>>>>>> + * part of the rb-tree:
>>>>>>>>>> + */
>>>>>>>>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>>>>>> +{
>>>>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
>>>>>>>>>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
>>>>>>>>>> +             RB_CLEAR_NODE(&se->latency_node);
>>>>>>>>>> +             return true;
>>>>>>>>>> +     }
>>>>>>>>>> +
>>>>>>>>>> +     return false;
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
>>>>>>>>>> +{
>>>>>>>>>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
>>>>>>>>>> +
>>>>>>>>>> +     if (!left)
>>>>>>>>>> +             return NULL;
>>>>>>>>>> +
>>>>>>>>>> +     return __latency_node_2_se(left);
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +#ifdef CONFIG_SCHED_DEBUG
>>>>>>>>>>  /**************************************************************
>>>>>>>>>>   * Scheduling class statistics methods:
>>>>>>>>>>   */
>>>>>>>>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>>>>>       check_schedstat_required();
>>>>>>>>>>       update_stats_enqueue_fair(cfs_rq, se, flags);
>>>>>>>>>>       check_spread(cfs_rq, se);
>>>>>>>>>> -     if (!curr)
>>>>>>>>>> +     if (!curr) {
>>>>>>>>>>               __enqueue_entity(cfs_rq, se);
>>>>>>>>>> +             __enqueue_latency(cfs_rq, se, flags);
>>>>>>>>>> +     }
>>>>>>>>>>       se->on_rq = 1;
>>>>>>>>>>
>>>>>>>>>>       if (cfs_rq->nr_running == 1) {
>>>>>>>>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>>>>>
>>>>>>>>>>       clear_buddies(cfs_rq, se);
>>>>>>>>>>
>>>>>>>>>> -     if (se != cfs_rq->curr)
>>>>>>>>>> +     if (se != cfs_rq->curr) {
>>>>>>>>>>               __dequeue_entity(cfs_rq, se);
>>>>>>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>>>>>>> +     }
>>>>>>>>>>       se->on_rq = 0;
>>>>>>>>>>       account_entity_dequeue(cfs_rq, se);
>>>>>>>>>>
>>>>>>>>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>>>>>>                */
>>>>>>>>>>               update_stats_wait_end_fair(cfs_rq, se);
>>>>>>>>>>               __dequeue_entity(cfs_rq, se);
>>>>>>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>>>>>>>               update_load_avg(cfs_rq, se, UPDATE_TG);
>>>>>>>>>>       }
>>>>>>>>>>
>>>>>>>>>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
>>>>>>>>>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>>>>>>  {
>>>>>>>>>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
>>>>>>>>>> -     struct sched_entity *se;
>>>>>>>>>> +     struct sched_entity *latency, *se;
>>>>>>>>>>
>>>>>>>>>>       /*
>>>>>>>>>>        * If curr is set we have to see if its left of the leftmost entity
>>>>>>>>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>>>>>>               se = cfs_rq->last;
>>>>>>>>>>       }
>>>>>>>>>>
>>>>>>>>>> +     /* Check for latency sensitive entity waiting for running */
>>>>>>>>>> +     latency = __pick_first_latency(cfs_rq);
>>>>>>>>>> +     if (latency && (latency != se) &&
>>>>>>>>>> +         wakeup_preempt_entity(latency, se) < 1)
>>>>>>>>>> +             se = latency;
>>>>>>>>>> +
>>>>>>>>>>       return se;
>>>>>>>>>>  }
>>>>>>>>>>
>>>>>>>>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
>>>>>>>>>>               update_stats_wait_start_fair(cfs_rq, prev);
>>>>>>>>>>               /* Put 'current' back into the tree. */
>>>>>>>>>>               __enqueue_entity(cfs_rq, prev);
>>>>>>>>>> +             __enqueue_latency(cfs_rq, prev, 0);
>>>>>>>>>>               /* in !on_rq case, update occurred at dequeue */
>>>>>>>>>>               update_load_avg(cfs_rq, prev, 0);
>>>>>>>>>>       }
>>>>>>>>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>>>>>>>>>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
>>>>>>>>>>  {
>>>>>>>>>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
>>>>>>>>>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
>>>>>>>>>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
>>>>>>>>>>  #ifdef CONFIG_SMP
>>>>>>>>>>       raw_spin_lock_init(&cfs_rq->removed.lock);
>>>>>>>>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
>>>>>>>>>>       se->my_q = cfs_rq;
>>>>>>>>>>
>>>>>>>>>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
>>>>>>>>>> +     RB_CLEAR_NODE(&se->latency_node);
>>>>>>>>>>
>>>>>>>>>>       /* guarantee group entities always have weight */
>>>>>>>>>>       update_load_set(&se->load, NICE_0_LOAD);
>>>>>>>>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
>>>>>>>>>>
>>>>>>>>>>       for_each_possible_cpu(i) {
>>>>>>>>>>               struct sched_entity *se = tg->se[i];
>>>>>>>>>> +             struct rq *rq = cpu_rq(i);
>>>>>>>>>> +             struct rq_flags rf;
>>>>>>>>>> +             bool queued;
>>>>>>>>>> +
>>>>>>>>>> +             rq_lock_irqsave(rq, &rf);
>>>>>>>>>>
>>>>>>>>>> +             queued = __dequeue_latency(se->cfs_rq, se);
>>>>>>>>>>               WRITE_ONCE(se->latency_offset, latency_offset);
>>>>>>>>>> +             if (queued)
>>>>>>>>>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
>>>>>>>>>> +
>>>>>>>>>> +
>>>>>>>>>> +             rq_unlock_irqrestore(rq, &rf);
>>>>>>>>>>       }
>>>>>>>>>>
>>>>>>>>>>       mutex_unlock(&shares_mutex);
>>>>>>>>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
>>>>>>>>>> index 9a2e71231083..21dd309e98a9 100644
>>>>>>>>>> --- a/kernel/sched/sched.h
>>>>>>>>>> +++ b/kernel/sched/sched.h
>>>>>>>>>> @@ -570,6 +570,7 @@ struct cfs_rq {
>>>>>>>>>>  #endif
>>>>>>>>>>
>>>>>>>>>>       struct rb_root_cached   tasks_timeline;
>>>>>>>>>> +     struct rb_root_cached   latency_timeline;
>>>>>>>>>>
>>>>>>>>>>       /*
>>>>>>>>>>        * 'curr' points to currently running entity on this cfs_rq.

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Vincent Guittot]

Le lundi 06 mars 2023 à 17:03:30 (+0530), Shrikanth Hegde a écrit :
> 
> 
> On 3/5/23 6:33 PM, Vincent Guittot wrote:
> > On Sat, 4 Mar 2023 at 16:13, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>
> >>
> >> On 3/3/23 10:01 PM, Vincent Guittot wrote:
> >>> Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
> >>>> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
> >>>>> On 3/2/23 6:47 PM, Vincent Guittot wrote:
> >>>>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
> >>>>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
> >>> [...]
> >>>
> >>>>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
> >>>>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
> >>>>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
> >>>>>>>>> and other is running these benchmarks (group2). Set the latency nice
> >>>>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
> >>>>>>>>> Total of 96 CPU.
> >>>>>>>>>
> >>>>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
> >>>>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
> >>>>>>>>> and attach the numbers to series.
> >>>>>>>>> Please suggest if any variation in the test i need to try.
> >>>>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
> >>>>>>>> run longer slice to move forward rather than preempting others all the
> >>>>>>>> time
> >>>>>>> hackbench still seems to regress in different latency nice values compared to
> >>>>>>> baseline of 6.2 in this case. up to 50% in some cases.
> >>>>>>>
> >>>>>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
> >>>>>>> running 2 CPU cgroups. No quota assigned.
> >>>>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
> >>>>>>> latency is not changed for this cgroup.
> >>>>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
> >>>>>>> nice values of 0, -20 and 19.
> >>>>>> According to your other emails, you are using the cgroup interface and
> >>>>>> not the task's one. Do I get it right ?
> >>>>> right. I create cgroup, attach bash command with echo $$,
> >>>>> assign the latency nice to cgroup, and run hackbench from that bash prompt.
> >>>>>
> >>>>>> I haven't run test such tests in a cgroup but at least the test with
> >>>>>> latency_nice == 0 should not make any noticeable difference. Does this
> >>>>>> include the re-arrange patch that you have proposed previously ?
> >>>>> No. This is only with V12 of the series.
> >>>>>
> >>>>>> Also, the tests that you did on v6, gave better result.
> >>>>>> https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@linux.vnet.ibm.com/
> >>>>>>
> >>>>>> Are you running same tests or you changed something in the mean time ?
> >>>>> Test machine got changed.
> >>>>> now i re-read my earlier mail. I see it was slightly different.
> >>>>> I had created only one cgroup and stress-ng was run
> >>>>> without any cgroup. Let me try that scenario and get the numbers.
> >>>> Tried the same method of testing i had done on V7 of the series. on this
> >>>> machine hackbench still regress's both on V12 as well as V7 of the series.
> >>>>
> >>>> Created one cpu cgroup called cgroup1. created two bash prompts.
> >>>> assigned "bash $$" to cgroup1 and on other bash prompt running,
> >>>> stress-ng --cpu=96 -l 50. Ran hackbench from cgroup1 prompt.
> >>>> assigned latency values to the cgroup1.
> >>> I have tried to reproduce your results on some of my systems but I can't see
> >>> the impacts that you are reporting below.
> >>> The fact that your other platform was not impacted as well could imply that
> >>> it's specific to this platform.
> >>> In particular, the lat nice=0 case should not show any real impact as it
> >>> should be similar to a nop. At least that what I can see in the tests on my
> >>> platforms and Prateek on his.
> >>>
> >>> Nevertheless, could you try to run your tests with the changes below ?
> >>> These are the only places which could have an impact even with lat nice = 0
> >>>
> >>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >>> index 8137bca80572..979571a98b28 100644
> >>> --- a/kernel/sched/fair.c
> >>> +++ b/kernel/sched/fair.c
> >>> @@ -4991,8 +4991,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>         if (delta < offset)
> >>>                 return;
> >>>
> >>> -       if ((delta > ideal_runtime) ||
> >>> -           (delta > get_latency_max()))
> >>> +       if (delta > ideal_runtime)
> >>>                 resched_curr(rq_of(cfs_rq));
> >>>  }
> >>>
> >>> @@ -7574,9 +7573,10 @@ static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *
> >>>          * Otherwise, use the latency weight to evaluate how much scheduling
> >>>          * delay is acceptable by se.
> >>>          */
> >>> -       if ((latency_offset < 0) || (curr->latency_offset < 0))
> >>> +       if ((latency_offset < 0) || (curr->latency_offset < 0)) {
> >>>                 latency_offset -= curr->latency_offset;
> >>> -       latency_offset = min_t(long, latency_offset, get_latency_max());
> >>> +               latency_offset = min_t(long, latency_offset, get_latency_max());
> >>> +       }
> >>>
> >>>         return latency_offset;
> >>>  }
> >>> @@ -7635,7 +7635,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >>>          * for low priority task. Make sure that long sleeping task will get a
> >>>          * chance to preempt current.
> >>>          */
> >>> -       gran = min_t(s64, gran, get_latency_max());
> >>>
> >>>         if (vdiff > gran)
> >>>                 return 1;
> >>>
> >> Above patch helps. thank you.
> > Great. At least we have narrow the problem to one f the 3 changes.
> >
> >> Numbers are comparable to 6.2 and there is slight improvement. Much better than V12 numbers.
> >>
> >> type       groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
> >>                   |             |lat nice=0| lat nice=-20| lat nice=+19
> >>
> >> Process       10  |     0.33    |   0.37   |   0.38     |   0.37
> >> Process       20  |     0.61    |   0.67   |   0.68     |   0.67
> >> Process       30  |     0.85    |   0.95   |   0.95     |   0.96
> >> Process       40  |     1.10    |   1.20   |   1.20     |   1.21
> >> Process       50  |     1.34    |   1.47   |   1.44     |   1.45
> >> Process       60  |     1.57    |   1.70   |   1.71     |   1.70
> >> thread        10  |     0.36    |   0.40   |   0.39     |   0.39
> >> thread        20  |     0.65    |   0.72   |   0.71     |   0.71
> >> Process(Pipe) 10  |     0.18    |   0.31   |   0.31     |   0.33
> >> Process(Pipe) 20  |     0.32    |   0.51   |   0.50     |   0.50
> >> Process(Pipe) 30  |     0.43    |   0.65   |   0.67     |   0.67
> >> Process(Pipe) 40  |     0.57    |   0.82   |   0.83     |   0.83
> >> Process(Pipe) 50  |     0.67    |   1.00   |   0.97     |   0.98
> >> Process(Pipe) 60  |     0.81    |   1.13   |   1.11     |   1.12
> >> thread(Pipe)  10  |     0.19    |   0.33   |   0.33     |   0.33
> >> thread(Pipe)  20  |     0.34    |   0.53   |   0.51     |   0.52
> >>
> >>
> >>
> >> type       groups |   v6.2      |v6.2+ V12+ | v6.2 + V12+| v6.2 + V12
> >>                   |             |above patch|above patch | above patch
> >>                   |             |lat nice=0 |lat nice=-20| lat nice=+19
> >>
> >> Process       10  |     0.36    |   0.33    |   0.34     |   0.34
> >> Process       20  |     0.62    |   0.60    |   0.61     |   0.61
> >> Process       30  |     0.87    |   0.84    |   0.85     |   0.84
> >> Process       40  |     1.13    |   1.09    |   1.10     |   1.09
> >> Process       50  |     1.38    |   1.33    |   1.33     |   1.34
> >> Process       60  |     1.64    |   1.56    |   1.58     |   1.56
> >> thread        10  |     0.35    |   0.35    |   0.35     |   0.35
> >> thread        20  |     0.64    |   0.63    |   0.63     |   0.63
> >> Process(Pipe) 10  |     0.20    |   0.18    |   0.18     |   0.18
> >> Process(Pipe) 20  |     0.32    |   0.31    |   0.31     |   0.32
> >> Process(Pipe) 30  |     0.44    |   0.43    |   0.43     |   0.43
> >> Process(Pipe) 40  |     0.56    |   0.57    |   0.56     |   0.55
> >> Process(Pipe) 50  |     0.70    |   0.67    |   0.67     |   0.67
> >> Process(Pipe) 60  |     0.83    |   0.79    |   0.81     |   0.80
> >> thread(Pipe)  10  |     0.21    |   0.19    |   0.19     |   0.19
> >> thread(Pipe)  20  |     0.35    |   0.33    |   0.34     |   0.33
> >>
> >>
> >> Do you want me to try any other experiment on this further?
> > Yes, would be good to know which of the 3 changes in the patch create
> > the regression
> >
> > I suspect the 1st change to be the root cause of your problem but It
> > would be good if you can confirm my assumption with some tests
> >
> > Thanks
> 
> Applied each change individually. 3rd change seems to cause the regression.
> Kept only the 3rd change and numbers are same as stable 6.2 for latency nice
> value of 0.

Ok, it's the patch 1 that aims to prevent some unfairness with low weight
waking task. And your platform probably falls in the last part of the commit:

" Strictly speaking, we should use cfs->min_vruntime instead of
curr->vruntime but it doesn't worth the additional overhead and complexity
as the vruntime of current should be close to min_vruntime if not equal."

Could you try the patch below on top of v12 ?

---
 kernel/sched/fair.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e2aeb4511686..77b03a280912 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5049,7 +5049,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 }

 static int
-wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se);
+wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq);

 /*
  * Pick the next process, keeping these things in mind, in this order:
@@ -5088,16 +5088,16 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 				second = curr;
 		}

-		if (second && wakeup_preempt_entity(second, left) < 1)
+		if (second && wakeup_preempt_entity(second, left, cfs_rq) < 1)
 			se = second;
 	}

-	if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left) < 1) {
+	if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left, cfs_rq) < 1) {
 		/*
 		 * Someone really wants this to run. If it's not unfair, run it.
 		 */
 		se = cfs_rq->next;
-	} else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left) < 1) {
+	} else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left, cfs_rq) < 1) {
 		/*
 		 * Prefer last buddy, try to return the CPU to a preempted task.
 		 */
@@ -5107,7 +5107,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 	/* Check for latency sensitive entity waiting for running */
 	latency = __pick_first_latency(cfs_rq);
 	if (latency && (latency != se) &&
-	    wakeup_preempt_entity(latency, se) < 1)
+	    wakeup_preempt_entity(latency, se, cfs_rq) < 1)
 		se = latency;

 	return se;
@@ -7808,7 +7808,7 @@ static unsigned long wakeup_gran(struct sched_entity *se)
  *
  */
 static int
-wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
+wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq)
 {
 	s64 gran, vdiff = curr->vruntime - se->vruntime;
 	s64 offset = wakeup_latency_gran(curr, se);
@@ -7818,6 +7818,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)

 	gran = offset + wakeup_gran(se);

+	if (vdiff > gran)
+		return 1;
 	/*
 	 * At wake up, the vruntime of a task is capped to not be older than
 	 * a sched_latency period compared to min_vruntime. This prevents long
@@ -7827,9 +7829,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
 	 * for low priority task. Make sure that long sleeping task will get a
 	 * chance to preempt current.
 	 */
-	gran = min_t(s64, gran, get_latency_max());
-
-	if (vdiff > gran)
+	vdiff = cfs_rq->min_vruntime - se->vruntime;
+	if (vdiff > get_latency_max())
 		return 1;

 	return 0;
@@ -7933,7 +7934,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
 		return;

 	update_curr(cfs_rq_of(se));
-	if (wakeup_preempt_entity(se, pse) == 1) {
+	if (wakeup_preempt_entity(se, pse, cfs_rq_of(se)) == 1) {
 		/*
 		 * Bias pick_next to pick the sched entity that is
 		 * triggering this preemption.
--
2.34.1

> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index cdcd991bbcf1..c89c201dd164 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -7827,7 +7827,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>          * for low priority task. Make sure that long sleeping task will get a
>          * chance to preempt current.
>          */
> -       gran = min_t(s64, gran, get_latency_max());
> 
>         if (vdiff > gran)
>                 return 1;
> 
> 
> 
> >>>> I will try to run with only task's set with latency_nice=0 as well.
> >>>>
> >>>> type    groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
> >>>>                |             |lat nice=0| lat nice=-20| lat nice=+19
> >>>>
> >>>> Process            10  |     0.33    |   0.37   |   0.38     |   0.37
> >>>> Process       20  |  0.61    |   0.67   |   0.68     |   0.67
> >>>> Process            30  |     0.85    |   0.95   |   0.95     |   0.96
> >>>> Process            40  |     1.10    |   1.20   |   1.20     |   1.21
> >>>> Process            50  |     1.34    |   1.47   |   1.44     |   1.45
> >>>> Process            60  |     1.57    |   1.70   |   1.71     |   1.70
> >>>> thread             10  |     0.36    |   0.40   |   0.39     |   0.39
> >>>> thread             20  |     0.65    |   0.72   |   0.71     |   0.71
> >>>> Process(Pipe) 10  |  0.18    |   0.31   |   0.31     |   0.33
> >>>> Process(Pipe) 20  |  0.32    |   0.51   |   0.50     |   0.50
> >>>> Process(Pipe) 30  |  0.43    |   0.65   |   0.67     |   0.67
> >>>> Process(Pipe) 40  |  0.57    |   0.82   |   0.83     |   0.83
> >>>> Process(Pipe) 50  |  0.67    |   1.00   |   0.97     |   0.98
> >>>> Process(Pipe) 60  |  0.81    |   1.13   |   1.11     |   1.12
> >>>> thread(Pipe)  10  |  0.19    |   0.33   |   0.33     |   0.33
> >>>> thread(Pipe)  20  |  0.34    |   0.53   |   0.51     |   0.52
> >>>>
> >>>>
> >>>>
> >>>> type    groups |   v6.2      |v6.2 + V7 | v6.2 + V7  | v6.2 + V7
> >>>>                |             |lat nice=0|lat nice=-20| lat nice=+19
> >>>> Process            10  |     0.33    |   0.37   |   0.37     |   0.37
> >>>> Process            20  |     0.61    |   0.67   |   0.67     |   0.67
> >>>> Process            30  |     0.85    |   0.96   |   0.94     |   0.95
> >>>> Process            40  |     1.10    |   1.20   |   1.20     |   1.20
> >>>> Process            50  |     1.34    |   1.45   |   1.46     |   1.45
> >>>> Process            60  |     1.57    |   1.71   |   1.68     |   1.72
> >>>> thread             10  |     0.36    |   0.40   |   0.40     |   0.40
> >>>> thread             20  |     0.65    |   0.71   |   0.71     |   0.71
> >>>> Process(Pipe) 10  |  0.18    |   0.30   |   0.30     |   0.31
> >>>> Process(Pipe) 20  |  0.32    |   0.50   |   0.50     |   0.50
> >>>> Process(Pipe) 30  |  0.43    |   0.67   |   0.67     |   0.66
> >>>> Process(Pipe) 40  |  0.57    |   0.86   |   0.84     |   0.84
> >>>> Process(Pipe) 50  |  0.67    |   0.99   |   0.97     |   0.97
> >>>> Process(Pipe) 60  |  0.81    |   1.10   |   1.13     |   1.13
> >>>> thread(Pipe)  10  |  0.19    |   0.34   |   0.34     |   0.33
> >>>> thread(Pipe)  20  |  0.34    |   0.55   |   0.53     |   0.54
> >>>>
> >>>>>>> Numbers are average of 10 runs in each case. Time is in seconds
> >>>>>>>
> >>>>>>> type       groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
> >>>>>>>                   |            | lat nice=0    | lat nice=-20| lat nice=+19
> >>>>>>>                   |            |               |             |
> >>>>>>> Process       10  |   0.36     |     0.41      |    0.43     |    0.42
> >>>>>>> Process       20  |   0.62     |     0.76      |    0.75     |    0.75
> >>>>>>> Process       30  |   0.87     |     1.05      |    1.04     |    1.06
> >>>>>>> Process       40  |   1.13     |     1.34      |    1.33     |    1.33
> >>>>>>> Process       50  |   1.38     |     1.62      |    1.66     |    1.63
> >>>>>>> Process       60  |   1.64     |     1.91      |    1.97     |    1.90
> >>>>>>> thread        10  |   0.35     |     0.41      |    0.44     |    0.42
> >>>>>>> thread        20  |   0.64     |     0.78      |    0.77     |    0.79
> >>>>>>> Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
> >>>>>>> Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
> >>>>>>> Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
> >>>>>>> Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
> >>>>>>> Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
> >>>>>>> Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
> >>>>>>> thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
> >>>>>>> thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>> Re-arrange seems to help the patch series by avoiding an cacheline miss.
> >>>>>>>>>
> >>>>>>>>> =========================
> >>>>>>>>> schbench
> >>>>>>>>> =========================
> >>>>>>>>>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
> >>>>>>>>> 1 Thread
> >>>>>>>>>   50.0th:        9.00  |    9.00        |        9.50
> >>>>>>>>>   75.0th:       10.50  |   10.00        |        9.50
> >>>>>>>>>   90.0th:       11.00  |   11.00        |       10.50
> >>>>>>>>>   95.0th:       11.00  |   11.00        |       11.00
> >>>>>>>>>   99.0th:       11.50  |   11.50        |       11.50
> >>>>>>>>>   99.5th:       12.50  |   12.00        |       12.00
> >>>>>>>>>   99.9th:       14.50  |   13.50        |       12.00
> >>>>>>>>> 2 Threads
> >>>>>>>>>   50.0th:        9.50  |    9.50        |        8.50
> >>>>>>>>>   75.0th:       11.00  |   10.50        |        9.50
> >>>>>>>>>   90.0th:       13.50  |   11.50        |       10.50
> >>>>>>>>>   95.0th:       14.00  |   12.00        |       11.00
> >>>>>>>>>   99.0th:       15.50  |   13.50        |       12.00
> >>>>>>>>>   99.5th:       16.00  |   14.00        |       12.00
> >>>>>>>>>   99.9th:       17.00  |   16.00        |       16.50
> >>>>>>>>> 4 Threads
> >>>>>>>>>   50.0th:       11.50  |   11.50        |       10.50
> >>>>>>>>>   75.0th:       13.50  |   12.50        |       12.50
> >>>>>>>>>   90.0th:       15.50  |   14.50        |       14.00
> >>>>>>>>>   95.0th:       16.50  |   15.50        |       14.50
> >>>>>>>>>   99.0th:       20.00  |   17.50        |       16.50
> >>>>>>>>>   99.5th:       20.50  |   18.50        |       17.00
> >>>>>>>>>   99.9th:       22.50  |   21.00        |       19.00
> >>>>>>>>> 8 Threads
> >>>>>>>>>   50.0th:       14.00  |   14.00        |       14.00
> >>>>>>>>>   75.0th:       16.00  |   16.00        |       16.00
> >>>>>>>>>   90.0th:       18.00  |   18.00        |       17.50
> >>>>>>>>>   95.0th:       18.50  |   18.50        |       18.50
> >>>>>>>>>   99.0th:       20.00  |   20.00        |       20.00
> >>>>>>>>>   99.5th:       20.50  |   21.50        |       21.00
> >>>>>>>>>   99.9th:       22.50  |   23.50        |       23.00
> >>>>>>>>> 16 Threads
> >>>>>>>>>   50.0th:       19.00  |   18.50        |       19.00
> >>>>>>>>>   75.0th:       23.00  |   22.50        |       23.00
> >>>>>>>>>   90.0th:       25.00  |   25.50        |       25.00
> >>>>>>>>>   95.0th:       26.50  |   26.50        |       26.00
> >>>>>>>>>   99.0th:       28.50  |   29.00        |       28.50
> >>>>>>>>>   99.5th:       31.00  |   30.00        |       30.00
> >>>>>>>>>   99.9th:     5626.00  | 4761.50        |       32.50
> >>>>>>>>> 32 Threads
> >>>>>>>>>   50.0th:       27.00  |   27.50        |       29.00
> >>>>>>>>>   75.0th:       35.50  |   36.50        |       38.50
> >>>>>>>>>   90.0th:       42.00  |   44.00        |       50.50
> >>>>>>>>>   95.0th:      447.50  | 2959.00        |     8544.00
> >>>>>>>>>   99.0th:     7372.00  | 17032.00       |    19136.00
> >>>>>>>>>   99.5th:    15360.00  | 19808.00       |    20704.00
> >>>>>>>>>   99.9th:    20640.00  | 30048.00       |    30048.00
> >>>>>>>>>
> >>>>>>>>> ====================
> >>>>>>>>> hackbench
> >>>>>>>>> ====================
> >>>>>>>>>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
> >>>>>>>>>
> >>>>>>>>> Process 10 Time:        0.35    |       0.42        |           0.41
> >>>>>>>>> Process 20 Time:        0.61    |       0.76        |           0.76
> >>>>>>>>> Process 30 Time:        0.87    |       1.06        |           1.05
> >>>>>>>>> thread 10 Time:         0.35    |       0.43        |           0.42
> >>>>>>>>> thread 20 Time:         0.66    |       0.79        |           0.78
> >>>>>>>>> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
> >>>>>>>>> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
> >>>>>>>>> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
> >>>>>>>>> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
> >>>>>>>>> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>       struct list_head                group_node;
> >>>>>>>>>>       unsigned int                    on_rq;
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> >>>>>>>>>> index 093cc1af73dc..752fd364216c 100644
> >>>>>>>>>> --- a/kernel/sched/core.c
> >>>>>>>>>> +++ b/kernel/sched/core.c
> >>>>>>>>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
> >>>>>>>>>>       p->se.nr_migrations             = 0;
> >>>>>>>>>>       p->se.vruntime                  = 0;
> >>>>>>>>>>       INIT_LIST_HEAD(&p->se.group_node);
> >>>>>>>>>> +     RB_CLEAR_NODE(&p->se.latency_node);
> >>>>>>>>>>
> >>>>>>>>>>  #ifdef CONFIG_FAIR_GROUP_SCHED
> >>>>>>>>>>       p->se.cfs_rq                    = NULL;
> >>>>>>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >>>>>>>>>> index 125a6ff53378..e2aeb4511686 100644
> >>>>>>>>>> --- a/kernel/sched/fair.c
> >>>>>>>>>> +++ b/kernel/sched/fair.c
> >>>>>>>>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
> >>>>>>>>>>
> >>>>>>>>>>       return __node_2_se(last);
> >>>>>>>>>>  }
> >>>>>>>>>> +#endif
> >>>>>>>>>>
> >>>>>>>>>> +/**************************************************************
> >>>>>>>>>> + * Scheduling class tree data structure manipulation methods:
> >>>>>>>>>> + * for latency
> >>>>>>>>>> + */
> >>>>>>>>>> +
> >>>>>>>>>> +static inline bool latency_before(struct sched_entity *a,
> >>>>>>>>>> +                             struct sched_entity *b)
> >>>>>>>>>> +{
> >>>>>>>>>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
> >>>>>>>>>> +}
> >>>>>>>>>> +
> >>>>>>>>>> +#define __latency_node_2_se(node) \
> >>>>>>>>>> +     rb_entry((node), struct sched_entity, latency_node)
> >>>>>>>>>> +
> >>>>>>>>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
> >>>>>>>>>> +{
> >>>>>>>>>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
> >>>>>>>>>> +}
> >>>>>>>>>> +
> >>>>>>>>>> +/*
> >>>>>>>>>> + * Enqueue an entity into the latency rb-tree:
> >>>>>>>>>> + */
> >>>>>>>>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>>>>>>>> +{
> >>>>>>>>>> +
> >>>>>>>>>> +     /* Only latency sensitive entity can be added to the list */
> >>>>>>>>>> +     if (se->latency_offset >= 0)
> >>>>>>>>>> +             return;
> >>>>>>>>>> +
> >>>>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node))
> >>>>>>>>>> +             return;
> >>>>>>>>>> +
> >>>>>>>>>> +     /*
> >>>>>>>>>> +      * The entity is always added the latency list at wakeup.
> >>>>>>>>>> +      * Then, a not waking up entity that is put back in the list after an
> >>>>>>>>>> +      * execution time less than sysctl_sched_min_granularity, means that
> >>>>>>>>>> +      * the entity has been preempted by a higher sched class or an entity
> >>>>>>>>>> +      * with higher latency constraint. In thi case, the entity is also put
> >>>>>>>>>> +      * back in the latency list so it gets a chance to run 1st during the
> >>>>>>>>>> +      * next slice.
> >>>>>>>>>> +      */
> >>>>>>>>>> +     if (!(flags & ENQUEUE_WAKEUP)) {
> >>>>>>>>>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
> >>>>>>>>>> +
> >>>>>>>>>> +             if (delta_exec >= sysctl_sched_min_granularity)
> >>>>>>>>>> +                     return;
> >>>>>>>>>> +     }
> >>>>>>>>>> +
> >>>>>>>>>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
> >>>>>>>>>> +}
> >>>>>>>>>> +
> >>>>>>>>>> +/*
> >>>>>>>>>> + * Dequeue an entity from the latency rb-tree and return true if it was really
> >>>>>>>>>> + * part of the rb-tree:
> >>>>>>>>>> + */
> >>>>>>>>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >>>>>>>>>> +{
> >>>>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
> >>>>>>>>>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
> >>>>>>>>>> +             RB_CLEAR_NODE(&se->latency_node);
> >>>>>>>>>> +             return true;
> >>>>>>>>>> +     }
> >>>>>>>>>> +
> >>>>>>>>>> +     return false;
> >>>>>>>>>> +}
> >>>>>>>>>> +
> >>>>>>>>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
> >>>>>>>>>> +{
> >>>>>>>>>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
> >>>>>>>>>> +
> >>>>>>>>>> +     if (!left)
> >>>>>>>>>> +             return NULL;
> >>>>>>>>>> +
> >>>>>>>>>> +     return __latency_node_2_se(left);
> >>>>>>>>>> +}
> >>>>>>>>>> +
> >>>>>>>>>> +#ifdef CONFIG_SCHED_DEBUG
> >>>>>>>>>>  /**************************************************************
> >>>>>>>>>>   * Scheduling class statistics methods:
> >>>>>>>>>>   */
> >>>>>>>>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>>>>>>>>       check_schedstat_required();
> >>>>>>>>>>       update_stats_enqueue_fair(cfs_rq, se, flags);
> >>>>>>>>>>       check_spread(cfs_rq, se);
> >>>>>>>>>> -     if (!curr)
> >>>>>>>>>> +     if (!curr) {
> >>>>>>>>>>               __enqueue_entity(cfs_rq, se);
> >>>>>>>>>> +             __enqueue_latency(cfs_rq, se, flags);
> >>>>>>>>>> +     }
> >>>>>>>>>>       se->on_rq = 1;
> >>>>>>>>>>
> >>>>>>>>>>       if (cfs_rq->nr_running == 1) {
> >>>>>>>>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> >>>>>>>>>>
> >>>>>>>>>>       clear_buddies(cfs_rq, se);
> >>>>>>>>>>
> >>>>>>>>>> -     if (se != cfs_rq->curr)
> >>>>>>>>>> +     if (se != cfs_rq->curr) {
> >>>>>>>>>>               __dequeue_entity(cfs_rq, se);
> >>>>>>>>>> +             __dequeue_latency(cfs_rq, se);
> >>>>>>>>>> +     }
> >>>>>>>>>>       se->on_rq = 0;
> >>>>>>>>>>       account_entity_dequeue(cfs_rq, se);
> >>>>>>>>>>
> >>>>>>>>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >>>>>>>>>>                */
> >>>>>>>>>>               update_stats_wait_end_fair(cfs_rq, se);
> >>>>>>>>>>               __dequeue_entity(cfs_rq, se);
> >>>>>>>>>> +             __dequeue_latency(cfs_rq, se);
> >>>>>>>>>>               update_load_avg(cfs_rq, se, UPDATE_TG);
> >>>>>>>>>>       }
> >>>>>>>>>>
> >>>>>>>>>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
> >>>>>>>>>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>>>>>>>>  {
> >>>>>>>>>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
> >>>>>>>>>> -     struct sched_entity *se;
> >>>>>>>>>> +     struct sched_entity *latency, *se;
> >>>>>>>>>>
> >>>>>>>>>>       /*
> >>>>>>>>>>        * If curr is set we have to see if its left of the leftmost entity
> >>>>>>>>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>>>>>>>>               se = cfs_rq->last;
> >>>>>>>>>>       }
> >>>>>>>>>>
> >>>>>>>>>> +     /* Check for latency sensitive entity waiting for running */
> >>>>>>>>>> +     latency = __pick_first_latency(cfs_rq);
> >>>>>>>>>> +     if (latency && (latency != se) &&
> >>>>>>>>>> +         wakeup_preempt_entity(latency, se) < 1)
> >>>>>>>>>> +             se = latency;
> >>>>>>>>>> +
> >>>>>>>>>>       return se;
> >>>>>>>>>>  }
> >>>>>>>>>>
> >>>>>>>>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
> >>>>>>>>>>               update_stats_wait_start_fair(cfs_rq, prev);
> >>>>>>>>>>               /* Put 'current' back into the tree. */
> >>>>>>>>>>               __enqueue_entity(cfs_rq, prev);
> >>>>>>>>>> +             __enqueue_latency(cfs_rq, prev, 0);
> >>>>>>>>>>               /* in !on_rq case, update occurred at dequeue */
> >>>>>>>>>>               update_load_avg(cfs_rq, prev, 0);
> >>>>>>>>>>       }
> >>>>>>>>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> >>>>>>>>>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
> >>>>>>>>>>  {
> >>>>>>>>>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> >>>>>>>>>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
> >>>>>>>>>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
> >>>>>>>>>>  #ifdef CONFIG_SMP
> >>>>>>>>>>       raw_spin_lock_init(&cfs_rq->removed.lock);
> >>>>>>>>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
> >>>>>>>>>>       se->my_q = cfs_rq;
> >>>>>>>>>>
> >>>>>>>>>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
> >>>>>>>>>> +     RB_CLEAR_NODE(&se->latency_node);
> >>>>>>>>>>
> >>>>>>>>>>       /* guarantee group entities always have weight */
> >>>>>>>>>>       update_load_set(&se->load, NICE_0_LOAD);
> >>>>>>>>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
> >>>>>>>>>>
> >>>>>>>>>>       for_each_possible_cpu(i) {
> >>>>>>>>>>               struct sched_entity *se = tg->se[i];
> >>>>>>>>>> +             struct rq *rq = cpu_rq(i);
> >>>>>>>>>> +             struct rq_flags rf;
> >>>>>>>>>> +             bool queued;
> >>>>>>>>>> +
> >>>>>>>>>> +             rq_lock_irqsave(rq, &rf);
> >>>>>>>>>>
> >>>>>>>>>> +             queued = __dequeue_latency(se->cfs_rq, se);
> >>>>>>>>>>               WRITE_ONCE(se->latency_offset, latency_offset);
> >>>>>>>>>> +             if (queued)
> >>>>>>>>>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
> >>>>>>>>>> +
> >>>>>>>>>> +
> >>>>>>>>>> +             rq_unlock_irqrestore(rq, &rf);
> >>>>>>>>>>       }
> >>>>>>>>>>
> >>>>>>>>>>       mutex_unlock(&shares_mutex);
> >>>>>>>>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> >>>>>>>>>> index 9a2e71231083..21dd309e98a9 100644
> >>>>>>>>>> --- a/kernel/sched/sched.h
> >>>>>>>>>> +++ b/kernel/sched/sched.h
> >>>>>>>>>> @@ -570,6 +570,7 @@ struct cfs_rq {
> >>>>>>>>>>  #endif
> >>>>>>>>>>
> >>>>>>>>>>       struct rb_root_cached   tasks_timeline;
> >>>>>>>>>> +     struct rb_root_cached   latency_timeline;
> >>>>>>>>>>
> >>>>>>>>>>       /*
> >>>>>>>>>>        * 'curr' points to currently running entity on this cfs_rq.
> 

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Shrikanth Hegde]

> Le lundi 06 mars 2023 à 17:03:30 (+0530), Shrikanth Hegde a écrit :
>>
>> On 3/5/23 6:33 PM, Vincent Guittot wrote:
>>> On Sat, 4 Mar 2023 at 16:13, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>
>>>> On 3/3/23 10:01 PM, Vincent Guittot wrote:
>>>>> Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
>>>>>> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
>>>>>>> On 3/2/23 6:47 PM, Vincent Guittot wrote:
>>>>>>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
>>>>>>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
>>>>> [...]
>>>>>
>>>>>>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
>>>>>>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
>>>>>>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
>>>>>>>>>>> and other is running these benchmarks (group2). Set the latency nice
>>>>>>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
>>>>>>>>>>> Total of 96 CPU.
>>>>>>>>>>>
>>>>>>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
>>>>>>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
>>>>>>>>>>> and attach the numbers to series.
>>>>>>>>>>> Please suggest if any variation in the test i need to try.
>>>>>>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
>>>>>>>>>> run longer slice to move forward rather than preempting others all the
>>>>>>>>>> time
>>>>>>>>> hackbench still seems to regress in different latency nice values compared to
>>>>>>>>> baseline of 6.2 in this case. up to 50% in some cases.
>>>>>>>>>
>>>>>>>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
>>>>>>>>> running 2 CPU cgroups. No quota assigned.
>>>>>>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
>>>>>>>>> latency is not changed for this cgroup.
>>>>>>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
>>>>>>>>> nice values of 0, -20 and 19.
>>>>>>>> According to your other emails, you are using the cgroup interface and
>>>>>>>> not the task's one. Do I get it right ?
>>>>>>> right. I create cgroup, attach bash command with echo $$,
>>>>>>> assign the latency nice to cgroup, and run hackbench from that bash prompt.
>>>>>>>
>>>>>>>> I haven't run test such tests in a cgroup but at least the test with
>>>>>>>> latency_nice == 0 should not make any noticeable difference. Does this
>>>>>>>> include the re-arrange patch that you have proposed previously ?

Ran the test on a different system altogether. I don't see similar regression there. 
In fact latency nice is helping in reducing the latency as expected.
It is much bigger system with 60 cores. i.e total of 480 cpu.

Tested in the same way. created two cgroups. one is running the micro benchmarks
and other is running stress-ng at different utilization point.
This data is at 50% utilization point. Similar observations w.r.t latency
is seen at 0%, 25%, 75% and 100% utilization as well. 

==========
schbench
==========
	       6.2	      6.2 + V12 + LN=0
Groups: 1
50.0th:                 14.0             12.5
75.0th:                 16.5             14.0
90.0th:                 18.5             15.5
95.0th:                 20.5             17.0
99.0th:                 27.5             21.0
99.5th:                 36.0             23.5
Groups: 2
50.0th:                 14.0             16.0
75.0th:                 17.0             18.0
90.0th:                 20.0             21.0
95.0th:                 23.0             23.0
99.0th:                 71.0             34.0
99.5th:               1170.0             96.0
99.9th:               5088.0           3212.0
Groups: 4
50.0th:                 20.5             19.5
75.0th:                 24.5             22.5
90.0th:                 31.0             26.0
95.0th:                260.5             28.0
99.0th:               3644.0             35.0
99.5th:               5152.0             44.5
99.9th:               8076.0            168.5
Groups: 8
50.0th:                 26.0             25.5
75.0th:                 32.5             31.5
90.0th:                 41.5             36.5
95.0th:                794.0             39.5
99.0th:               5992.0             66.0
99.5th:               7208.0            159.0
99.9th:               9392.0           1604.0
Groups: 16
50.0th:                 37.5             34.0
75.0th:                 49.5             44.5
90.0th:                 70.0             53.5
95.0th:               1284.0             58.5
99.0th:               5600.0            102.5
99.5th:               7216.0            368.5
99.9th:               9328.0           5192.0
Groups: 32
50.0th:                 59.0             54.5
75.0th:                 83.0             74.5
90.0th:                118.5             91.0
95.0th:               1921.0            100.5
99.0th:               6672.0            317.0
99.5th:               8252.0           2264.0
99.9th:              10448.0           8388.0


===========
hackbench
==========

type                 Groups      6.2      | 6.2 + V12 + LN=0
Process               10         0.19     |  0.19
Process               20         0.34     |  0.34
Process               30         0.45     |  0.44
Process               40         0.58     |  0.57
Process               50         0.70     |  0.69
Process               60         0.82     |  0.80
thread                10         0.20     |  0.20
thread                20         0.36     |  0.36
Process(Pipe)         10         0.24     |  0.21
Process(Pipe)         20         0.46     |  0.40
Process(Pipe)         30         0.65     |  0.58
Process(Pipe)         40         0.90     |  0.68
Process(Pipe)         50         1.04     |  0.83
Process(Pipe)         60         1.16     |  0.86
thread(Pipe)          10         0.19     |  0.18
thread(Pipe)          20         0.46     |  0.37


>>>>>>> No. This is only with V12 of the series.
>>>>>>>
>>>>>>>> Also, the tests that you did on v6, gave better result.
>>>>>>>> https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@linux.vnet.ibm.com/
>>>>>>>>
>>>>>>>> Are you running same tests or you changed something in the mean time ?
>>>>>>> Test machine got changed.
>>>>>>> now i re-read my earlier mail. I see it was slightly different.
>>>>>>> I had created only one cgroup and stress-ng was run
>>>>>>> without any cgroup. Let me try that scenario and get the numbers.
>>>>>> Tried the same method of testing i had done on V7 of the series. on this
>>>>>> machine hackbench still regress's both on V12 as well as V7 of the series.
>>>>>>
>>>>>> Created one cpu cgroup called cgroup1. created two bash prompts.
>>>>>> assigned "bash $$" to cgroup1 and on other bash prompt running,
>>>>>> stress-ng --cpu=96 -l 50. Ran hackbench from cgroup1 prompt.
>>>>>> assigned latency values to the cgroup1.
>>>>> I have tried to reproduce your results on some of my systems but I can't see
>>>>> the impacts that you are reporting below.
>>>>> The fact that your other platform was not impacted as well could imply that
>>>>> it's specific to this platform.
>>>>> In particular, the lat nice=0 case should not show any real impact as it
>>>>> should be similar to a nop. At least that what I can see in the tests on my
>>>>> platforms and Prateek on his.
>>>>>
>>>>> Nevertheless, could you try to run your tests with the changes below ?
>>>>> These are the only places which could have an impact even with lat nice = 0
>>>>>
>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>>>> index 8137bca80572..979571a98b28 100644
>>>>> --- a/kernel/sched/fair.c
>>>>> +++ b/kernel/sched/fair.c
>>>>> @@ -4991,8 +4991,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>         if (delta < offset)
>>>>>                 return;
>>>>>
>>>>> -       if ((delta > ideal_runtime) ||
>>>>> -           (delta > get_latency_max()))
>>>>> +       if (delta > ideal_runtime)
>>>>>                 resched_curr(rq_of(cfs_rq));
>>>>>  }
>>>>>
>>>>> @@ -7574,9 +7573,10 @@ static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *
>>>>>          * Otherwise, use the latency weight to evaluate how much scheduling
>>>>>          * delay is acceptable by se.
>>>>>          */
>>>>> -       if ((latency_offset < 0) || (curr->latency_offset < 0))
>>>>> +       if ((latency_offset < 0) || (curr->latency_offset < 0)) {
>>>>>                 latency_offset -= curr->latency_offset;
>>>>> -       latency_offset = min_t(long, latency_offset, get_latency_max());
>>>>> +               latency_offset = min_t(long, latency_offset, get_latency_max());
>>>>> +       }
>>>>>
>>>>>         return latency_offset;
>>>>>  }
>>>>> @@ -7635,7 +7635,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>>>          * for low priority task. Make sure that long sleeping task will get a
>>>>>          * chance to preempt current.
>>>>>          */
>>>>> -       gran = min_t(s64, gran, get_latency_max());
>>>>>
>>>>>         if (vdiff > gran)
>>>>>                 return 1;
>>>>>
>>>> Above patch helps. thank you.
>>> Great. At least we have narrow the problem to one f the 3 changes.
>>>
>>>> Numbers are comparable to 6.2 and there is slight improvement. Much better than V12 numbers.
>>>>
>>>> type       groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
>>>>                   |             |lat nice=0| lat nice=-20| lat nice=+19
>>>>
>>>> Process       10  |     0.33    |   0.37   |   0.38     |   0.37
>>>> Process       20  |     0.61    |   0.67   |   0.68     |   0.67
>>>> Process       30  |     0.85    |   0.95   |   0.95     |   0.96
>>>> Process       40  |     1.10    |   1.20   |   1.20     |   1.21
>>>> Process       50  |     1.34    |   1.47   |   1.44     |   1.45
>>>> Process       60  |     1.57    |   1.70   |   1.71     |   1.70
>>>> thread        10  |     0.36    |   0.40   |   0.39     |   0.39
>>>> thread        20  |     0.65    |   0.72   |   0.71     |   0.71
>>>> Process(Pipe) 10  |     0.18    |   0.31   |   0.31     |   0.33
>>>> Process(Pipe) 20  |     0.32    |   0.51   |   0.50     |   0.50
>>>> Process(Pipe) 30  |     0.43    |   0.65   |   0.67     |   0.67
>>>> Process(Pipe) 40  |     0.57    |   0.82   |   0.83     |   0.83
>>>> Process(Pipe) 50  |     0.67    |   1.00   |   0.97     |   0.98
>>>> Process(Pipe) 60  |     0.81    |   1.13   |   1.11     |   1.12
>>>> thread(Pipe)  10  |     0.19    |   0.33   |   0.33     |   0.33
>>>> thread(Pipe)  20  |     0.34    |   0.53   |   0.51     |   0.52
>>>>
>>>>
>>>>
>>>> type       groups |   v6.2      |v6.2+ V12+ | v6.2 + V12+| v6.2 + V12
>>>>                   |             |above patch|above patch | above patch
>>>>                   |             |lat nice=0 |lat nice=-20| lat nice=+19
>>>>
>>>> Process       10  |     0.36    |   0.33    |   0.34     |   0.34
>>>> Process       20  |     0.62    |   0.60    |   0.61     |   0.61
>>>> Process       30  |     0.87    |   0.84    |   0.85     |   0.84
>>>> Process       40  |     1.13    |   1.09    |   1.10     |   1.09
>>>> Process       50  |     1.38    |   1.33    |   1.33     |   1.34
>>>> Process       60  |     1.64    |   1.56    |   1.58     |   1.56
>>>> thread        10  |     0.35    |   0.35    |   0.35     |   0.35
>>>> thread        20  |     0.64    |   0.63    |   0.63     |   0.63
>>>> Process(Pipe) 10  |     0.20    |   0.18    |   0.18     |   0.18
>>>> Process(Pipe) 20  |     0.32    |   0.31    |   0.31     |   0.32
>>>> Process(Pipe) 30  |     0.44    |   0.43    |   0.43     |   0.43
>>>> Process(Pipe) 40  |     0.56    |   0.57    |   0.56     |   0.55
>>>> Process(Pipe) 50  |     0.70    |   0.67    |   0.67     |   0.67
>>>> Process(Pipe) 60  |     0.83    |   0.79    |   0.81     |   0.80
>>>> thread(Pipe)  10  |     0.21    |   0.19    |   0.19     |   0.19
>>>> thread(Pipe)  20  |     0.35    |   0.33    |   0.34     |   0.33
>>>>
>>>>
>>>> Do you want me to try any other experiment on this further?
>>> Yes, would be good to know which of the 3 changes in the patch create
>>> the regression
>>>
>>> I suspect the 1st change to be the root cause of your problem but It
>>> would be good if you can confirm my assumption with some tests
>>>
>>> Thanks
>> Applied each change individually. 3rd change seems to cause the regression.
>> Kept only the 3rd change and numbers are same as stable 6.2 for latency nice
>> value of 0.
> Ok, it's the patch 1 that aims to prevent some unfairness with low weight
> waking task. And your platform probably falls in the last part of the commit:
>
> " Strictly speaking, we should use cfs->min_vruntime instead of
> curr->vruntime but it doesn't worth the additional overhead and complexity
> as the vruntime of current should be close to min_vruntime if not equal."
>
> Could you try the patch below on top of v12 ?
>
> ---
>  kernel/sched/fair.c | 21 +++++++++++----------
>  1 file changed, 11 insertions(+), 10 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index e2aeb4511686..77b03a280912 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -5049,7 +5049,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>  }
>
>  static int
> -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se);
> +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq);
>
>  /*
>   * Pick the next process, keeping these things in mind, in this order:
> @@ -5088,16 +5088,16 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>  				second = curr;
>  		}
>
> -		if (second && wakeup_preempt_entity(second, left) < 1)
> +		if (second && wakeup_preempt_entity(second, left, cfs_rq) < 1)
>  			se = second;
>  	}
>
> -	if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left) < 1) {
> +	if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left, cfs_rq) < 1) {
>  		/*
>  		 * Someone really wants this to run. If it's not unfair, run it.
>  		 */
>  		se = cfs_rq->next;
> -	} else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left) < 1) {
> +	} else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left, cfs_rq) < 1) {
>  		/*
>  		 * Prefer last buddy, try to return the CPU to a preempted task.
>  		 */
> @@ -5107,7 +5107,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>  	/* Check for latency sensitive entity waiting for running */
>  	latency = __pick_first_latency(cfs_rq);
>  	if (latency && (latency != se) &&
> -	    wakeup_preempt_entity(latency, se) < 1)
> +	    wakeup_preempt_entity(latency, se, cfs_rq) < 1)
>  		se = latency;
>
>  	return se;
> @@ -7808,7 +7808,7 @@ static unsigned long wakeup_gran(struct sched_entity *se)
>   *
>   */
>  static int
> -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq)
>  {
>  	s64 gran, vdiff = curr->vruntime - se->vruntime;
>  	s64 offset = wakeup_latency_gran(curr, se);
> @@ -7818,6 +7818,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>
>  	gran = offset + wakeup_gran(se);
>
> +	if (vdiff > gran)
> +		return 1;
>  	/*
>  	 * At wake up, the vruntime of a task is capped to not be older than
>  	 * a sched_latency period compared to min_vruntime. This prevents long
> @@ -7827,9 +7829,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>  	 * for low priority task. Make sure that long sleeping task will get a
>  	 * chance to preempt current.
>  	 */
> -	gran = min_t(s64, gran, get_latency_max());
> -
> -	if (vdiff > gran)
> +	vdiff = cfs_rq->min_vruntime - se->vruntime;
> +	if (vdiff > get_latency_max())
>  		return 1;
>
>  	return 0;
> @@ -7933,7 +7934,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
>  		return;
>
>  	update_curr(cfs_rq_of(se));
> -	if (wakeup_preempt_entity(se, pse) == 1) {
> +	if (wakeup_preempt_entity(se, pse, cfs_rq_of(se)) == 1) {
>  		/*
>  		 * Bias pick_next to pick the sched entity that is
>  		 * triggering this preemption.
> --
> 2.34.1

Tried above patch on top of V12. Numbers are worse than V12. We maybe running into
a corner case on this system.

Type                Groups       6.2     | 6.2+V12                                 
                                                                                   
 Process              10        0.33     |  0.44                                   
 Process              20        0.61     |  0.90                                   
 Process              30        0.87     |  1.29                                   
 Process              40        1.10     |  1.69                                   
 Process              50        1.34     |  2.08                                   
 Process              60        1.58     |  2.39                                   
 thread               10        0.36     |  0.53                                   
 thread               20        0.64     |  0.94                                   
 Process(Pipe)        10        0.18     |  0.46                                   
 Process(Pipe)        20        0.32     |  0.75                                   
 Process(Pipe)        30        0.42     |  1.01                                   
 Process(Pipe)        40        0.56     |  1.15                                   
 Process(Pipe)        50        0.68     |  1.38                                   
 Process(Pipe)        60        0.80     |  1.56  
 

>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index cdcd991bbcf1..c89c201dd164 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -7827,7 +7827,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>          * for low priority task. Make sure that long sleeping task will get a
>>          * chance to preempt current.
>>          */
>> -       gran = min_t(s64, gran, get_latency_max());
>>
>>         if (vdiff > gran)
>>                 return 1;
>>
>>
>>
>>>>>> I will try to run with only task's set with latency_nice=0 as well.
>>>>>>
>>>>>> type    groups |   v6.2      |v6.2 + V12| v6.2 + V12  | v6.2 + V12
>>>>>>                |             |lat nice=0| lat nice=-20| lat nice=+19
>>>>>>
>>>>>> Process            10  |     0.33    |   0.37   |   0.38     |   0.37
>>>>>> Process       20  |  0.61    |   0.67   |   0.68     |   0.67
>>>>>> Process            30  |     0.85    |   0.95   |   0.95     |   0.96
>>>>>> Process            40  |     1.10    |   1.20   |   1.20     |   1.21
>>>>>> Process            50  |     1.34    |   1.47   |   1.44     |   1.45
>>>>>> Process            60  |     1.57    |   1.70   |   1.71     |   1.70
>>>>>> thread             10  |     0.36    |   0.40   |   0.39     |   0.39
>>>>>> thread             20  |     0.65    |   0.72   |   0.71     |   0.71
>>>>>> Process(Pipe) 10  |  0.18    |   0.31   |   0.31     |   0.33
>>>>>> Process(Pipe) 20  |  0.32    |   0.51   |   0.50     |   0.50
>>>>>> Process(Pipe) 30  |  0.43    |   0.65   |   0.67     |   0.67
>>>>>> Process(Pipe) 40  |  0.57    |   0.82   |   0.83     |   0.83
>>>>>> Process(Pipe) 50  |  0.67    |   1.00   |   0.97     |   0.98
>>>>>> Process(Pipe) 60  |  0.81    |   1.13   |   1.11     |   1.12
>>>>>> thread(Pipe)  10  |  0.19    |   0.33   |   0.33     |   0.33
>>>>>> thread(Pipe)  20  |  0.34    |   0.53   |   0.51     |   0.52
>>>>>>
>>>>>>
>>>>>>
>>>>>> type    groups |   v6.2      |v6.2 + V7 | v6.2 + V7  | v6.2 + V7
>>>>>>                |             |lat nice=0|lat nice=-20| lat nice=+19
>>>>>> Process            10  |     0.33    |   0.37   |   0.37     |   0.37
>>>>>> Process            20  |     0.61    |   0.67   |   0.67     |   0.67
>>>>>> Process            30  |     0.85    |   0.96   |   0.94     |   0.95
>>>>>> Process            40  |     1.10    |   1.20   |   1.20     |   1.20
>>>>>> Process            50  |     1.34    |   1.45   |   1.46     |   1.45
>>>>>> Process            60  |     1.57    |   1.71   |   1.68     |   1.72
>>>>>> thread             10  |     0.36    |   0.40   |   0.40     |   0.40
>>>>>> thread             20  |     0.65    |   0.71   |   0.71     |   0.71
>>>>>> Process(Pipe) 10  |  0.18    |   0.30   |   0.30     |   0.31
>>>>>> Process(Pipe) 20  |  0.32    |   0.50   |   0.50     |   0.50
>>>>>> Process(Pipe) 30  |  0.43    |   0.67   |   0.67     |   0.66
>>>>>> Process(Pipe) 40  |  0.57    |   0.86   |   0.84     |   0.84
>>>>>> Process(Pipe) 50  |  0.67    |   0.99   |   0.97     |   0.97
>>>>>> Process(Pipe) 60  |  0.81    |   1.10   |   1.13     |   1.13
>>>>>> thread(Pipe)  10  |  0.19    |   0.34   |   0.34     |   0.33
>>>>>> thread(Pipe)  20  |  0.34    |   0.55   |   0.53     |   0.54
>>>>>>
>>>>>>>>> Numbers are average of 10 runs in each case. Time is in seconds
>>>>>>>>>
>>>>>>>>> type       groups |   v6.2     |  v6.2 + V12   | v6.2 + V12  | v6.2 + V12
>>>>>>>>>                   |            | lat nice=0    | lat nice=-20| lat nice=+19
>>>>>>>>>                   |            |               |             |
>>>>>>>>> Process       10  |   0.36     |     0.41      |    0.43     |    0.42
>>>>>>>>> Process       20  |   0.62     |     0.76      |    0.75     |    0.75
>>>>>>>>> Process       30  |   0.87     |     1.05      |    1.04     |    1.06
>>>>>>>>> Process       40  |   1.13     |     1.34      |    1.33     |    1.33
>>>>>>>>> Process       50  |   1.38     |     1.62      |    1.66     |    1.63
>>>>>>>>> Process       60  |   1.64     |     1.91      |    1.97     |    1.90
>>>>>>>>> thread        10  |   0.35     |     0.41      |    0.44     |    0.42
>>>>>>>>> thread        20  |   0.64     |     0.78      |    0.77     |    0.79
>>>>>>>>> Process(Pipe) 10  |   0.20     |     0.34      |    0.33     |    0.34
>>>>>>>>> Process(Pipe) 20  |   0.32     |     0.52      |    0.53     |    0.52
>>>>>>>>> Process(Pipe) 30  |   0.44     |     0.70      |    0.70     |    0.69
>>>>>>>>> Process(Pipe) 40  |   0.56     |     0.88      |    0.89     |    0.88
>>>>>>>>> Process(Pipe) 50  |   0.70     |     1.08      |    1.08     |    1.07
>>>>>>>>> Process(Pipe) 60  |   0.83     |     1.27      |    1.27     |    1.26
>>>>>>>>> thread(Pipe)  10  |   0.21     |     0.35      |    0.34     |    0.36
>>>>>>>>> thread(Pipe)  10  |   0.35     |     0.55      |    0.58     |    0.55
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> Re-arrange seems to help the patch series by avoiding an cacheline miss.
>>>>>>>>>>>
>>>>>>>>>>> =========================
>>>>>>>>>>> schbench
>>>>>>>>>>> =========================
>>>>>>>>>>>                  6.2   |  6.2 + V12     |     6.2 + V12 + re-arrange
>>>>>>>>>>> 1 Thread
>>>>>>>>>>>   50.0th:        9.00  |    9.00        |        9.50
>>>>>>>>>>>   75.0th:       10.50  |   10.00        |        9.50
>>>>>>>>>>>   90.0th:       11.00  |   11.00        |       10.50
>>>>>>>>>>>   95.0th:       11.00  |   11.00        |       11.00
>>>>>>>>>>>   99.0th:       11.50  |   11.50        |       11.50
>>>>>>>>>>>   99.5th:       12.50  |   12.00        |       12.00
>>>>>>>>>>>   99.9th:       14.50  |   13.50        |       12.00
>>>>>>>>>>> 2 Threads
>>>>>>>>>>>   50.0th:        9.50  |    9.50        |        8.50
>>>>>>>>>>>   75.0th:       11.00  |   10.50        |        9.50
>>>>>>>>>>>   90.0th:       13.50  |   11.50        |       10.50
>>>>>>>>>>>   95.0th:       14.00  |   12.00        |       11.00
>>>>>>>>>>>   99.0th:       15.50  |   13.50        |       12.00
>>>>>>>>>>>   99.5th:       16.00  |   14.00        |       12.00
>>>>>>>>>>>   99.9th:       17.00  |   16.00        |       16.50
>>>>>>>>>>> 4 Threads
>>>>>>>>>>>   50.0th:       11.50  |   11.50        |       10.50
>>>>>>>>>>>   75.0th:       13.50  |   12.50        |       12.50
>>>>>>>>>>>   90.0th:       15.50  |   14.50        |       14.00
>>>>>>>>>>>   95.0th:       16.50  |   15.50        |       14.50
>>>>>>>>>>>   99.0th:       20.00  |   17.50        |       16.50
>>>>>>>>>>>   99.5th:       20.50  |   18.50        |       17.00
>>>>>>>>>>>   99.9th:       22.50  |   21.00        |       19.00
>>>>>>>>>>> 8 Threads
>>>>>>>>>>>   50.0th:       14.00  |   14.00        |       14.00
>>>>>>>>>>>   75.0th:       16.00  |   16.00        |       16.00
>>>>>>>>>>>   90.0th:       18.00  |   18.00        |       17.50
>>>>>>>>>>>   95.0th:       18.50  |   18.50        |       18.50
>>>>>>>>>>>   99.0th:       20.00  |   20.00        |       20.00
>>>>>>>>>>>   99.5th:       20.50  |   21.50        |       21.00
>>>>>>>>>>>   99.9th:       22.50  |   23.50        |       23.00
>>>>>>>>>>> 16 Threads
>>>>>>>>>>>   50.0th:       19.00  |   18.50        |       19.00
>>>>>>>>>>>   75.0th:       23.00  |   22.50        |       23.00
>>>>>>>>>>>   90.0th:       25.00  |   25.50        |       25.00
>>>>>>>>>>>   95.0th:       26.50  |   26.50        |       26.00
>>>>>>>>>>>   99.0th:       28.50  |   29.00        |       28.50
>>>>>>>>>>>   99.5th:       31.00  |   30.00        |       30.00
>>>>>>>>>>>   99.9th:     5626.00  | 4761.50        |       32.50
>>>>>>>>>>> 32 Threads
>>>>>>>>>>>   50.0th:       27.00  |   27.50        |       29.00
>>>>>>>>>>>   75.0th:       35.50  |   36.50        |       38.50
>>>>>>>>>>>   90.0th:       42.00  |   44.00        |       50.50
>>>>>>>>>>>   95.0th:      447.50  | 2959.00        |     8544.00
>>>>>>>>>>>   99.0th:     7372.00  | 17032.00       |    19136.00
>>>>>>>>>>>   99.5th:    15360.00  | 19808.00       |    20704.00
>>>>>>>>>>>   99.9th:    20640.00  | 30048.00       |    30048.00
>>>>>>>>>>>
>>>>>>>>>>> ====================
>>>>>>>>>>> hackbench
>>>>>>>>>>> ====================
>>>>>>>>>>>                         6.2     |  6.2 + V12        |     6.2+ V12 +re-arrange
>>>>>>>>>>>
>>>>>>>>>>> Process 10 Time:        0.35    |       0.42        |           0.41
>>>>>>>>>>> Process 20 Time:        0.61    |       0.76        |           0.76
>>>>>>>>>>> Process 30 Time:        0.87    |       1.06        |           1.05
>>>>>>>>>>> thread 10 Time:         0.35    |       0.43        |           0.42
>>>>>>>>>>> thread 20 Time:         0.66    |       0.79        |           0.78
>>>>>>>>>>> Process(Pipe) 10 Time:  0.21    |       0.33        |           0.32
>>>>>>>>>>> Process(Pipe) 20 Time:  0.34    |       0.52        |           0.52
>>>>>>>>>>> Process(Pipe) 30 Time:  0.46    |       0.72        |           0.71
>>>>>>>>>>> thread(Pipe) 10 Time:   0.21    |       0.34        |           0.34
>>>>>>>>>>> thread(Pipe) 20 Time:   0.36    |       0.56        |           0.56
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>       struct list_head                group_node;
>>>>>>>>>>>>       unsigned int                    on_rq;
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>>>>>>>>>> index 093cc1af73dc..752fd364216c 100644
>>>>>>>>>>>> --- a/kernel/sched/core.c
>>>>>>>>>>>> +++ b/kernel/sched/core.c
>>>>>>>>>>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
>>>>>>>>>>>>       p->se.nr_migrations             = 0;
>>>>>>>>>>>>       p->se.vruntime                  = 0;
>>>>>>>>>>>>       INIT_LIST_HEAD(&p->se.group_node);
>>>>>>>>>>>> +     RB_CLEAR_NODE(&p->se.latency_node);
>>>>>>>>>>>>
>>>>>>>>>>>>  #ifdef CONFIG_FAIR_GROUP_SCHED
>>>>>>>>>>>>       p->se.cfs_rq                    = NULL;
>>>>>>>>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>>>>>>>>>>> index 125a6ff53378..e2aeb4511686 100644
>>>>>>>>>>>> --- a/kernel/sched/fair.c
>>>>>>>>>>>> +++ b/kernel/sched/fair.c
>>>>>>>>>>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq)
>>>>>>>>>>>>
>>>>>>>>>>>>       return __node_2_se(last);
>>>>>>>>>>>>  }
>>>>>>>>>>>> +#endif
>>>>>>>>>>>>
>>>>>>>>>>>> +/**************************************************************
>>>>>>>>>>>> + * Scheduling class tree data structure manipulation methods:
>>>>>>>>>>>> + * for latency
>>>>>>>>>>>> + */
>>>>>>>>>>>> +
>>>>>>>>>>>> +static inline bool latency_before(struct sched_entity *a,
>>>>>>>>>>>> +                             struct sched_entity *b)
>>>>>>>>>>>> +{
>>>>>>>>>>>> +     return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0;
>>>>>>>>>>>> +}
>>>>>>>>>>>> +
>>>>>>>>>>>> +#define __latency_node_2_se(node) \
>>>>>>>>>>>> +     rb_entry((node), struct sched_entity, latency_node)
>>>>>>>>>>>> +
>>>>>>>>>>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b)
>>>>>>>>>>>> +{
>>>>>>>>>>>> +     return latency_before(__latency_node_2_se(a), __latency_node_2_se(b));
>>>>>>>>>>>> +}
>>>>>>>>>>>> +
>>>>>>>>>>>> +/*
>>>>>>>>>>>> + * Enqueue an entity into the latency rb-tree:
>>>>>>>>>>>> + */
>>>>>>>>>>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>>>>>>> +{
>>>>>>>>>>>> +
>>>>>>>>>>>> +     /* Only latency sensitive entity can be added to the list */
>>>>>>>>>>>> +     if (se->latency_offset >= 0)
>>>>>>>>>>>> +             return;
>>>>>>>>>>>> +
>>>>>>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node))
>>>>>>>>>>>> +             return;
>>>>>>>>>>>> +
>>>>>>>>>>>> +     /*
>>>>>>>>>>>> +      * The entity is always added the latency list at wakeup.
>>>>>>>>>>>> +      * Then, a not waking up entity that is put back in the list after an
>>>>>>>>>>>> +      * execution time less than sysctl_sched_min_granularity, means that
>>>>>>>>>>>> +      * the entity has been preempted by a higher sched class or an entity
>>>>>>>>>>>> +      * with higher latency constraint. In thi case, the entity is also put
>>>>>>>>>>>> +      * back in the latency list so it gets a chance to run 1st during the
>>>>>>>>>>>> +      * next slice.
>>>>>>>>>>>> +      */
>>>>>>>>>>>> +     if (!(flags & ENQUEUE_WAKEUP)) {
>>>>>>>>>>>> +             u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime;
>>>>>>>>>>>> +
>>>>>>>>>>>> +             if (delta_exec >= sysctl_sched_min_granularity)
>>>>>>>>>>>> +                     return;
>>>>>>>>>>>> +     }
>>>>>>>>>>>> +
>>>>>>>>>>>> +     rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less);
>>>>>>>>>>>> +}
>>>>>>>>>>>> +
>>>>>>>>>>>> +/*
>>>>>>>>>>>> + * Dequeue an entity from the latency rb-tree and return true if it was really
>>>>>>>>>>>> + * part of the rb-tree:
>>>>>>>>>>>> + */
>>>>>>>>>>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>>>>>>>> +{
>>>>>>>>>>>> +     if (!RB_EMPTY_NODE(&se->latency_node)) {
>>>>>>>>>>>> +             rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline);
>>>>>>>>>>>> +             RB_CLEAR_NODE(&se->latency_node);
>>>>>>>>>>>> +             return true;
>>>>>>>>>>>> +     }
>>>>>>>>>>>> +
>>>>>>>>>>>> +     return false;
>>>>>>>>>>>> +}
>>>>>>>>>>>> +
>>>>>>>>>>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq)
>>>>>>>>>>>> +{
>>>>>>>>>>>> +     struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline);
>>>>>>>>>>>> +
>>>>>>>>>>>> +     if (!left)
>>>>>>>>>>>> +             return NULL;
>>>>>>>>>>>> +
>>>>>>>>>>>> +     return __latency_node_2_se(left);
>>>>>>>>>>>> +}
>>>>>>>>>>>> +
>>>>>>>>>>>> +#ifdef CONFIG_SCHED_DEBUG
>>>>>>>>>>>>  /**************************************************************
>>>>>>>>>>>>   * Scheduling class statistics methods:
>>>>>>>>>>>>   */
>>>>>>>>>>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>>>>>>>       check_schedstat_required();
>>>>>>>>>>>>       update_stats_enqueue_fair(cfs_rq, se, flags);
>>>>>>>>>>>>       check_spread(cfs_rq, se);
>>>>>>>>>>>> -     if (!curr)
>>>>>>>>>>>> +     if (!curr) {
>>>>>>>>>>>>               __enqueue_entity(cfs_rq, se);
>>>>>>>>>>>> +             __enqueue_latency(cfs_rq, se, flags);
>>>>>>>>>>>> +     }
>>>>>>>>>>>>       se->on_rq = 1;
>>>>>>>>>>>>
>>>>>>>>>>>>       if (cfs_rq->nr_running == 1) {
>>>>>>>>>>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>>>>>>>>>>>>
>>>>>>>>>>>>       clear_buddies(cfs_rq, se);
>>>>>>>>>>>>
>>>>>>>>>>>> -     if (se != cfs_rq->curr)
>>>>>>>>>>>> +     if (se != cfs_rq->curr) {
>>>>>>>>>>>>               __dequeue_entity(cfs_rq, se);
>>>>>>>>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>>>>>>>>> +     }
>>>>>>>>>>>>       se->on_rq = 0;
>>>>>>>>>>>>       account_entity_dequeue(cfs_rq, se);
>>>>>>>>>>>>
>>>>>>>>>>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>>>>>>>>                */
>>>>>>>>>>>>               update_stats_wait_end_fair(cfs_rq, se);
>>>>>>>>>>>>               __dequeue_entity(cfs_rq, se);
>>>>>>>>>>>> +             __dequeue_latency(cfs_rq, se);
>>>>>>>>>>>>               update_load_avg(cfs_rq, se, UPDATE_TG);
>>>>>>>>>>>>       }
>>>>>>>>>>>>
>>>>>>>>>>>> @@ -4979,7 +5062,7 @@ static struct sched_entity *
>>>>>>>>>>>>  pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>>>>>>>>  {
>>>>>>>>>>>>       struct sched_entity *left = __pick_first_entity(cfs_rq);
>>>>>>>>>>>> -     struct sched_entity *se;
>>>>>>>>>>>> +     struct sched_entity *latency, *se;
>>>>>>>>>>>>
>>>>>>>>>>>>       /*
>>>>>>>>>>>>        * If curr is set we have to see if its left of the leftmost entity
>>>>>>>>>>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>>>>>>>>               se = cfs_rq->last;
>>>>>>>>>>>>       }
>>>>>>>>>>>>
>>>>>>>>>>>> +     /* Check for latency sensitive entity waiting for running */
>>>>>>>>>>>> +     latency = __pick_first_latency(cfs_rq);
>>>>>>>>>>>> +     if (latency && (latency != se) &&
>>>>>>>>>>>> +         wakeup_preempt_entity(latency, se) < 1)
>>>>>>>>>>>> +             se = latency;
>>>>>>>>>>>> +
>>>>>>>>>>>>       return se;
>>>>>>>>>>>>  }
>>>>>>>>>>>>
>>>>>>>>>>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
>>>>>>>>>>>>               update_stats_wait_start_fair(cfs_rq, prev);
>>>>>>>>>>>>               /* Put 'current' back into the tree. */
>>>>>>>>>>>>               __enqueue_entity(cfs_rq, prev);
>>>>>>>>>>>> +             __enqueue_latency(cfs_rq, prev, 0);
>>>>>>>>>>>>               /* in !on_rq case, update occurred at dequeue */
>>>>>>>>>>>>               update_load_avg(cfs_rq, prev, 0);
>>>>>>>>>>>>       }
>>>>>>>>>>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
>>>>>>>>>>>>  void init_cfs_rq(struct cfs_rq *cfs_rq)
>>>>>>>>>>>>  {
>>>>>>>>>>>>       cfs_rq->tasks_timeline = RB_ROOT_CACHED;
>>>>>>>>>>>> +     cfs_rq->latency_timeline = RB_ROOT_CACHED;
>>>>>>>>>>>>       u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20)));
>>>>>>>>>>>>  #ifdef CONFIG_SMP
>>>>>>>>>>>>       raw_spin_lock_init(&cfs_rq->removed.lock);
>>>>>>>>>>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq,
>>>>>>>>>>>>       se->my_q = cfs_rq;
>>>>>>>>>>>>
>>>>>>>>>>>>       se->latency_offset = calc_latency_offset(tg->latency_prio);
>>>>>>>>>>>> +     RB_CLEAR_NODE(&se->latency_node);
>>>>>>>>>>>>
>>>>>>>>>>>>       /* guarantee group entities always have weight */
>>>>>>>>>>>>       update_load_set(&se->load, NICE_0_LOAD);
>>>>>>>>>>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio)
>>>>>>>>>>>>
>>>>>>>>>>>>       for_each_possible_cpu(i) {
>>>>>>>>>>>>               struct sched_entity *se = tg->se[i];
>>>>>>>>>>>> +             struct rq *rq = cpu_rq(i);
>>>>>>>>>>>> +             struct rq_flags rf;
>>>>>>>>>>>> +             bool queued;
>>>>>>>>>>>> +
>>>>>>>>>>>> +             rq_lock_irqsave(rq, &rf);
>>>>>>>>>>>>
>>>>>>>>>>>> +             queued = __dequeue_latency(se->cfs_rq, se);
>>>>>>>>>>>>               WRITE_ONCE(se->latency_offset, latency_offset);
>>>>>>>>>>>> +             if (queued)
>>>>>>>>>>>> +                     __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP);
>>>>>>>>>>>> +
>>>>>>>>>>>> +
>>>>>>>>>>>> +             rq_unlock_irqrestore(rq, &rf);
>>>>>>>>>>>>       }
>>>>>>>>>>>>
>>>>>>>>>>>>       mutex_unlock(&shares_mutex);
>>>>>>>>>>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
>>>>>>>>>>>> index 9a2e71231083..21dd309e98a9 100644
>>>>>>>>>>>> --- a/kernel/sched/sched.h
>>>>>>>>>>>> +++ b/kernel/sched/sched.h
>>>>>>>>>>>> @@ -570,6 +570,7 @@ struct cfs_rq {
>>>>>>>>>>>>  #endif
>>>>>>>>>>>>
>>>>>>>>>>>>       struct rb_root_cached   tasks_timeline;
>>>>>>>>>>>> +     struct rb_root_cached   latency_timeline;
>>>>>>>>>>>>
>>>>>>>>>>>>       /*
>>>>>>>>>>>>        * 'curr' points to currently running entity on this cfs_rq.

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Vincent Guittot]

Le mardi 07 mars 2023 à 00:34:49 (+0530), Shrikanth Hegde a écrit :
> 
> > Le lundi 06 mars 2023 à 17:03:30 (+0530), Shrikanth Hegde a écrit :
> >>
> >> On 3/5/23 6:33 PM, Vincent Guittot wrote:
> >>> On Sat, 4 Mar 2023 at 16:13, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>
> >>>> On 3/3/23 10:01 PM, Vincent Guittot wrote:
> >>>>> Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
> >>>>>> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
> >>>>>>> On 3/2/23 6:47 PM, Vincent Guittot wrote:
> >>>>>>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>>>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
> >>>>>>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>>>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
> >>>>> [...]
> >>>>>
> >>>>>>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
> >>>>>>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
> >>>>>>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
> >>>>>>>>>>> and other is running these benchmarks (group2). Set the latency nice
> >>>>>>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
> >>>>>>>>>>> Total of 96 CPU.
> >>>>>>>>>>>
> >>>>>>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
> >>>>>>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
> >>>>>>>>>>> and attach the numbers to series.
> >>>>>>>>>>> Please suggest if any variation in the test i need to try.
> >>>>>>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
> >>>>>>>>>> run longer slice to move forward rather than preempting others all the
> >>>>>>>>>> time
> >>>>>>>>> hackbench still seems to regress in different latency nice values compared to
> >>>>>>>>> baseline of 6.2 in this case. up to 50% in some cases.
> >>>>>>>>>
> >>>>>>>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
> >>>>>>>>> running 2 CPU cgroups. No quota assigned.
> >>>>>>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
> >>>>>>>>> latency is not changed for this cgroup.
> >>>>>>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
> >>>>>>>>> nice values of 0, -20 and 19.
> >>>>>>>> According to your other emails, you are using the cgroup interface and
> >>>>>>>> not the task's one. Do I get it right ?
> >>>>>>> right. I create cgroup, attach bash command with echo $$,
> >>>>>>> assign the latency nice to cgroup, and run hackbench from that bash prompt.
> >>>>>>>
> >>>>>>>> I haven't run test such tests in a cgroup but at least the test with
> >>>>>>>> latency_nice == 0 should not make any noticeable difference. Does this
> >>>>>>>> include the re-arrange patch that you have proposed previously ?
> 
> Ran the test on a different system altogether. I don't see similar regression there. 
> In fact latency nice is helping in reducing the latency as expected.
> It is much bigger system with 60 cores. i.e total of 480 cpu.
> 
> Tested in the same way. created two cgroups. one is running the micro benchmarks
> and other is running stress-ng at different utilization point.
> This data is at 50% utilization point. Similar observations w.r.t latency
> is seen at 0%, 25%, 75% and 100% utilization as well. 
> 

Thanks for testing on a different system which seems to get results aligned with what
Prateek and I have seen on our system.


> ==========
> schbench
> ==========
> 	       6.2	      6.2 + V12 + LN=0
> Groups: 1
> 50.0th:                 14.0             12.5
> 75.0th:                 16.5             14.0
> 90.0th:                 18.5             15.5
> 95.0th:                 20.5             17.0
> 99.0th:                 27.5             21.0
> 99.5th:                 36.0             23.5
> Groups: 2
> 50.0th:                 14.0             16.0
> 75.0th:                 17.0             18.0
> 90.0th:                 20.0             21.0
> 95.0th:                 23.0             23.0
> 99.0th:                 71.0             34.0
> 99.5th:               1170.0             96.0
> 99.9th:               5088.0           3212.0
> Groups: 4
> 50.0th:                 20.5             19.5
> 75.0th:                 24.5             22.5
> 90.0th:                 31.0             26.0
> 95.0th:                260.5             28.0
> 99.0th:               3644.0             35.0
> 99.5th:               5152.0             44.5
> 99.9th:               8076.0            168.5
> Groups: 8
> 50.0th:                 26.0             25.5
> 75.0th:                 32.5             31.5
> 90.0th:                 41.5             36.5
> 95.0th:                794.0             39.5
> 99.0th:               5992.0             66.0
> 99.5th:               7208.0            159.0
> 99.9th:               9392.0           1604.0
> Groups: 16
> 50.0th:                 37.5             34.0
> 75.0th:                 49.5             44.5
> 90.0th:                 70.0             53.5
> 95.0th:               1284.0             58.5
> 99.0th:               5600.0            102.5
> 99.5th:               7216.0            368.5
> 99.9th:               9328.0           5192.0
> Groups: 32
> 50.0th:                 59.0             54.5
> 75.0th:                 83.0             74.5
> 90.0th:                118.5             91.0
> 95.0th:               1921.0            100.5
> 99.0th:               6672.0            317.0
> 99.5th:               8252.0           2264.0
> 99.9th:              10448.0           8388.0
> 
> 
> ===========
> hackbench
> ==========
> 
> type                 Groups      6.2      | 6.2 + V12 + LN=0
> Process               10         0.19     |  0.19
> Process               20         0.34     |  0.34
> Process               30         0.45     |  0.44
> Process               40         0.58     |  0.57
> Process               50         0.70     |  0.69
> Process               60         0.82     |  0.80
> thread                10         0.20     |  0.20
> thread                20         0.36     |  0.36
> Process(Pipe)         10         0.24     |  0.21
> Process(Pipe)         20         0.46     |  0.40
> Process(Pipe)         30         0.65     |  0.58
> Process(Pipe)         40         0.90     |  0.68
> Process(Pipe)         50         1.04     |  0.83
> Process(Pipe)         60         1.16     |  0.86
> thread(Pipe)          10         0.19     |  0.18
> thread(Pipe)          20         0.46     |  0.37
> 
> 

[...]

> >>>>
> >>>>
> >>>> Do you want me to try any other experiment on this further?
> >>> Yes, would be good to know which of the 3 changes in the patch create
> >>> the regression
> >>>
> >>> I suspect the 1st change to be the root cause of your problem but It
> >>> would be good if you can confirm my assumption with some tests
> >>>
> >>> Thanks
> >> Applied each change individually. 3rd change seems to cause the regression.
> >> Kept only the 3rd change and numbers are same as stable 6.2 for latency nice
> >> value of 0.
> > Ok, it's the patch 1 that aims to prevent some unfairness with low weight
> > waking task. And your platform probably falls in the last part of the commit:
> >
> > " Strictly speaking, we should use cfs->min_vruntime instead of
> > curr->vruntime but it doesn't worth the additional overhead and complexity
> > as the vruntime of current should be close to min_vruntime if not equal."
> >
> > Could you try the patch below on top of v12 ?
> >
> > ---
> >  kernel/sched/fair.c | 21 +++++++++++----------
> >  1 file changed, 11 insertions(+), 10 deletions(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index e2aeb4511686..77b03a280912 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -5049,7 +5049,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >  }
> >
> >  static int
> > -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se);
> > +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq);
> >
> >  /*
> >   * Pick the next process, keeping these things in mind, in this order:
> > @@ -5088,16 +5088,16 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >  				second = curr;
> >  		}
> >
> > -		if (second && wakeup_preempt_entity(second, left) < 1)
> > +		if (second && wakeup_preempt_entity(second, left, cfs_rq) < 1)
> >  			se = second;
> >  	}
> >
> > -	if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left) < 1) {
> > +	if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left, cfs_rq) < 1) {
> >  		/*
> >  		 * Someone really wants this to run. If it's not unfair, run it.
> >  		 */
> >  		se = cfs_rq->next;
> > -	} else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left) < 1) {
> > +	} else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left, cfs_rq) < 1) {
> >  		/*
> >  		 * Prefer last buddy, try to return the CPU to a preempted task.
> >  		 */
> > @@ -5107,7 +5107,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >  	/* Check for latency sensitive entity waiting for running */
> >  	latency = __pick_first_latency(cfs_rq);
> >  	if (latency && (latency != se) &&
> > -	    wakeup_preempt_entity(latency, se) < 1)
> > +	    wakeup_preempt_entity(latency, se, cfs_rq) < 1)
> >  		se = latency;
> >
> >  	return se;
> > @@ -7808,7 +7808,7 @@ static unsigned long wakeup_gran(struct sched_entity *se)
> >   *
> >   */
> >  static int
> > -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> > +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq)
> >  {
> >  	s64 gran, vdiff = curr->vruntime - se->vruntime;
> >  	s64 offset = wakeup_latency_gran(curr, se);
> > @@ -7818,6 +7818,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >
> >  	gran = offset + wakeup_gran(se);
> >
> > +	if (vdiff > gran)
> > +		return 1;
> >  	/*
> >  	 * At wake up, the vruntime of a task is capped to not be older than
> >  	 * a sched_latency period compared to min_vruntime. This prevents long
> > @@ -7827,9 +7829,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >  	 * for low priority task. Make sure that long sleeping task will get a
> >  	 * chance to preempt current.
> >  	 */
> > -	gran = min_t(s64, gran, get_latency_max());
> > -
> > -	if (vdiff > gran)
> > +	vdiff = cfs_rq->min_vruntime - se->vruntime;
> > +	if (vdiff > get_latency_max())
> >  		return 1;
> >
> >  	return 0;
> > @@ -7933,7 +7934,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
> >  		return;
> >
> >  	update_curr(cfs_rq_of(se));
> > -	if (wakeup_preempt_entity(se, pse) == 1) {
> > +	if (wakeup_preempt_entity(se, pse, cfs_rq_of(se)) == 1) {
> >  		/*
> >  		 * Bias pick_next to pick the sched entity that is
> >  		 * triggering this preemption.
> > --
> > 2.34.1
> 
> Tried above patch on top of V12. Numbers are worse than V12. We maybe running into
> a corner case on this system.

Yes it can be a corner case.

Nevertheless, the patch above has a problem and does an unsigned comparison instead of a signed
one. I have forced the signed comparison in the patch below to be applied on top  of
previous one:

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 77b03a280912..22a497f92dbb 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7830,7 +7830,7 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct
         * chance to preempt current.
         */
        vdiff = cfs_rq->min_vruntime - se->vruntime;
-       if (vdiff > get_latency_max())
+       if (vdiff > (s64)get_latency_max())
                return 1;

        return 0;


> 
> Type                Groups       6.2     | 6.2+V12                                 
>                                                                                    
>  Process              10        0.33     |  0.44                                   
>  Process              20        0.61     |  0.90                                   
>  Process              30        0.87     |  1.29                                   
>  Process              40        1.10     |  1.69                                   
>  Process              50        1.34     |  2.08                                   
>  Process              60        1.58     |  2.39                                   
>  thread               10        0.36     |  0.53                                   
>  thread               20        0.64     |  0.94                                   
>  Process(Pipe)        10        0.18     |  0.46                                   
>  Process(Pipe)        20        0.32     |  0.75                                   
>  Process(Pipe)        30        0.42     |  1.01                                   
>  Process(Pipe)        40        0.56     |  1.15                                   
>  Process(Pipe)        50        0.68     |  1.38                                   
>  Process(Pipe)        60        0.80     |  1.56  
>  
> 
> >> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >> index cdcd991bbcf1..c89c201dd164 100644
> >> --- a/kernel/sched/fair.c
> >> +++ b/kernel/sched/fair.c
> >> @@ -7827,7 +7827,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >>          * for low priority task. Make sure that long sleeping task will get a
> >>          * chance to preempt current.
> >>          */
> >> -       gran = min_t(s64, gran, get_latency_max());
> >>
> >>         if (vdiff > gran)
> >>                 return 1;
> >>
> >>
> >>

[...]

> >>>>>>>>>>>>        * 'curr' points to currently running entity on this cfs_rq.
> 

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Shrikanth Hegde]

On 3/7/23 3:49 PM, Vincent Guittot wrote:
> Le mardi 07 mars 2023 à 00:34:49 (+0530), Shrikanth Hegde a écrit :
>>> Le lundi 06 mars 2023 à 17:03:30 (+0530), Shrikanth Hegde a écrit :
>>>> On 3/5/23 6:33 PM, Vincent Guittot wrote:
>>>>> On Sat, 4 Mar 2023 at 16:13, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>> On 3/3/23 10:01 PM, Vincent Guittot wrote:
>>>>>>> Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
>>>>>>>> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
>>>>>>>>> On 3/2/23 6:47 PM, Vincent Guittot wrote:
>>>>>>>>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>>>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
>>>>>>>>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>>>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
>>>>>>> [...]
>>>>>>>
>>>>>>>>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
>>>>>>>>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
>>>>>>>>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
>>>>>>>>>>>>> and other is running these benchmarks (group2). Set the latency nice
>>>>>>>>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
>>>>>>>>>>>>> Total of 96 CPU.
>>>>>>>>>>>>>
>>>>>>>>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
>>>>>>>>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
>>>>>>>>>>>>> and attach the numbers to series.
>>>>>>>>>>>>> Please suggest if any variation in the test i need to try.
>>>>>>>>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
>>>>>>>>>>>> run longer slice to move forward rather than preempting others all the
>>>>>>>>>>>> time
>>>>>>>>>>> hackbench still seems to regress in different latency nice values compared to
>>>>>>>>>>> baseline of 6.2 in this case. up to 50% in some cases.
>>>>>>>>>>>
>>>>>>>>>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
>>>>>>>>>>> running 2 CPU cgroups. No quota assigned.
>>>>>>>>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
>>>>>>>>>>> latency is not changed for this cgroup.
>>>>>>>>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
>>>>>>>>>>> nice values of 0, -20 and 19.
>>>>>>>>>> According to your other emails, you are using the cgroup interface and
>>>>>>>>>> not the task's one. Do I get it right ?
>>>>>>>>> right. I create cgroup, attach bash command with echo $$,
>>>>>>>>> assign the latency nice to cgroup, and run hackbench from that bash prompt.
>>>>>>>>>
>>>>>>>>>> I haven't run test such tests in a cgroup but at least the test with
>>>>>>>>>> latency_nice == 0 should not make any noticeable difference. Does this
>>>>>>>>>> include the re-arrange patch that you have proposed previously ?
>> Ran the test on a different system altogether. I don't see similar regression there. 
>> In fact latency nice is helping in reducing the latency as expected.
>> It is much bigger system with 60 cores. i.e total of 480 cpu.
>>
>> Tested in the same way. created two cgroups. one is running the micro benchmarks
>> and other is running stress-ng at different utilization point.
>> This data is at 50% utilization point. Similar observations w.r.t latency
>> is seen at 0%, 25%, 75% and 100% utilization as well. 
>>
> Thanks for testing on a different system which seems to get results aligned with what
> Prateek and I have seen on our system.
>
>
>> ==========
>> schbench
>> ==========
>> 	       6.2	      6.2 + V12 + LN=0
>> Groups: 1
>> 50.0th:                 14.0             12.5
>> 75.0th:                 16.5             14.0
>> 90.0th:                 18.5             15.5
>> 95.0th:                 20.5             17.0
>> 99.0th:                 27.5             21.0
>> 99.5th:                 36.0             23.5
>> Groups: 2
>> 50.0th:                 14.0             16.0
>> 75.0th:                 17.0             18.0
>> 90.0th:                 20.0             21.0
>> 95.0th:                 23.0             23.0
>> 99.0th:                 71.0             34.0
>> 99.5th:               1170.0             96.0
>> 99.9th:               5088.0           3212.0
>> Groups: 4
>> 50.0th:                 20.5             19.5
>> 75.0th:                 24.5             22.5
>> 90.0th:                 31.0             26.0
>> 95.0th:                260.5             28.0
>> 99.0th:               3644.0             35.0
>> 99.5th:               5152.0             44.5
>> 99.9th:               8076.0            168.5
>> Groups: 8
>> 50.0th:                 26.0             25.5
>> 75.0th:                 32.5             31.5
>> 90.0th:                 41.5             36.5
>> 95.0th:                794.0             39.5
>> 99.0th:               5992.0             66.0
>> 99.5th:               7208.0            159.0
>> 99.9th:               9392.0           1604.0
>> Groups: 16
>> 50.0th:                 37.5             34.0
>> 75.0th:                 49.5             44.5
>> 90.0th:                 70.0             53.5
>> 95.0th:               1284.0             58.5
>> 99.0th:               5600.0            102.5
>> 99.5th:               7216.0            368.5
>> 99.9th:               9328.0           5192.0
>> Groups: 32
>> 50.0th:                 59.0             54.5
>> 75.0th:                 83.0             74.5
>> 90.0th:                118.5             91.0
>> 95.0th:               1921.0            100.5
>> 99.0th:               6672.0            317.0
>> 99.5th:               8252.0           2264.0
>> 99.9th:              10448.0           8388.0
>>
>>
>> ===========
>> hackbench
>> ==========
>>
>> type                 Groups      6.2      | 6.2 + V12 + LN=0
>> Process               10         0.19     |  0.19
>> Process               20         0.34     |  0.34
>> Process               30         0.45     |  0.44
>> Process               40         0.58     |  0.57
>> Process               50         0.70     |  0.69
>> Process               60         0.82     |  0.80
>> thread                10         0.20     |  0.20
>> thread                20         0.36     |  0.36
>> Process(Pipe)         10         0.24     |  0.21
>> Process(Pipe)         20         0.46     |  0.40
>> Process(Pipe)         30         0.65     |  0.58
>> Process(Pipe)         40         0.90     |  0.68
>> Process(Pipe)         50         1.04     |  0.83
>> Process(Pipe)         60         1.16     |  0.86
>> thread(Pipe)          10         0.19     |  0.18
>> thread(Pipe)          20         0.46     |  0.37
>>
>>
> [...]
>
>>>>>>
>>>>>> Do you want me to try any other experiment on this further?
>>>>> Yes, would be good to know which of the 3 changes in the patch create
>>>>> the regression
>>>>>
>>>>> I suspect the 1st change to be the root cause of your problem but It
>>>>> would be good if you can confirm my assumption with some tests
>>>>>
>>>>> Thanks
>>>> Applied each change individually. 3rd change seems to cause the regression.
>>>> Kept only the 3rd change and numbers are same as stable 6.2 for latency nice
>>>> value of 0.
>>> Ok, it's the patch 1 that aims to prevent some unfairness with low weight
>>> waking task. And your platform probably falls in the last part of the commit:
>>>
>>> " Strictly speaking, we should use cfs->min_vruntime instead of
>>> curr->vruntime but it doesn't worth the additional overhead and complexity
>>> as the vruntime of current should be close to min_vruntime if not equal."
>>>
>>> Could you try the patch below on top of v12 ?
>>>
>>> ---
>>>  kernel/sched/fair.c | 21 +++++++++++----------
>>>  1 file changed, 11 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>> index e2aeb4511686..77b03a280912 100644
>>> --- a/kernel/sched/fair.c
>>> +++ b/kernel/sched/fair.c
>>> @@ -5049,7 +5049,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>  }
>>>
>>>  static int
>>> -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se);
>>> +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq);
>>>
>>>  /*
>>>   * Pick the next process, keeping these things in mind, in this order:
>>> @@ -5088,16 +5088,16 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>  				second = curr;
>>>  		}
>>>
>>> -		if (second && wakeup_preempt_entity(second, left) < 1)
>>> +		if (second && wakeup_preempt_entity(second, left, cfs_rq) < 1)
>>>  			se = second;
>>>  	}
>>>
>>> -	if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left) < 1) {
>>> +	if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left, cfs_rq) < 1) {
>>>  		/*
>>>  		 * Someone really wants this to run. If it's not unfair, run it.
>>>  		 */
>>>  		se = cfs_rq->next;
>>> -	} else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left) < 1) {
>>> +	} else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left, cfs_rq) < 1) {
>>>  		/*
>>>  		 * Prefer last buddy, try to return the CPU to a preempted task.
>>>  		 */
>>> @@ -5107,7 +5107,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>  	/* Check for latency sensitive entity waiting for running */
>>>  	latency = __pick_first_latency(cfs_rq);
>>>  	if (latency && (latency != se) &&
>>> -	    wakeup_preempt_entity(latency, se) < 1)
>>> +	    wakeup_preempt_entity(latency, se, cfs_rq) < 1)
>>>  		se = latency;
>>>
>>>  	return se;
>>> @@ -7808,7 +7808,7 @@ static unsigned long wakeup_gran(struct sched_entity *se)
>>>   *
>>>   */
>>>  static int
>>> -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>> +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq)
>>>  {
>>>  	s64 gran, vdiff = curr->vruntime - se->vruntime;
>>>  	s64 offset = wakeup_latency_gran(curr, se);
>>> @@ -7818,6 +7818,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>
>>>  	gran = offset + wakeup_gran(se);
>>>
>>> +	if (vdiff > gran)
>>> +		return 1;
>>>  	/*
>>>  	 * At wake up, the vruntime of a task is capped to not be older than
>>>  	 * a sched_latency period compared to min_vruntime. This prevents long
>>> @@ -7827,9 +7829,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>  	 * for low priority task. Make sure that long sleeping task will get a
>>>  	 * chance to preempt current.
>>>  	 */
>>> -	gran = min_t(s64, gran, get_latency_max());
>>> -
>>> -	if (vdiff > gran)
>>> +	vdiff = cfs_rq->min_vruntime - se->vruntime;
>>> +	if (vdiff > get_latency_max())
>>>  		return 1;
>>>
>>>  	return 0;
>>> @@ -7933,7 +7934,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
>>>  		return;
>>>
>>>  	update_curr(cfs_rq_of(se));
>>> -	if (wakeup_preempt_entity(se, pse) == 1) {
>>> +	if (wakeup_preempt_entity(se, pse, cfs_rq_of(se)) == 1) {
>>>  		/*
>>>  		 * Bias pick_next to pick the sched entity that is
>>>  		 * triggering this preemption.
>>> --
>>> 2.34.1
>> Tried above patch on top of V12. Numbers are worse than V12. We maybe running into
>> a corner case on this system.
> Yes it can be a corner case.
>
> Nevertheless, the patch above has a problem and does an unsigned comparison instead of a signed
> one. I have forced the signed comparison in the patch below to be applied on top  of
> previous one:
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 77b03a280912..22a497f92dbb 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -7830,7 +7830,7 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct
>          * chance to preempt current.
>          */
>         vdiff = cfs_rq->min_vruntime - se->vruntime;
> -       if (vdiff > get_latency_max())
> +       if (vdiff > (s64)get_latency_max())
>                 return 1;
>
>         return 0;

Tested the above patch on top of previous patch + V12. 
Numbers are still worse than V12. Same as V12+previous patch.


>
>
>> Type                Groups       6.2     | 6.2+V12                                 
>>                                                                                    
>>  Process              10        0.33     |  0.44                                   
>>  Process              20        0.61     |  0.90                                   
>>  Process              30        0.87     |  1.29                                   
>>  Process              40        1.10     |  1.69                                   
>>  Process              50        1.34     |  2.08                                   
>>  Process              60        1.58     |  2.39                                   
>>  thread               10        0.36     |  0.53                                   
>>  thread               20        0.64     |  0.94                                   
>>  Process(Pipe)        10        0.18     |  0.46                                   
>>  Process(Pipe)        20        0.32     |  0.75                                   
>>  Process(Pipe)        30        0.42     |  1.01                                   
>>  Process(Pipe)        40        0.56     |  1.15                                   
>>  Process(Pipe)        50        0.68     |  1.38                                   
>>  Process(Pipe)        60        0.80     |  1.56  
>>  
>>
>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>>> index cdcd991bbcf1..c89c201dd164 100644
>>>> --- a/kernel/sched/fair.c
>>>> +++ b/kernel/sched/fair.c
>>>> @@ -7827,7 +7827,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>>          * for low priority task. Make sure that long sleeping task will get a
>>>>          * chance to preempt current.
>>>>          */
>>>> -       gran = min_t(s64, gran, get_latency_max());
>>>>
>>>>         if (vdiff > gran)
>>>>                 return 1;
>>>>
>>>>
>>>>
> [...]
>
>>

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Vincent Guittot]

On Tue, 7 Mar 2023 at 11:52, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>
>
> On 3/7/23 3:49 PM, Vincent Guittot wrote:
> > Le mardi 07 mars 2023 à 00:34:49 (+0530), Shrikanth Hegde a écrit :
> >>> Le lundi 06 mars 2023 à 17:03:30 (+0530), Shrikanth Hegde a écrit :
> >>>> On 3/5/23 6:33 PM, Vincent Guittot wrote:
> >>>>> On Sat, 4 Mar 2023 at 16:13, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>>> On 3/3/23 10:01 PM, Vincent Guittot wrote:
> >>>>>>> Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
> >>>>>>>> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
> >>>>>>>>> On 3/2/23 6:47 PM, Vincent Guittot wrote:
> >>>>>>>>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>>>>>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
> >>>>>>>>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
> >>>>>>>>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
> >>>>>>> [...]
> >>>>>>>
> >>>>>>>>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
> >>>>>>>>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
> >>>>>>>>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
> >>>>>>>>>>>>> and other is running these benchmarks (group2). Set the latency nice
> >>>>>>>>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
> >>>>>>>>>>>>> Total of 96 CPU.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
> >>>>>>>>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
> >>>>>>>>>>>>> and attach the numbers to series.
> >>>>>>>>>>>>> Please suggest if any variation in the test i need to try.
> >>>>>>>>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
> >>>>>>>>>>>> run longer slice to move forward rather than preempting others all the
> >>>>>>>>>>>> time
> >>>>>>>>>>> hackbench still seems to regress in different latency nice values compared to
> >>>>>>>>>>> baseline of 6.2 in this case. up to 50% in some cases.
> >>>>>>>>>>>
> >>>>>>>>>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
> >>>>>>>>>>> running 2 CPU cgroups. No quota assigned.
> >>>>>>>>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
> >>>>>>>>>>> latency is not changed for this cgroup.
> >>>>>>>>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
> >>>>>>>>>>> nice values of 0, -20 and 19.
> >>>>>>>>>> According to your other emails, you are using the cgroup interface and
> >>>>>>>>>> not the task's one. Do I get it right ?
> >>>>>>>>> right. I create cgroup, attach bash command with echo $$,
> >>>>>>>>> assign the latency nice to cgroup, and run hackbench from that bash prompt.
> >>>>>>>>>
> >>>>>>>>>> I haven't run test such tests in a cgroup but at least the test with
> >>>>>>>>>> latency_nice == 0 should not make any noticeable difference. Does this
> >>>>>>>>>> include the re-arrange patch that you have proposed previously ?
> >> Ran the test on a different system altogether. I don't see similar regression there.
> >> In fact latency nice is helping in reducing the latency as expected.
> >> It is much bigger system with 60 cores. i.e total of 480 cpu.
> >>
> >> Tested in the same way. created two cgroups. one is running the micro benchmarks
> >> and other is running stress-ng at different utilization point.
> >> This data is at 50% utilization point. Similar observations w.r.t latency
> >> is seen at 0%, 25%, 75% and 100% utilization as well.
> >>
> > Thanks for testing on a different system which seems to get results aligned with what
> > Prateek and I have seen on our system.
> >
> >
> >> ==========
> >> schbench
> >> ==========
> >>             6.2            6.2 + V12 + LN=0
> >> Groups: 1
> >> 50.0th:                 14.0             12.5
> >> 75.0th:                 16.5             14.0
> >> 90.0th:                 18.5             15.5
> >> 95.0th:                 20.5             17.0
> >> 99.0th:                 27.5             21.0
> >> 99.5th:                 36.0             23.5
> >> Groups: 2
> >> 50.0th:                 14.0             16.0
> >> 75.0th:                 17.0             18.0
> >> 90.0th:                 20.0             21.0
> >> 95.0th:                 23.0             23.0
> >> 99.0th:                 71.0             34.0
> >> 99.5th:               1170.0             96.0
> >> 99.9th:               5088.0           3212.0
> >> Groups: 4
> >> 50.0th:                 20.5             19.5
> >> 75.0th:                 24.5             22.5
> >> 90.0th:                 31.0             26.0
> >> 95.0th:                260.5             28.0
> >> 99.0th:               3644.0             35.0
> >> 99.5th:               5152.0             44.5
> >> 99.9th:               8076.0            168.5
> >> Groups: 8
> >> 50.0th:                 26.0             25.5
> >> 75.0th:                 32.5             31.5
> >> 90.0th:                 41.5             36.5
> >> 95.0th:                794.0             39.5
> >> 99.0th:               5992.0             66.0
> >> 99.5th:               7208.0            159.0
> >> 99.9th:               9392.0           1604.0
> >> Groups: 16
> >> 50.0th:                 37.5             34.0
> >> 75.0th:                 49.5             44.5
> >> 90.0th:                 70.0             53.5
> >> 95.0th:               1284.0             58.5
> >> 99.0th:               5600.0            102.5
> >> 99.5th:               7216.0            368.5
> >> 99.9th:               9328.0           5192.0
> >> Groups: 32
> >> 50.0th:                 59.0             54.5
> >> 75.0th:                 83.0             74.5
> >> 90.0th:                118.5             91.0
> >> 95.0th:               1921.0            100.5
> >> 99.0th:               6672.0            317.0
> >> 99.5th:               8252.0           2264.0
> >> 99.9th:              10448.0           8388.0
> >>
> >>
> >> ===========
> >> hackbench
> >> ==========
> >>
> >> type                 Groups      6.2      | 6.2 + V12 + LN=0
> >> Process               10         0.19     |  0.19
> >> Process               20         0.34     |  0.34
> >> Process               30         0.45     |  0.44
> >> Process               40         0.58     |  0.57
> >> Process               50         0.70     |  0.69
> >> Process               60         0.82     |  0.80
> >> thread                10         0.20     |  0.20
> >> thread                20         0.36     |  0.36
> >> Process(Pipe)         10         0.24     |  0.21
> >> Process(Pipe)         20         0.46     |  0.40
> >> Process(Pipe)         30         0.65     |  0.58
> >> Process(Pipe)         40         0.90     |  0.68
> >> Process(Pipe)         50         1.04     |  0.83
> >> Process(Pipe)         60         1.16     |  0.86
> >> thread(Pipe)          10         0.19     |  0.18
> >> thread(Pipe)          20         0.46     |  0.37
> >>
> >>
> > [...]
> >
> >>>>>>
> >>>>>> Do you want me to try any other experiment on this further?
> >>>>> Yes, would be good to know which of the 3 changes in the patch create
> >>>>> the regression
> >>>>>
> >>>>> I suspect the 1st change to be the root cause of your problem but It
> >>>>> would be good if you can confirm my assumption with some tests
> >>>>>
> >>>>> Thanks
> >>>> Applied each change individually. 3rd change seems to cause the regression.
> >>>> Kept only the 3rd change and numbers are same as stable 6.2 for latency nice
> >>>> value of 0.
> >>> Ok, it's the patch 1 that aims to prevent some unfairness with low weight
> >>> waking task. And your platform probably falls in the last part of the commit:
> >>>
> >>> " Strictly speaking, we should use cfs->min_vruntime instead of
> >>> curr->vruntime but it doesn't worth the additional overhead and complexity
> >>> as the vruntime of current should be close to min_vruntime if not equal."
> >>>
> >>> Could you try the patch below on top of v12 ?
> >>>
> >>> ---
> >>>  kernel/sched/fair.c | 21 +++++++++++----------
> >>>  1 file changed, 11 insertions(+), 10 deletions(-)
> >>>
> >>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >>> index e2aeb4511686..77b03a280912 100644
> >>> --- a/kernel/sched/fair.c
> >>> +++ b/kernel/sched/fair.c
> >>> @@ -5049,7 +5049,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> >>>  }
> >>>
> >>>  static int
> >>> -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se);
> >>> +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq);
> >>>
> >>>  /*
> >>>   * Pick the next process, keeping these things in mind, in this order:
> >>> @@ -5088,16 +5088,16 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>                             second = curr;
> >>>             }
> >>>
> >>> -           if (second && wakeup_preempt_entity(second, left) < 1)
> >>> +           if (second && wakeup_preempt_entity(second, left, cfs_rq) < 1)
> >>>                     se = second;
> >>>     }
> >>>
> >>> -   if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left) < 1) {
> >>> +   if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left, cfs_rq) < 1) {
> >>>             /*
> >>>              * Someone really wants this to run. If it's not unfair, run it.
> >>>              */
> >>>             se = cfs_rq->next;
> >>> -   } else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left) < 1) {
> >>> +   } else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left, cfs_rq) < 1) {
> >>>             /*
> >>>              * Prefer last buddy, try to return the CPU to a preempted task.
> >>>              */
> >>> @@ -5107,7 +5107,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
> >>>     /* Check for latency sensitive entity waiting for running */
> >>>     latency = __pick_first_latency(cfs_rq);
> >>>     if (latency && (latency != se) &&
> >>> -       wakeup_preempt_entity(latency, se) < 1)
> >>> +       wakeup_preempt_entity(latency, se, cfs_rq) < 1)
> >>>             se = latency;
> >>>
> >>>     return se;
> >>> @@ -7808,7 +7808,7 @@ static unsigned long wakeup_gran(struct sched_entity *se)
> >>>   *
> >>>   */
> >>>  static int
> >>> -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >>> +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq)
> >>>  {
> >>>     s64 gran, vdiff = curr->vruntime - se->vruntime;
> >>>     s64 offset = wakeup_latency_gran(curr, se);
> >>> @@ -7818,6 +7818,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >>>
> >>>     gran = offset + wakeup_gran(se);
> >>>
> >>> +   if (vdiff > gran)
> >>> +           return 1;
> >>>     /*
> >>>      * At wake up, the vruntime of a task is capped to not be older than
> >>>      * a sched_latency period compared to min_vruntime. This prevents long
> >>> @@ -7827,9 +7829,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >>>      * for low priority task. Make sure that long sleeping task will get a
> >>>      * chance to preempt current.
> >>>      */
> >>> -   gran = min_t(s64, gran, get_latency_max());
> >>> -
> >>> -   if (vdiff > gran)
> >>> +   vdiff = cfs_rq->min_vruntime - se->vruntime;
> >>> +   if (vdiff > get_latency_max())
> >>>             return 1;
> >>>
> >>>     return 0;
> >>> @@ -7933,7 +7934,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
> >>>             return;
> >>>
> >>>     update_curr(cfs_rq_of(se));
> >>> -   if (wakeup_preempt_entity(se, pse) == 1) {
> >>> +   if (wakeup_preempt_entity(se, pse, cfs_rq_of(se)) == 1) {
> >>>             /*
> >>>              * Bias pick_next to pick the sched entity that is
> >>>              * triggering this preemption.
> >>> --
> >>> 2.34.1
> >> Tried above patch on top of V12. Numbers are worse than V12. We maybe running into
> >> a corner case on this system.
> > Yes it can be a corner case.
> >
> > Nevertheless, the patch above has a problem and does an unsigned comparison instead of a signed
> > one. I have forced the signed comparison in the patch below to be applied on top  of
> > previous one:
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 77b03a280912..22a497f92dbb 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -7830,7 +7830,7 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct
> >          * chance to preempt current.
> >          */
> >         vdiff = cfs_rq->min_vruntime - se->vruntime;
> > -       if (vdiff > get_latency_max())
> > +       if (vdiff > (s64)get_latency_max())
> >                 return 1;
> >
> >         return 0;
>
> Tested the above patch on top of previous patch + V12.
> Numbers are still worse than V12. Same as V12+previous patch.

So It really looks like a corner case for this system and I'm not sure
we can do anything as others don't face he problem
>
>
> >
> >
> >> Type                Groups       6.2     | 6.2+V12
> >>
> >>  Process              10        0.33     |  0.44
> >>  Process              20        0.61     |  0.90
> >>  Process              30        0.87     |  1.29
> >>  Process              40        1.10     |  1.69
> >>  Process              50        1.34     |  2.08
> >>  Process              60        1.58     |  2.39
> >>  thread               10        0.36     |  0.53
> >>  thread               20        0.64     |  0.94
> >>  Process(Pipe)        10        0.18     |  0.46
> >>  Process(Pipe)        20        0.32     |  0.75
> >>  Process(Pipe)        30        0.42     |  1.01
> >>  Process(Pipe)        40        0.56     |  1.15
> >>  Process(Pipe)        50        0.68     |  1.38
> >>  Process(Pipe)        60        0.80     |  1.56
> >>
> >>
> >>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >>>> index cdcd991bbcf1..c89c201dd164 100644
> >>>> --- a/kernel/sched/fair.c
> >>>> +++ b/kernel/sched/fair.c
> >>>> @@ -7827,7 +7827,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
> >>>>          * for low priority task. Make sure that long sleeping task will get a
> >>>>          * chance to preempt current.
> >>>>          */
> >>>> -       gran = min_t(s64, gran, get_latency_max());
> >>>>
> >>>>         if (vdiff > gran)
> >>>>                 return 1;
> >>>>
> >>>>
> >>>>
> > [...]
> >
> >>
>

Re: [PATCH v12 8/8] sched/fair: Add latency list

[Shrikanth Hegde]

> On Tue, 7 Mar 2023 at 11:52, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>
>>
>> On 3/7/23 3:49 PM, Vincent Guittot wrote:
>>> Le mardi 07 mars 2023 à 00:34:49 (+0530), Shrikanth Hegde a écrit :
>>>>> Le lundi 06 mars 2023 à 17:03:30 (+0530), Shrikanth Hegde a écrit :
>>>>>> On 3/5/23 6:33 PM, Vincent Guittot wrote:
>>>>>>> On Sat, 4 Mar 2023 at 16:13, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>>> On 3/3/23 10:01 PM, Vincent Guittot wrote:
>>>>>>>>> Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit :
>>>>>>>>>> On 3/2/23 8:30 PM, Shrikanth Hegde wrote:
>>>>>>>>>>> On 3/2/23 6:47 PM, Vincent Guittot wrote:
>>>>>>>>>>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>>>>>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote:
>>>>>>>>>>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@linux.vnet.ibm.com> wrote:
>>>>>>>>>>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote:
>>>>>>>>> [...]
>>>>>>>>>
>>>>>>>>>>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is
>>>>>>>>>>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of
>>>>>>>>>>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1)
>>>>>>>>>>>>>>> and other is running these benchmarks (group2). Set the latency nice
>>>>>>>>>>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8.
>>>>>>>>>>>>>>> Total of 96 CPU.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems
>>>>>>>>>>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run
>>>>>>>>>>>>>>> and attach the numbers to series.
>>>>>>>>>>>>>>> Please suggest if any variation in the test i need to try.
>>>>>>>>>>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to
>>>>>>>>>>>>>> run longer slice to move forward rather than preempting others all the
>>>>>>>>>>>>>> time
>>>>>>>>>>>>> hackbench still seems to regress in different latency nice values compared to
>>>>>>>>>>>>> baseline of 6.2 in this case. up to 50% in some cases.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 12 core powerpc system  with SMT=8 i.e 96 CPU
>>>>>>>>>>>>> running 2 CPU cgroups. No quota assigned.
>>>>>>>>>>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU.
>>>>>>>>>>>>> latency is not changed for this cgroup.
>>>>>>>>>>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency
>>>>>>>>>>>>> nice values of 0, -20 and 19.
>>>>>>>>>>>> According to your other emails, you are using the cgroup interface and
>>>>>>>>>>>> not the task's one. Do I get it right ?
>>>>>>>>>>> right. I create cgroup, attach bash command with echo $$,
>>>>>>>>>>> assign the latency nice to cgroup, and run hackbench from that bash prompt.
>>>>>>>>>>>
>>>>>>>>>>>> I haven't run test such tests in a cgroup but at least the test with
>>>>>>>>>>>> latency_nice == 0 should not make any noticeable difference. Does this
>>>>>>>>>>>> include the re-arrange patch that you have proposed previously ?
>>>> Ran the test on a different system altogether. I don't see similar regression there.
>>>> In fact latency nice is helping in reducing the latency as expected.
>>>> It is much bigger system with 60 cores. i.e total of 480 cpu.
>>>>
>>>> Tested in the same way. created two cgroups. one is running the micro benchmarks
>>>> and other is running stress-ng at different utilization point.
>>>> This data is at 50% utilization point. Similar observations w.r.t latency
>>>> is seen at 0%, 25%, 75% and 100% utilization as well.
>>>>
>>> Thanks for testing on a different system which seems to get results aligned with what
>>> Prateek and I have seen on our system.
>>>
>>>
>>>> ==========
>>>> schbench
>>>> ==========
>>>>             6.2            6.2 + V12 + LN=0
>>>> Groups: 1
>>>> 50.0th:                 14.0             12.5
>>>> 75.0th:                 16.5             14.0
>>>> 90.0th:                 18.5             15.5
>>>> 95.0th:                 20.5             17.0
>>>> 99.0th:                 27.5             21.0
>>>> 99.5th:                 36.0             23.5
>>>> Groups: 2
>>>> 50.0th:                 14.0             16.0
>>>> 75.0th:                 17.0             18.0
>>>> 90.0th:                 20.0             21.0
>>>> 95.0th:                 23.0             23.0
>>>> 99.0th:                 71.0             34.0
>>>> 99.5th:               1170.0             96.0
>>>> 99.9th:               5088.0           3212.0
>>>> Groups: 4
>>>> 50.0th:                 20.5             19.5
>>>> 75.0th:                 24.5             22.5
>>>> 90.0th:                 31.0             26.0
>>>> 95.0th:                260.5             28.0
>>>> 99.0th:               3644.0             35.0
>>>> 99.5th:               5152.0             44.5
>>>> 99.9th:               8076.0            168.5
>>>> Groups: 8
>>>> 50.0th:                 26.0             25.5
>>>> 75.0th:                 32.5             31.5
>>>> 90.0th:                 41.5             36.5
>>>> 95.0th:                794.0             39.5
>>>> 99.0th:               5992.0             66.0
>>>> 99.5th:               7208.0            159.0
>>>> 99.9th:               9392.0           1604.0
>>>> Groups: 16
>>>> 50.0th:                 37.5             34.0
>>>> 75.0th:                 49.5             44.5
>>>> 90.0th:                 70.0             53.5
>>>> 95.0th:               1284.0             58.5
>>>> 99.0th:               5600.0            102.5
>>>> 99.5th:               7216.0            368.5
>>>> 99.9th:               9328.0           5192.0
>>>> Groups: 32
>>>> 50.0th:                 59.0             54.5
>>>> 75.0th:                 83.0             74.5
>>>> 90.0th:                118.5             91.0
>>>> 95.0th:               1921.0            100.5
>>>> 99.0th:               6672.0            317.0
>>>> 99.5th:               8252.0           2264.0
>>>> 99.9th:              10448.0           8388.0
>>>>
>>>>
>>>> ===========
>>>> hackbench
>>>> ==========
>>>>
>>>> type                 Groups      6.2      | 6.2 + V12 + LN=0
>>>> Process               10         0.19     |  0.19
>>>> Process               20         0.34     |  0.34
>>>> Process               30         0.45     |  0.44
>>>> Process               40         0.58     |  0.57
>>>> Process               50         0.70     |  0.69
>>>> Process               60         0.82     |  0.80
>>>> thread                10         0.20     |  0.20
>>>> thread                20         0.36     |  0.36
>>>> Process(Pipe)         10         0.24     |  0.21
>>>> Process(Pipe)         20         0.46     |  0.40
>>>> Process(Pipe)         30         0.65     |  0.58
>>>> Process(Pipe)         40         0.90     |  0.68
>>>> Process(Pipe)         50         1.04     |  0.83
>>>> Process(Pipe)         60         1.16     |  0.86
>>>> thread(Pipe)          10         0.19     |  0.18
>>>> thread(Pipe)          20         0.46     |  0.37
>>>>
>>>>
>>> [...]
>>>
>>>>>>>>
>>>>>>>> Do you want me to try any other experiment on this further?
>>>>>>> Yes, would be good to know which of the 3 changes in the patch create
>>>>>>> the regression
>>>>>>>
>>>>>>> I suspect the 1st change to be the root cause of your problem but It
>>>>>>> would be good if you can confirm my assumption with some tests
>>>>>>>
>>>>>>> Thanks
>>>>>> Applied each change individually. 3rd change seems to cause the regression.
>>>>>> Kept only the 3rd change and numbers are same as stable 6.2 for latency nice
>>>>>> value of 0.
>>>>> Ok, it's the patch 1 that aims to prevent some unfairness with low weight
>>>>> waking task. And your platform probably falls in the last part of the commit:
>>>>>
>>>>> " Strictly speaking, we should use cfs->min_vruntime instead of
>>>>> curr->vruntime but it doesn't worth the additional overhead and complexity
>>>>> as the vruntime of current should be close to min_vruntime if not equal."
>>>>>
>>>>> Could you try the patch below on top of v12 ?
>>>>>
>>>>> ---
>>>>>  kernel/sched/fair.c | 21 +++++++++++----------
>>>>>  1 file changed, 11 insertions(+), 10 deletions(-)
>>>>>
>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>>>> index e2aeb4511686..77b03a280912 100644
>>>>> --- a/kernel/sched/fair.c
>>>>> +++ b/kernel/sched/fair.c
>>>>> @@ -5049,7 +5049,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>>>  }
>>>>>
>>>>>  static int
>>>>> -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se);
>>>>> +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq);
>>>>>
>>>>>  /*
>>>>>   * Pick the next process, keeping these things in mind, in this order:
>>>>> @@ -5088,16 +5088,16 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>                             second = curr;
>>>>>             }
>>>>>
>>>>> -           if (second && wakeup_preempt_entity(second, left) < 1)
>>>>> +           if (second && wakeup_preempt_entity(second, left, cfs_rq) < 1)
>>>>>                     se = second;
>>>>>     }
>>>>>
>>>>> -   if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left) < 1) {
>>>>> +   if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left, cfs_rq) < 1) {
>>>>>             /*
>>>>>              * Someone really wants this to run. If it's not unfair, run it.
>>>>>              */
>>>>>             se = cfs_rq->next;
>>>>> -   } else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left) < 1) {
>>>>> +   } else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left, cfs_rq) < 1) {
>>>>>             /*
>>>>>              * Prefer last buddy, try to return the CPU to a preempted task.
>>>>>              */
>>>>> @@ -5107,7 +5107,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
>>>>>     /* Check for latency sensitive entity waiting for running */
>>>>>     latency = __pick_first_latency(cfs_rq);
>>>>>     if (latency && (latency != se) &&
>>>>> -       wakeup_preempt_entity(latency, se) < 1)
>>>>> +       wakeup_preempt_entity(latency, se, cfs_rq) < 1)
>>>>>             se = latency;
>>>>>
>>>>>     return se;
>>>>> @@ -7808,7 +7808,7 @@ static unsigned long wakeup_gran(struct sched_entity *se)
>>>>>   *
>>>>>   */
>>>>>  static int
>>>>> -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>>> +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq)
>>>>>  {
>>>>>     s64 gran, vdiff = curr->vruntime - se->vruntime;
>>>>>     s64 offset = wakeup_latency_gran(curr, se);
>>>>> @@ -7818,6 +7818,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>>>
>>>>>     gran = offset + wakeup_gran(se);
>>>>>
>>>>> +   if (vdiff > gran)
>>>>> +           return 1;
>>>>>     /*
>>>>>      * At wake up, the vruntime of a task is capped to not be older than
>>>>>      * a sched_latency period compared to min_vruntime. This prevents long
>>>>> @@ -7827,9 +7829,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>>>      * for low priority task. Make sure that long sleeping task will get a
>>>>>      * chance to preempt current.
>>>>>      */
>>>>> -   gran = min_t(s64, gran, get_latency_max());
>>>>> -
>>>>> -   if (vdiff > gran)
>>>>> +   vdiff = cfs_rq->min_vruntime - se->vruntime;
>>>>> +   if (vdiff > get_latency_max())
>>>>>             return 1;
>>>>>
>>>>>     return 0;
>>>>> @@ -7933,7 +7934,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
>>>>>             return;
>>>>>
>>>>>     update_curr(cfs_rq_of(se));
>>>>> -   if (wakeup_preempt_entity(se, pse) == 1) {
>>>>> +   if (wakeup_preempt_entity(se, pse, cfs_rq_of(se)) == 1) {
>>>>>             /*
>>>>>              * Bias pick_next to pick the sched entity that is
>>>>>              * triggering this preemption.
>>>>> --
>>>>> 2.34.1
>>>> Tried above patch on top of V12. Numbers are worse than V12. We maybe running into
>>>> a corner case on this system.
>>> Yes it can be a corner case.
>>>
>>> Nevertheless, the patch above has a problem and does an unsigned comparison instead of a signed
>>> one. I have forced the signed comparison in the patch below to be applied on top  of
>>> previous one:
>>>
>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>> index 77b03a280912..22a497f92dbb 100644
>>> --- a/kernel/sched/fair.c
>>> +++ b/kernel/sched/fair.c
>>> @@ -7830,7 +7830,7 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct
>>>          * chance to preempt current.
>>>          */
>>>         vdiff = cfs_rq->min_vruntime - se->vruntime;
>>> -       if (vdiff > get_latency_max())
>>> +       if (vdiff > (s64)get_latency_max())
>>>                 return 1;
>>>
>>>         return 0;
>>
>> Tested the above patch on top of previous patch + V12.
>> Numbers are still worse than V12. Same as V12+previous patch.
> 
> So It really looks like a corner case for this system and I'm not sure
> we can do anything as others don't face he problem
>>

Thank you very much. I will try to debug. This would help me in understand the code better.
maybe it could be some tunable. Also it was with cgroupv1. I will switch to cgroupv2 and try. 
I doubt that though.

In case if i find anything, i will update. 

>>
>>>
>>>
>>>> Type                Groups       6.2     | 6.2+V12
>>>>
>>>>  Process              10        0.33     |  0.44
>>>>  Process              20        0.61     |  0.90
>>>>  Process              30        0.87     |  1.29
>>>>  Process              40        1.10     |  1.69
>>>>  Process              50        1.34     |  2.08
>>>>  Process              60        1.58     |  2.39
>>>>  thread               10        0.36     |  0.53
>>>>  thread               20        0.64     |  0.94
>>>>  Process(Pipe)        10        0.18     |  0.46
>>>>  Process(Pipe)        20        0.32     |  0.75
>>>>  Process(Pipe)        30        0.42     |  1.01
>>>>  Process(Pipe)        40        0.56     |  1.15
>>>>  Process(Pipe)        50        0.68     |  1.38
>>>>  Process(Pipe)        60        0.80     |  1.56
>>>>
>>>>
>>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>>>>> index cdcd991bbcf1..c89c201dd164 100644
>>>>>> --- a/kernel/sched/fair.c
>>>>>> +++ b/kernel/sched/fair.c
>>>>>> @@ -7827,7 +7827,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se)
>>>>>>          * for low priority task. Make sure that long sleeping task will get a
>>>>>>          * chance to preempt current.
>>>>>>          */
>>>>>> -       gran = min_t(s64, gran, get_latency_max());
>>>>>>
>>>>>>         if (vdiff > gran)
>>>>>>                 return 1;
>>>>>>
>>>>>>
>>>>>>
>>> [...]
>>>
>>>>
>>