[PATCH v2] cgroup: move cgroup_subsys_state parent field for cache locality

[PATCH v2] cgroup: move cgroup_subsys_state parent field for cache locality

[Todd Poynor]

From: Todd Poynor <toddpoynor@google.com>

Various structures embed a struct cgroup_subsys_state, typically at
the top of the containing structure.  It is common for code that
accesses the structures to perform operations that iterate over the
chain of parent css pointers, also accessing data in each containing
structure.  In particular, struct cpuacct is used by fairly hot code
paths in the scheduler such as cpuacct_charge().

Move the parent css pointer field to the end of the structure to
increase the chances of residing in the same cache line as the data
from the containing structure.

Signed-off-by: Todd Poynor <toddpoynor@google.com>
---
 include/linux/cgroup-defs.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

root_cpuacct fields .cpuusage and. css.parent show up as frequently-
accessed memory in separate cache lines (and usually the only thing
accessed in those cache lines until eviction) in armv8 simulations.
A quick search turned up struct blkcg, struct mem_cgroup, and
struct freezer as other examples using a similar struct layout and
access code.

Instead, could move the parent field to the top of css, and have hot
code paths use __cacheline_aligned with hot data prior to css... or
open to suggestions, thanks.

v2 fixes subject line.

diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index 6a3f850cabab..53c698207ad0 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -106,9 +106,6 @@ struct cgroup_subsys_state {
 	/* reference count - access via css_[try]get() and css_put() */
 	struct percpu_ref refcnt;
 
-	/* PI: the parent css */
-	struct cgroup_subsys_state *parent;
-
 	/* siblings list anchored at the parent's ->children */
 	struct list_head sibling;
 	struct list_head children;
@@ -138,6 +135,12 @@ struct cgroup_subsys_state {
 	/* percpu_ref killing and RCU release */
 	struct rcu_head rcu_head;
 	struct work_struct destroy_work;
+
+	/*
+	 * PI: the parent css.	Placed here for cache proximity to following
+	 * fields of the containing structure.
+	 */
+	struct cgroup_subsys_state *parent;
 };
 
 /*
-- 
2.12.2.715.g7642488e1d-goog

[PATCH v2] cgroup: move cgroup_subsys_state parent field for cache locality

[Todd Poynor]

From: Todd Poynor <toddpoynor-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>

Various structures embed a struct cgroup_subsys_state, typically at
the top of the containing structure.  It is common for code that
accesses the structures to perform operations that iterate over the
chain of parent css pointers, also accessing data in each containing
structure.  In particular, struct cpuacct is used by fairly hot code
paths in the scheduler such as cpuacct_charge().

Move the parent css pointer field to the end of the structure to
increase the chances of residing in the same cache line as the data
from the containing structure.

Signed-off-by: Todd Poynor <toddpoynor-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
---
 include/linux/cgroup-defs.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

root_cpuacct fields .cpuusage and. css.parent show up as frequently-
accessed memory in separate cache lines (and usually the only thing
accessed in those cache lines until eviction) in armv8 simulations.
A quick search turned up struct blkcg, struct mem_cgroup, and
struct freezer as other examples using a similar struct layout and
access code.

Instead, could move the parent field to the top of css, and have hot
code paths use __cacheline_aligned with hot data prior to css... or
open to suggestions, thanks.

v2 fixes subject line.

diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index 6a3f850cabab..53c698207ad0 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -106,9 +106,6 @@ struct cgroup_subsys_state {
 	/* reference count - access via css_[try]get() and css_put() */
 	struct percpu_ref refcnt;
 
-	/* PI: the parent css */
-	struct cgroup_subsys_state *parent;
-
 	/* siblings list anchored at the parent's ->children */
 	struct list_head sibling;
 	struct list_head children;
@@ -138,6 +135,12 @@ struct cgroup_subsys_state {
 	/* percpu_ref killing and RCU release */
 	struct rcu_head rcu_head;
 	struct work_struct destroy_work;
+
+	/*
+	 * PI: the parent css.	Placed here for cache proximity to following
+	 * fields of the containing structure.
+	 */
+	struct cgroup_subsys_state *parent;
 };
 
 /*
-- 
2.12.2.715.g7642488e1d-goog

Re: [PATCH v2] cgroup: move cgroup_subsys_state parent field for cache locality

[Tejun Heo]

Hello,

On Thu, Apr 06, 2017 at 06:47:57PM -0700, Todd Poynor wrote:
> From: Todd Poynor <toddpoynor@google.com>
> 
> Various structures embed a struct cgroup_subsys_state, typically at
> the top of the containing structure.  It is common for code that
> accesses the structures to perform operations that iterate over the
> chain of parent css pointers, also accessing data in each containing
> structure.  In particular, struct cpuacct is used by fairly hot code
> paths in the scheduler such as cpuacct_charge().
> 
> Move the parent css pointer field to the end of the structure to
> increase the chances of residing in the same cache line as the data
> from the containing structure.
> 
> Signed-off-by: Todd Poynor <toddpoynor@google.com>

Applied to cgroup/for-4.12.

Thanks.

-- 
tejun

Re: [PATCH v2] cgroup: move cgroup_subsys_state parent field for cache locality

[Tejun Heo]

Hello,

On Thu, Apr 06, 2017 at 06:47:57PM -0700, Todd Poynor wrote:
> From: Todd Poynor <toddpoynor-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> 
> Various structures embed a struct cgroup_subsys_state, typically at
> the top of the containing structure.  It is common for code that
> accesses the structures to perform operations that iterate over the
> chain of parent css pointers, also accessing data in each containing
> structure.  In particular, struct cpuacct is used by fairly hot code
> paths in the scheduler such as cpuacct_charge().
> 
> Move the parent css pointer field to the end of the structure to
> increase the chances of residing in the same cache line as the data
> from the containing structure.
> 
> Signed-off-by: Todd Poynor <toddpoynor-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>

Applied to cgroup/for-4.12.

Thanks.

-- 
tejun