From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932910AbbFCTMQ (ORCPT ); Wed, 3 Jun 2015 15:12:16 -0400 Received: from mga01.intel.com ([192.55.52.88]:19325 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932800AbbFCTL2 (ORCPT ); Wed, 3 Jun 2015 15:11:28 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.13,548,1427785200"; d="scan'208";a="720238422" From: Vikas Shivappa To: linux-kernel@vger.kernel.org Cc: vikas.shivappa@intel.com, x86@kernel.org, hpa@zytor.com, tglx@linutronix.de, mingo@kernel.org, tj@kernel.org, peterz@infradead.org, matt.fleming@intel.com, will.auld@intel.com, kanaka.d.juvva@intel.com, vikas.shivappa@linux.intel.com Subject: [PATCH 08/10] x86/intel_rdt: Implement scheduling support for Intel RDT Date: Wed, 3 Jun 2015 12:09:59 -0700 Message-Id: <1433358601-20255-9-git-send-email-vikas.shivappa@linux.intel.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1433358601-20255-1-git-send-email-vikas.shivappa@linux.intel.com> References: <1433358601-20255-1-git-send-email-vikas.shivappa@linux.intel.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Adds support for IA32_PQR_ASSOC MSR writes during task scheduling. For Cache Allocation, MSR write would let the task fill in the cache 'subset' represented by the cgroup's cache_mask. The high 32 bits in the per processor MSR IA32_PQR_ASSOC represents the CLOSid. During context switch kernel implements this by writing the CLOSid of the cgroup to which the task belongs to the CPU's IA32_PQR_ASSOC MSR. This patch also implements a common software cache for IA32_PQR_MSR(RMID 0:9, CLOSId 32:63) to be used by both Cache monitoring(CMT) and Cache allocation. CMT updates the RMID where as cache_alloc updates the CLOSid in the software cache. During scheduling when the new RMID/CLOSid value is different from the cached values, IA32_PQR_MSR is updated. Since the measured rdmsr latency for IA32_PQR_MSR is very high(~250 cycles) this software cache is necessary to avoid reading the MSR to compare the current CLOSid value. The following considerations are done for the PQR MSR write so that it minimally impacts scheduler hot path: - This path does not exist on any non-intel platforms. - On Intel platforms, this would not exist by default unless CGROUP_RDT is enabled. - remains a no-op when CGROUP_RDT is enabled and intel SKU does not support the feature. - When feature is available and enabled, never does MSR write till the user manually creates a cgroup directory *and* assigns a cache_mask different from root cgroup directory. Since the child node inherits the parents cache mask , by cgroup creation there is no scheduling hot path impact from the new cgroup. - MSR write is only done when there is a task with different Closid is scheduled on the CPU. Typically if the task groups are bound to be scheduled on a set of CPUs , the number of MSR writes is greatly reduced. - A per CPU cache of CLOSids is maintained to do the check so that we dont have to do a rdmsr which actually costs a lot of cycles. - For cgroup directories having same cache_mask the CLOSids are reused. This minimizes the number of CLOSids used and hence reduces the MSR write frequency. Signed-off-by: Vikas Shivappa --- Changes as per Thomas feedback: -removed the unnecessary wrapper rdt_enabled. -removed unnecessary spin lock and rculock in the scheduling code. -merged all scheduling code into one patch not seperating the RDT common software cache code. arch/x86/include/asm/intel_rdt.h | 42 ++++++++++++++++++++++++++++++ arch/x86/include/asm/rdt_common.h | 18 +++++++++++++ arch/x86/include/asm/switch_to.h | 3 +++ arch/x86/kernel/cpu/intel_rdt.c | 17 ++++++++++++ arch/x86/kernel/cpu/perf_event_intel_cqm.c | 20 +++++--------- 5 files changed, 87 insertions(+), 13 deletions(-) create mode 100644 arch/x86/include/asm/rdt_common.h diff --git a/arch/x86/include/asm/intel_rdt.h b/arch/x86/include/asm/intel_rdt.h index ba4601f..dd1eba2 100644 --- a/arch/x86/include/asm/intel_rdt.h +++ b/arch/x86/include/asm/intel_rdt.h @@ -4,10 +4,16 @@ #ifdef CONFIG_CGROUP_RDT #include +#include + #define MAX_CBM_LENGTH 32 #define IA32_L3_CBM_BASE 0xc90 #define CBM_FROM_INDEX(x) (IA32_L3_CBM_BASE + x) +DECLARE_PER_CPU(struct intel_pqr_state, pqr_state); +extern struct static_key rdt_enable_key; +extern void __intel_rdt_sched_in(void); + struct rdt_subsys_info { unsigned long *closmap; }; @@ -35,5 +41,41 @@ static inline struct intel_rdt *parent_rdt(struct intel_rdt *ir) return css_rdt(ir->css.parent); } +/* + * Return rdt group to which this task belongs. + */ +static inline struct intel_rdt *task_rdt(struct task_struct *task) +{ + return css_rdt(task_css(task, intel_rdt_cgrp_id)); +} + +/* + * intel_rdt_sched_in() - Writes the task's CLOSid to IA32_PQR_MSR + * + * Following considerations are made so that this has minimal impact + * on scheduler hot path: + * - This will stay as no-op unless we are running on an Intel SKU + * which supports L3 cache allocation. + * - When support is present and enabled, does not do any + * IA32_PQR_MSR writes until the user starts really using the feature + * ie creates a rdt cgroup directory and assigns a cache_mask thats + * different from the root cgroup's cache_mask. + * - Caches the per cpu CLOSid values and does the MSR write only + * when a task with a different CLOSid is scheduled in. That + * means the task belongs to a different cgroup. + * - Closids are allocated so that different cgroup directories + * with same cache_mask gets the same CLOSid. This minimizes CLOSids + * used and reduces MSR write frequency. + */ +static inline void intel_rdt_sched_in(void) +{ + if (static_key_false(&rdt_enable_key)) + __intel_rdt_sched_in(); +} + +#else + +static inline void intel_rdt_sched_in(void) {} + #endif #endif diff --git a/arch/x86/include/asm/rdt_common.h b/arch/x86/include/asm/rdt_common.h new file mode 100644 index 0000000..1af7dbc --- /dev/null +++ b/arch/x86/include/asm/rdt_common.h @@ -0,0 +1,18 @@ +#ifndef _X86_RDT_H_ +#define _X86_RDT_H_ + +#define MSR_IA32_PQR_ASSOC 0x0c8f + +/* + * struct intel_pqr_state - Structure to store the IA32_PQR_ASSOC MSR contents. + * @rmid: Resource monitoring Id. PQR has this in least 10 bits. + * @clos: Class of service Id. PQR has this in high 32 bits. + */ +struct intel_pqr_state { + raw_spinlock_t lock; + u32 rmid; + u32 clos; + int cnt; +}; + +#endif diff --git a/arch/x86/include/asm/switch_to.h b/arch/x86/include/asm/switch_to.h index 751bf4b..9149577 100644 --- a/arch/x86/include/asm/switch_to.h +++ b/arch/x86/include/asm/switch_to.h @@ -8,6 +8,9 @@ struct tss_struct; void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p, struct tss_struct *tss); +#include +#define finish_arch_switch(prev) intel_rdt_sched_in() + #ifdef CONFIG_X86_32 #ifdef CONFIG_CC_STACKPROTECTOR diff --git a/arch/x86/kernel/cpu/intel_rdt.c b/arch/x86/kernel/cpu/intel_rdt.c index f857381..510de67 100644 --- a/arch/x86/kernel/cpu/intel_rdt.c +++ b/arch/x86/kernel/cpu/intel_rdt.c @@ -34,6 +34,8 @@ static struct clos_cbm_map *ccmap; static struct rdt_subsys_info rdtss_info; static DEFINE_MUTEX(rdt_group_mutex); struct intel_rdt rdt_root_group; +struct static_key __read_mostly rdt_enable_key = STATIC_KEY_INIT_FALSE; + /* * Mask of CPUs for writing CBM values. We only need one CPU per-socket. */ @@ -87,6 +89,20 @@ static inline void clos_put(unsigned int closid) clos_free(closid); } +void __intel_rdt_sched_in(void) +{ + struct intel_pqr_state *state = this_cpu_ptr(&pqr_state); + struct task_struct *task = current; + struct intel_rdt *ir; + + ir = task_rdt(task); + if (ir->clos == state->clos) + return; + + wrmsr(MSR_IA32_PQR_ASSOC, state->rmid, ir->clos); + state->clos = ir->clos; +} + static struct cgroup_subsys_state * intel_rdt_css_alloc(struct cgroup_subsys_state *parent_css) { @@ -339,6 +355,7 @@ static int __init intel_rdt_late_init(void) for_each_online_cpu(i) rdt_cpumask_update(i); + static_key_slow_inc(&rdt_enable_key); pr_info("Intel cache allocation enabled\n"); out_err: diff --git a/arch/x86/kernel/cpu/perf_event_intel_cqm.c b/arch/x86/kernel/cpu/perf_event_intel_cqm.c index d43c498..7220a51 100644 --- a/arch/x86/kernel/cpu/perf_event_intel_cqm.c +++ b/arch/x86/kernel/cpu/perf_event_intel_cqm.c @@ -7,22 +7,16 @@ #include #include #include +#include #include "perf_event.h" -#define MSR_IA32_PQR_ASSOC 0x0c8f #define MSR_IA32_QM_CTR 0x0c8e #define MSR_IA32_QM_EVTSEL 0x0c8d static unsigned int cqm_max_rmid = -1; static unsigned int cqm_l3_scale; /* supposedly cacheline size */ -struct intel_cqm_state { - raw_spinlock_t lock; - int rmid; - int cnt; -}; - -static DEFINE_PER_CPU(struct intel_cqm_state, cqm_state); +DEFINE_PER_CPU(struct intel_pqr_state, pqr_state); /* * Protects cache_cgroups and cqm_rmid_free_lru and cqm_rmid_limbo_lru. @@ -961,7 +955,7 @@ out: static void intel_cqm_event_start(struct perf_event *event, int mode) { - struct intel_cqm_state *state = this_cpu_ptr(&cqm_state); + struct intel_pqr_state *state = this_cpu_ptr(&pqr_state); unsigned int rmid = event->hw.cqm_rmid; unsigned long flags; @@ -978,14 +972,14 @@ static void intel_cqm_event_start(struct perf_event *event, int mode) WARN_ON_ONCE(state->rmid); state->rmid = rmid; - wrmsrl(MSR_IA32_PQR_ASSOC, state->rmid); + wrmsr(MSR_IA32_PQR_ASSOC, state->rmid, state->clos); raw_spin_unlock_irqrestore(&state->lock, flags); } static void intel_cqm_event_stop(struct perf_event *event, int mode) { - struct intel_cqm_state *state = this_cpu_ptr(&cqm_state); + struct intel_pqr_state *state = this_cpu_ptr(&pqr_state); unsigned long flags; if (event->hw.cqm_state & PERF_HES_STOPPED) @@ -998,7 +992,7 @@ static void intel_cqm_event_stop(struct perf_event *event, int mode) if (!--state->cnt) { state->rmid = 0; - wrmsrl(MSR_IA32_PQR_ASSOC, 0); + wrmsr(MSR_IA32_PQR_ASSOC, 0, state->clos); } else { WARN_ON_ONCE(!state->rmid); } @@ -1243,7 +1237,7 @@ static inline void cqm_pick_event_reader(int cpu) static void intel_cqm_cpu_prepare(unsigned int cpu) { - struct intel_cqm_state *state = &per_cpu(cqm_state, cpu); + struct intel_pqr_state *state = &per_cpu(pqr_state, cpu); struct cpuinfo_x86 *c = &cpu_data(cpu); raw_spin_lock_init(&state->lock); -- 1.9.1