* [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c
@ 2017-06-14 1:12 Yi Sun
2017-06-14 1:12 ` [PATCH v12 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
` (22 more replies)
0 siblings, 23 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Hi all,
We plan to bring a new PSR (Platform Shared Resource) feature called
Intel L2 Cache Allocation Technology (L2 CAT) to Xen. It has been enabled
in Linux Kernel.
Besides the L2 CAT implementaion, we refactor the psr.c to make it more
flexible and easily to extend to add new features. We abstract the general
operations of all features and encapsulate them into a structure. Then,
the development of new feature is simple to mainly implement these callback
functions.
The patch set can be found at:
https://github.com/yisun-git/xen.git l2_cat_v12
---
Acked and Reviewed list before V12:
a - Acked-by
r - Reviewed-by
r patch 1 - docs: create Cache Allocation Technology (CAT) and Code and
Data Prioritization (CDP) feature document
a patch 2 - x86: move cpuid_count_leaf from cpuid.c to processor.h.
ar patch 3 - x86: refactor psr: remove L3 CAT/CDP codes.
r patch 6 - x86: refactor psr: L3 CAT: implement get hw info flow.
r patch 18 - x86: L2 CAT: implement get value flow.
r patch 19 - x86: L2 CAT: implement set value flow.
a patch 20 - tools: L2 CAT: support get HW info for L2 CAT.
a patch 21 - tools: L2 CAT: support show cbm for L2 CAT.
a patch 22 - tools: L2 CAT: support set cbm for L2 CAT.
a patch 23 - docs: add L2 CAT description in docs.
---
V12 change list:
Patch 4:
- add comment for 'feat_init'.
(suggested by Jan Beulich)
- use 'ARRAY_SIZE()' to check array boundary.
(suggested by Jan Beulich)
- use 'XENLOG_WARNING' for error message.
(suggested by Jan Beulich)
- move 'type[]' assignment for feat_props object into its delcaration
to make the object be 'const'.
(suggested by Jan Beulich)
- remove "L3" and "L2" indication in printk.
(suggested by Jan Beulich)
- move position of 'ref_lock' definition.
Patch 5:
- fix coding style issue.
(suggested by Jan Beulich)
- in 'get_max_cos_max', invert the condition and dropping 'continue'.
(suggested by Jan Beulich)
- modify 'psr_assoc_cos' to directly return 'reg' value.
(suggested by Jan Beulich)
- modify code style.
(suggested by Jan Beulich)
- do not break domain creation if 'psr_cos_ids' allocation fails.
(suggested by Jan Beulich)
Patch 6:
- use 'ASSERT_UNREACHABLE()' to record bug.
(suggested by Jan Beulich)
Patch 7:
- remove 'psr_get_feat_and_type' and implement its functionality in
'psr_get_val'.
- use 'ASSERT_UNREACHABLE' to record error.
(suggested by Jan Beulich)
- return success only if the 'val' is set in 'psr_get_val'.
(suggested by Jan Beulich)
Patch 8:
- remove the memebers position changes in 'psr_socket_info'.
(suggested by Jan Beulich)
- rename 'dom_ids' to 'dom_set'.
(suggested by Jan Beulich)
- call 'bitmap_zero' to clear bitmap.
(suggested by Jan Beulich)
- combine two if()-s in 'psr_ctxt_switch_to' and add comment.
(suggested by Jan Beulich)
- remove redundant check in 'psr_get_val'.
(suggested by Jan Beulich)
- use 'domain_lock()' to protect 'psr_cos_ids' so that the codes do not
depend on 'domctl_lock'.
(suggested by Jan Beulich)
- adjust codes to avoid cast in 'psr_set_val'.
(suggested by Jan Beulich)
- adjust codes to avoid duplication of error paths in 'psr_set_val'.
(suggested by Jan Beulich)
- clear the domain bit anyway in 'psr_free_cos'.
(suggested by Jan Beulich)
- use the default cos id when getting old_cos in 'psr_set_val'.
Patch 9:
- use 'ARRAY_SIZE' to calculate array boundary.
(suggested by Jan Beulich)
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exist but props does not exist.
(suggested by Jan Beulich)
- return 0 only when value is set in 'insert_val_into_array'.
(suggested by Jan Beulich)
Patch 10:
- use 'ARRAY_SIZE' to calculate array boundary.
(suggested by Jan Beulich)
- move same type variables declaration to one line.
(suggested by Jan Beulich)
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exists but props does not exist.
(suggested by Jan Beulich)
- move 'i' definition into the loop to limit its scope.
(suggested by Jan Beulich)
Patch 11:
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exists but props does not exist.
(suggested by Jan Beulich)
- declare same type vairables in one line.
(suggested by Jan Beulich)
- use 'ARRAY_SIZE' to calculate array boundary.
(suggested by Jan Beulich)
Patch 12:
- declare same type varaibles in one line.
(suggested by Jan Beulich)
- replace 'feat_type' to 'props' in 'struct cos_write_info'.
(suggested by Jan Beulich)
- assign the 'cos_num' to a local variable.
(suggested by Jan Beulich)
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exists but props does not exist.
(suggested by Jan Beulich)
Patch 13:
- move 'type[]' assignment into l3_cdp_props declaration to make it be
'const'.
(suggested by Jan Beulich)
- remove "L2 CAT" indication in printk.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- change 'val' type to uint64_t.
(suggested by Jan Beulich)
- use 1ull.
(suggested by Jan Beulich)
- restore mask(0) MSR to default value.
(suggested by Jan Beulich)
Patch 14:
- remove input parameter 'info' in 'psr_cbm_type_to_feat_type'. Use
'feat_props' array to check.
(suggested by Jan Beulich)
Patch 15:
- add comment to explain how to deal with the case that user set new val
for both DATA and CODE at same time.
- add parameter for 'psr_cbm_type_to_feat_type' to return the feature type
according to it.
- use the feature type returned by 'psr_cbm_type_to_feat_type' to check
if we need insert the new value into all items of the feature value array.
- use conditional expression for wrmsrl.
(suggested by Jan Beulich)
Patch 16:
- move 'type[]' assignment into l2_cat_props declaration to make it be
'const'.
(suggested by Jan Beulich)
- add "L2 CAT" indicator in printk.
(suggested by Jan Beulich)
- restore mask(0) MSR to default value.
(suggested by Jan Beulich)
Yi Sun (23):
docs: create Cache Allocation Technology (CAT) and Code and Data
Prioritization (CDP) feature document
x86: move cpuid_count_leaf from cpuid.c to processor.h.
x86: refactor psr: remove L3 CAT/CDP codes.
x86: refactor psr: L3 CAT: implement main data structures, CPU init
and free flows.
x86: refactor psr: L3 CAT: implement Domain init/free and schedule
flows.
x86: refactor psr: L3 CAT: implement get hw info flow.
x86: refactor psr: L3 CAT: implement get value flow.
x86: refactor psr: L3 CAT: set value: implement framework.
x86: refactor psr: L3 CAT: set value: assemble features value array.
x86: refactor psr: L3 CAT: set value: implement cos finding flow.
x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
x86: refactor psr: L3 CAT: set value: implement write msr flow.
x86: refactor psr: CDP: implement CPU init flow.
x86: refactor psr: CDP: implement get hw info flow.
x86: refactor psr: CDP: implement set value callback function.
x86: L2 CAT: implement CPU init flow.
x86: L2 CAT: implement get hw info flow.
x86: L2 CAT: implement get value flow.
x86: L2 CAT: implement set value flow.
tools: L2 CAT: support get HW info for L2 CAT.
tools: L2 CAT: support show cbm for L2 CAT.
tools: L2 CAT: support set cbm for L2 CAT.
docs: add L2 CAT description in docs.
docs/features/intel_psr_cat_cdp.pandoc | 461 +++++++++++
docs/man/xl.pod.1.in | 25 +-
docs/misc/xl-psr.markdown | 18 +-
tools/libxc/include/xenctrl.h | 7 +-
tools/libxc/xc_psr.c | 45 +-
tools/libxl/libxl.h | 9 +
tools/libxl/libxl_psr.c | 28 +-
tools/libxl/libxl_types.idl | 1 +
tools/xl/xl_cmdtable.c | 6 +-
tools/xl/xl_psr.c | 168 ++--
xen/arch/x86/cpuid.c | 6 -
xen/arch/x86/domain.c | 3 +-
xen/arch/x86/domctl.c | 51 +-
xen/arch/x86/psr.c | 1411 ++++++++++++++++++++++++--------
xen/arch/x86/sysctl.c | 38 +-
xen/include/asm-x86/msr-index.h | 1 +
xen/include/asm-x86/processor.h | 7 +
xen/include/asm-x86/psr.h | 28 +-
xen/include/public/domctl.h | 2 +
xen/include/public/sysctl.h | 3 +-
20 files changed, 1876 insertions(+), 442 deletions(-)
create mode 100644 docs/features/intel_psr_cat_cdp.pandoc
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* [PATCH v12 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
` (21 subsequent siblings)
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch creates CAT and CDP feature document in doc/features/. It describes
key points to implement L3 CAT/CDP and L2 CAT which is described in details in
Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES".
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v11:
- modify data structure descriptions according to code changes.
- modify revision.
v9:
- add 'CMT' to the list of acronyms.
(suggested by Wei Liu)
- change feature list to feature array.
- modify data structure descriptions according to latest codes.
- modify revision.
v8:
- change revision info.
(suggested by Konrad Rzeszutek Wilk)
- add content int 'Areas for improvement'.
(suggested by Konrad Rzeszutek Wilk)
v7:
- correct typo.
(suggested by Konrad Rzeszutek Wilk)
- replace application/VM to domain.
(suggested by Konrad Rzeszutek Wilk)
- amend description of `feat_mask` to make it clearer.
(suggested by Konrad Rzeszutek Wilk)
- update revision.
(suggested by Konrad Rzeszutek Wilk)
- other minor fixes.
(suggested by Konrad Rzeszutek Wilk)
v6:
- write a new feature document to cover L3 CAT/CDP and L2 CAT.
(suggested by Kevin Tian)
- adjust 'Terminology' position in document.
(suggested by Dario Faggioli)
- fix wordings.
(suggested by Dario Faggioli, Kevin Tian and Konrad Rzeszutek Wilk)
- add SDM chapter title in commit message.
(suggested by Konrad Rzeszutek Wilk)
- add more explanations.
(suggested by Kevin Tian)
v4:
- change file name to be more descriptive, 'intel_psr_l2_cat.pandoc'.
(suggested by Dario Faggioli)
- remove 'Ext' and 'New' prefixes.
(suggested by Dario Faggioli)
- remove change log in Revison part.
(suggested by Dario Faggioli)
- adjust Xen release number to 4.9 to show this feature targets 4.9.
(suggested by Dario Faggioli)
- provide 'Terminology' and more sections.
(suggested by Dario Faggioli)
- fix wordings.
(suggested by Konrad Rzeszutek Wilk)
- remove chapter number.
(suggested by Konrad Rzeszutek Wilk)
v3:
- make design document be a patch.
(suggested by Konrad Rzeszutek Wilk)
v2:
- provide chapter for the L2 CAT.
(suggested by Meng Xu)
---
---
docs/features/intel_psr_cat_cdp.pandoc | 461 +++++++++++++++++++++++++++++++++
1 file changed, 461 insertions(+)
create mode 100644 docs/features/intel_psr_cat_cdp.pandoc
diff --git a/docs/features/intel_psr_cat_cdp.pandoc b/docs/features/intel_psr_cat_cdp.pandoc
new file mode 100644
index 0000000..79b4739
--- /dev/null
+++ b/docs/features/intel_psr_cat_cdp.pandoc
@@ -0,0 +1,461 @@
+% Intel Cache Allocation Technology and Code and Data Prioritization Features
+% Revision 1.11
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+ Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+ Component(s): Hypervisor, toolstack
+
+ Hardware: L3 CAT: Haswell and beyond CPUs
+ CDP : Broadwell and beyond CPUs
+ L2 CAT: Atom codename Goldmont and beyond CPUs
+---------------- ----------------------------------------------------
+
+# Terminology
+
+* CAT Cache Allocation Technology
+* CBM Capacity BitMasks
+* CDP Code and Data Prioritization
+* CMT Cache Monitoring Technology
+* COS/CLOS Class of Service
+* MSRs Machine Specific Registers
+* PSR Intel Platform Shared Resource
+
+# Overview
+
+Intel provides a set of allocation capabilities including Cache Allocatation
+Technology (CAT) and Code and Data Prioritization (CDP).
+
+CAT allows an OS or hypervisor to control allocation of a CPU's shared cache
+based on application/domain priority or Class of Service (COS). Each COS is
+configured using capacity bitmasks (CBMs) which represent cache capacity and
+indicate the degree of overlap and isolation between classes. Once CAT is co-
+nfigured, the processor allows access to portions of cache according to the
+established COS. Intel Xeon processor E5 v4 family (and some others) introduce
+capabilities to configure and make use of the CAT mechanism on the L3 cache.
+Intel Goldmont processor provides support for control over the L2 cache.
+
+Code and Data Prioritization (CDP) Technology is an extension of CAT. CDP
+enables isolation and separate prioritization of code and data fetches to
+the L3 cache in a SW configurable manner, which can enable workload priorit-
+ization and tuning of cache capacity to the characteristics of the workload.
+CDP extends CAT by providing separate code and data masks per Class of Service
+(COS). When SW configures to enable CDP, L3 CAT is disabled.
+
+# User details
+
+* Feature Enabling:
+
+ Add "psr=cat" to boot line parameter to enable all supported level CAT featu-
+ res. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW.
+
+* xl interfaces:
+
+ 1. `psr-cat-show [OPTIONS] domain-id`:
+
+ Show L2 CAT or L3 CAT/CDP CBM of the domain designated by Xen domain-id.
+
+ Option `-l`:
+ `-l2`: Show cbm for L2 cache.
+ `-l3`: Show cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ 2. `psr-cat-set [OPTIONS] domain-id cbm`:
+
+ Set L2 CAT or L3 CAT/CDP CBM to the domain designated by Xen domain-id.
+
+ Option `-s`: Specify the socket to process, otherwise all sockets are
+ processed.
+
+ Option `-l`:
+ `-l2`: Specify cbm for L2 cache.
+ `-l3`: Specify cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ Option `-c` or `-d`:
+ `-c`: Set L3 CDP code cbm.
+ `-d`: Set L3 CDP data cbm.
+
+ 3. `psr-hwinfo [OPTIONS]`:
+
+ Show CMT & L2 CAT & L3 CAT/CDP HW information on every socket.
+
+ Option `-m, --cmt`: Show Cache Monitoring Technology (CMT) hardware info.
+
+ Option `-a, --cat`: Show CAT/CDP hardware info.
+
+# Technical details
+
+L3 CAT/CDP and L2 CAT are all members of Intel PSR features, they share the base
+PSR infrastructure in Xen.
+
+## Hardware perspective
+
+ CAT/CDP defines a range of MSRs to assign different cache access patterns
+ which are known as CBMs, each CBM is associated with a COS.
+
+ ```
+ E.g. L2 CAT:
+ +----------------------------+----------------+
+ IA32_PQR_ASSOC | MSR (per socket) | Address |
+ +----+---+-------+ +----------------------------+----------------+
+ | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
+ +----+---+-------+ +----------------------------+----------------+
+ └-------------> | ... | ... |
+ +----------------------------+----------------+
+ | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
+ +----------------------------+----------------+
+ ```
+
+ L3 CAT/CDP uses a range of MSRs from 0xC90 ~ 0xC90+n (n<128).
+
+ L2 CAT uses a range of MSRs from 0xD10 ~ 0xD10+n (n<64), following the L3
+ CAT/CDP MSRs, setting different L2 cache accessing patterns from L3 cache is
+ supported.
+
+ Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the
+ hardware indicating the cache space a domain should be limited to as well as
+ providing an indication of overlap and isolation in the CAT-capable cache from
+ other domains contending for the cache.
+
+ Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
+ note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
+ 0FF0H, 003CH, etc.).
+
+ ```
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Default Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Overlapped Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | | | | | Isolated Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | | |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+ ```
+
+ We can get the CBM length through CPUID. The default value of CBM is calcul-
+ ated by `(1ull << cbm_len) - 1`. That is a fully open bitmask, all ones bitm-
+ ask. The COS[0] always stores the default value without change.
+
+ There is a `IA32_PQR_ASSOC` register which stores the COS ID of the VCPU. HW
+ enforces cache allocation according to the corresponding CBM.
+
+## The relationship between L3 CAT/CDP and L2 CAT
+
+ HW may support all features. By default, CDP is disabled on the processor.
+ If the L3 CAT MSRs are used without enabling CDP, the processor operates in
+ a traditional CAT-only mode. When CDP is enabled:
+ * the CAT mask MSRs are re-mapped into interleaved pairs of mask MSRs for
+ data or code fetches.
+ * the range of COS for CAT is re-indexed, with the lower-half of the COS
+ range available for CDP.
+
+ L2 CAT is independent of L3 CAT/CDP, which means L2 CAT can be enabled while
+ L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are both enabled.
+
+ As a requirement, the bits of CBM of CAT/CDP must be continuous.
+
+ N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same associate
+ register `IA32_PQR_ASSOC`, which means one COS is associated with a pair of
+ L2 CAT CBM and L3 CAT/CDP CBM.
+
+ Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or other
+ PSR features in future). In some cases, a domain is permitted to have a COS
+ that is beyond one (or more) of PSR features but within the others. For
+ instance, let's assume the max COS of L2 CAT is 8 but the max COS of L3
+ CAT is 16, when a domain is assigned 9 as COS, the L3 CAT CBM associated to
+ COS 9 would be enforced, but for L2 CAT, the HW works as default value is
+ set since COS 9 is beyond the max COS (8) of L2 CAT.
+
+## Design Overview
+
+* Core COS/CBM association
+
+ When enforcing CAT/CDP, all cores of domains have the same default COS (COS0)
+ which is associated with the fully open CBM (all ones bitmask) to access all
+ cache. The default COS is used only in hypervisor and is transparent to tool
+ stack and user.
+
+ System administrator can change PSR allocation policy at runtime by tool stack.
+ Since L2 CAT shares COS with L3 CAT/CDP, a COS corresponds to a 2-tuple, like
+ [L2 CBM, L3 CBM] with only-CAT enabled, when CDP is enabled, one COS correspo-
+ nds to a 3-tuple, like [L2 CBM, L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT
+ nor L3 CDP is enabled, things would be easier, one COS corresponds to one L2
+ CBM.
+
+* VCPU schedule
+
+ When context switch happens, the COS of VCPU is written to per-thread MSR
+ `IA32_PQR_ASSOC`, and then hardware enforces cache allocation according to
+ the corresponding CBM.
+
+* Multi-sockets
+
+ Different sockets may have different CAT/CDP capability (e.g. max COS) alth-
+ ough it is consistent on the same socket. So the capability of per-socket CAT/
+ CDP is specified.
+
+ 'psr-cat-set' can set CBM for one domain per socket. On each socket, we main-
+ tain a COS array for all domains. One domain uses one COS at one time. One COS
+ stores the CBM of the domain to work. So, when a VCPU of the domain is migrat-
+ ed from socket 1 to socket 2, it follows configuration on socket 2.
+
+ E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets do-
+ main 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
+ is migrated from socket 1 to 2, the COS ID used is 7, that means 0x3f is the
+ CBM to work for this domain 1 now.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+ 1. Boot line parameter "psr=cat" enables L2 CAT and L3 CAT if hardware suppo-
+ rted. "psr=cdp" enables CDP if hardware supported.
+
+ 2. SYSCTL:
+ - XEN_SYSCTL_PSR_CAT_get_l3_info: Get L3 CAT/CDP information.
+ - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
+
+ 3. DOMCTL:
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: Get L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: Set L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE: Get CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: Set CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA: Get CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: Set CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
+
+* xl interfaces:
+
+ 1. psr-cat-show -lX domain-id
+ Show LX cbm for a domain.
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
+
+ 2. psr-cat-set -lX domain-id cbm
+ Set LX cbm for a domain.
+ => XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
+
+ 3. psr-hwinfo
+ Show PSR HW information, including L3 CAT/CDP/L2 CAT
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info
+
+* Key data structure:
+
+ 1. Feature properties
+
+ ```
+ static const struct feat_props {
+ unsigned int cos_num;
+ enum cbm_type type[PSR_MAX_COS_NUM];
+ bool (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len);
+ void (*write_msr)(unsigned int cos, uint32_t val, enum cbm_type type);
+ } *feat_props[PSR_SOCKET_FEAT_NUM];
+ ```
+
+ Every feature has its own properties, e.g. some data and actions. A
+ feature property pointer array is declared to save every feature's
+ properties.
+
+ - Member `cos_num`
+
+ `cos_num` is the number of COS registers the feature uses, e.g. L3/L2
+ CAT uses 1 register but CDP uses 2 registers.
+
+ - Member `type`
+
+ `type` is an array to save all 'enum cbm_type' values of the feature.
+ It is used with cos_num together to get/write a feature's COS registers
+ values one by one.
+
+ - Member `get_feat_info`
+
+ `get_feat_info` is used to return feature HW info through sysctl.
+
+ - Member `write_msr`
+
+ `write_msr` is used to write out feature MSR register.
+
+ 2. Feature node
+
+ ```
+ struct feat_node {
+ unsigned int cos_max;
+ unsigned int cbm_len;
+ uint32_t cos_reg_val[MAX_COS_REG_CNT];
+ };
+ ```
+
+ When a PSR enforcement feature is enabled, it will be added into a
+ feature array.
+
+ - Member `cos_max`
+
+ `cos_max` is one of the hardware info of CAT. It means the max number
+ of COS registers. As L3 CAT/CDP/L2 CAT all have it, it is declared in
+ `feat_node`.
+
+ - Member `cbm_len`
+
+ `cbm_len` is one of the hardware info of CAT. It means the max number
+ of bits to set.
+
+ - Member `cos_reg_val`
+
+ `cos_reg_val` is an array to maintain the value set in all COS registers
+ of the feature. The array is indexed by COS ID.
+
+ 3. Per-socket PSR features information structure
+
+ ```
+ struct psr_socket_info {
+ bool feat_init;
+ struct feat_node *features[PSR_SOCKET_FEAT_NUM];
+ spinlock_t ref_lock;
+ unsigned int cos_ref[MAX_COS_REG_CNT];
+ DECLARE_BITMAP(dom_ids, DOMID_IDLE + 1);
+ };
+ ```
+
+ We collect all PSR allocation features information of a socket in this
+ `struct psr_socket_info`.
+
+ - Member `feat_init`
+
+ `feat_init` is a flag, to indicate whether the CPU init on a socket
+ has been done.
+
+ - Member `features`
+
+ `features` is a pointer array to save all enabled features poniters
+ according to feature position defined in `enum psr_feat_type`.
+
+ - Member `ref_lock`
+
+ `ref_lock` is a spin lock to protect `cos_ref`.
+
+ - Member `cos_ref`
+
+ `cos_ref` is an array which maintains the reference of one COS. It maps
+ to cos_reg_val[MAX_COS_REG_NUM] in `struct feat_node`. If one COS is
+ used by one domain, the corresponding reference will increase by one. If
+ a domain releases the COS, the reference will decrease by one. The array
+ is indexed by COS ID.
+
+ - Member `dom_ids`
+
+ `dom_ids` is a bitmap, every bit corresponds to a domain. Index is
+ domain_id. It is used to help restore domain_id to 0 when a socket is
+ offline and then online again.
+
+# Limitations
+
+CAT/CDP can only work on HW which enables it(check by CPUID). So far, there is
+no HW which enables both L2 CAT and L3 CAT/CDP. But SW implementation has cons-
+idered such scenario to enable both L2 CAT and L3 CAT/CDP.
+
+# Testing
+
+We can execute above xl commands to verify L2 CAT and L3 CAT/CDP on different
+HWs support them.
+
+For example:
+ root@:~$ xl psr-hwinfo --cat
+ Cache Allocation Technology (CAT): L2
+ Socket ID : 0
+ Maximum COS : 3
+ CBM length : 8
+ Default CBM : 0xff
+
+ root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
+
+ root@:~$ xl psr-cat-show -l2 1
+ Socket ID : 0
+ Default CBM : 0xff
+ ID NAME CBM
+ 1 ubuntu14 0x7f
+
+# Areas for improvement
+
+A hexadecimal number is used to set/show CBM for a domain now. Although this
+is convenient to cover overlap/isolated bitmask requirement, it is not user-
+friendly.
+
+To improve this, the libxl interfaces can be wrapped in libvirt to provide more
+usr-friendly interfaces to user, e.g. a percentage number of the cache to set
+and show.
+
+# Known issues
+
+N/A
+
+# References
+
+"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date Revision Version Notes
+---------- -------- -------- -------------------------------------------
+2016-08-12 1.0 Xen 4.9 Design document written
+2017-02-13 1.7 Xen 4.9 Changes:
+ 1. Modify the design document to cover L3
+ CAT/CDP and L2 CAT;
+ 2. Fix typos;
+ 3. Amend description of `feat_mask` to make
+ it clearer;
+ 4. Other minor changes.
+2017-02-15 1.8 Xen 4.9 Changes:
+ 1. Add content in 'Areas for improvement';
+ 2. Adjust revision number.
+2017-03-16 1.9 Xen 4.9 Changes:
+ 1. Add 'CMT' in 'Terminology';
+ 2. Change 'feature list' to 'feature array'.
+ 3. Modify data structure descriptions.
+ 4. Adjust revision number.
+2017-05-03 1.11 Xen 4.9 Changes:
+ 1. Modify data structure descriptions.
+ 2. Adjust revision number.
+---------- -------- -------- -------------------------------------------
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-06-14 1:12 ` [PATCH v12 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 03/23] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
` (20 subsequent siblings)
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch moves 'cpuid_count_leaf' from cpuid.c to processor.h to
make it available to external codes.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
v9:
- create this patch alone to move 'cpuid_count_leaf'.
(suggested by Wei Liu)
v6:
- use 'struct cpuid_leaf' in psr.c. So we have to access 'cpuid_count_leaf'
which has to be moved to processor.h.
(suggested by Andrew Cooper)
---
---
xen/arch/x86/cpuid.c | 6 ------
xen/include/asm-x86/processor.h | 7 +++++++
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/cpuid.c b/xen/arch/x86/cpuid.c
index 1c6a6c6..a4a0c4c 100644
--- a/xen/arch/x86/cpuid.c
+++ b/xen/arch/x86/cpuid.c
@@ -35,12 +35,6 @@ static void cpuid_leaf(uint32_t leaf, struct cpuid_leaf *data)
cpuid(leaf, &data->a, &data->b, &data->c, &data->d);
}
-static void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
- struct cpuid_leaf *data)
-{
- cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
-}
-
static void sanitise_featureset(uint32_t *fs)
{
/* for_each_set_bit() uses unsigned longs. Extend with zeroes. */
diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h
index 75632d9..9283fe6 100644
--- a/xen/include/asm-x86/processor.h
+++ b/xen/include/asm-x86/processor.h
@@ -13,6 +13,7 @@
#include <asm/types.h>
#include <asm/cpufeature.h>
#include <asm/desc.h>
+#include <asm/x86_emulate.h>
#endif
#include <asm/x86-defns.h>
@@ -264,6 +265,12 @@ static always_inline unsigned int cpuid_count_ebx(
return ebx;
}
+static always_inline void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
+ struct cpuid_leaf *data)
+{
+ cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
+}
+
static inline unsigned long read_cr0(void)
{
unsigned long cr0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 03/23] x86: refactor psr: remove L3 CAT/CDP codes.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-06-14 1:12 ` [PATCH v12 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-06-14 1:12 ` [PATCH v12 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
` (19 subsequent siblings)
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
The current cache allocation codes in psr.c do not consider
future features addition and are not friendly to extend.
To make psr.c be more flexible to add new features and fulfill
the program principle, open for extension but closed for
modification, we have to refactor the psr.c:
1. Analyze cache allocation features and abstract general data
structures.
2. Analyze the init and all other functions flow, abstract all
steps that different features may have different implementations.
Make these steps be callback functions and register feature
specific fuctions. Then, the main processes will not be changed
when introducing a new feature.
Because the quantity of refactor codes is big and the logics are
changed a lot, it will cause reviewers confused if just change
old codes. Reviewers have to understand both old codes and new
implementations. After review iterations from V1 to V3, Jan has
proposed to remove all old cache allocation codes firstly, then
implement new codes step by step. This will help to make codes
be more easily reviewable.
There is no construction without destruction. So, this patch
removes all current L3 CAT/CDP codes in psr.c. The following
patches will introduce the new mechanism.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v4:
- create this patch to make codes easily understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 470 +----------------------------------------------------
1 file changed, 5 insertions(+), 465 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0b5073c..96a8589 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -23,24 +23,6 @@
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
-struct psr_cat_cbm {
- union {
- uint64_t cbm;
- struct {
- uint64_t code;
- uint64_t data;
- };
- };
- unsigned int ref;
-};
-
-struct psr_cat_socket_info {
- unsigned int cbm_len;
- unsigned int cos_max;
- struct psr_cat_cbm *cos_to_cbm;
- spinlock_t cbm_lock;
-};
-
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -48,26 +30,11 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
-static unsigned long *__read_mostly cat_socket_enable;
-static struct psr_cat_socket_info *__read_mostly cat_socket_info;
-static unsigned long *__read_mostly cdp_socket_enable;
-
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
-static unsigned int __read_mostly opt_cos_max = 255;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
-static struct psr_cat_cbm *temp_cos_to_cbm;
-
-static unsigned int get_socket_cpu(unsigned int socket)
-{
- if ( likely(socket < nr_sockets) )
- return cpumask_any(socket_cpumask[socket]);
-
- return nr_cpu_ids;
-}
-
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -107,9 +74,6 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
- if ( val_str && !strcmp(s, "cos_max") )
- opt_cos_max = simple_strtoul(val_str, NULL, 0);
-
s = ss + 1;
} while ( ss );
}
@@ -213,16 +177,7 @@ static inline void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( cat_socket_info )
- {
- unsigned int socket = cpu_to_socket(smp_processor_id());
-
- if ( test_bit(socket, cat_socket_enable) )
- psra->cos_mask = ((1ull << get_count_order(
- cat_socket_info[socket].cos_max)) - 1) << 32;
- }
-
- if ( psr_cmt_enabled() || psra->cos_mask )
+ if ( psr_cmt_enabled() )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -231,12 +186,6 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
-static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
- uint64_t cos_mask)
-{
- *reg = (*reg & ~cos_mask) | (((uint64_t)cos << 32) & cos_mask);
-}
-
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -245,459 +194,54 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
- if ( psra->cos_mask )
- psr_assoc_cos(®, d->arch.psr_cos_ids ?
- d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
- 0, psra->cos_mask);
-
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
psra->val = reg;
}
}
-static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
-{
- if ( !cat_socket_info )
- return ERR_PTR(-ENODEV);
-
- if ( socket >= nr_sockets )
- return ERR_PTR(-ENOTSOCK);
-
- if ( !test_bit(socket, cat_socket_enable) )
- return ERR_PTR(-ENOENT);
-
- return cat_socket_info + socket;
-}
-
-static inline bool_t cdp_is_enabled(unsigned int socket)
-{
- return cdp_socket_enable && test_bit(socket, cdp_socket_enable);
-}
int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
uint32_t *cos_max, uint32_t *flags)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- *cbm_len = info->cbm_len;
- *cos_max = info->cos_max;
-
- *flags = 0;
- if ( cdp_is_enabled(socket) )
- *flags |= XEN_SYSCTL_PSR_CAT_L3_CDP;
-
return 0;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
- bool_t cdp_enabled = cdp_is_enabled(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- if ( cdp_enabled )
- return -EXDEV;
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].code;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].data;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- }
-
- return 0;
-}
-
-static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
-{
- unsigned int first_bit, zero_bit;
-
- /* Set bits should only in the range of [0, cbm_len). */
- if ( cbm & (~0ull << cbm_len) )
- return 0;
-
- /* At least one bit need to be set. */
- if ( cbm == 0 )
- return 0;
-
- first_bit = find_first_bit(&cbm, cbm_len);
- zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
-
- /* Set bits should be contiguous. */
- if ( zero_bit < cbm_len &&
- find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
- return 0;
-
- return 1;
-}
-
-struct cos_cbm_info
-{
- unsigned int cos;
- bool_t cdp;
- uint64_t cbm_code;
- uint64_t cbm_data;
-};
-
-static void do_write_l3_cbm(void *data)
-{
- struct cos_cbm_info *info = data;
-
- if ( info->cdp )
- {
- wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(info->cos), info->cbm_code);
- wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(info->cos), info->cbm_data);
- }
- else
- wrmsrl(MSR_IA32_PSR_L3_MASK(info->cos), info->cbm_code);
-}
-
-static int write_l3_cbm(unsigned int socket, unsigned int cos,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp)
-{
- struct cos_cbm_info info =
- {
- .cos = cos,
- .cbm_code = cbm_code,
- .cbm_data = cbm_data,
- .cdp = cdp,
- };
-
- if ( socket == cpu_to_socket(smp_processor_id()) )
- do_write_l3_cbm(&info);
- else
- {
- unsigned int cpu = get_socket_cpu(socket);
-
- if ( cpu >= nr_cpu_ids )
- return -ENOTSOCK;
- on_selected_cpus(cpumask_of(cpu), do_write_l3_cbm, &info, 1);
- }
-
return 0;
}
-static int find_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
-{
- unsigned int cos;
-
- for ( cos = 0; cos <= cos_max; cos++ )
- {
- if ( (map[cos].ref || cos == 0) &&
- ((!cdp_enabled && map[cos].cbm == cbm_code) ||
- (cdp_enabled && map[cos].code == cbm_code &&
- map[cos].data == cbm_data)) )
- return cos;
- }
-
- return -ENOENT;
-}
-
-static int pick_avail_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- unsigned int old_cos)
-{
- unsigned int cos;
-
- /* If old cos is referred only by the domain, then use it. */
- if ( map[old_cos].ref == 1 && old_cos != 0 )
- return old_cos;
-
- /* Find an unused one other than cos0. */
- for ( cos = 1; cos <= cos_max; cos++ )
- if ( map[cos].ref == 0 )
- return cos;
-
- return -ENOENT;
-}
-
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type)
{
- unsigned int old_cos, cos_max;
- int cos, ret;
- uint64_t cbm_data, cbm_code;
- bool_t cdp_enabled = cdp_is_enabled(socket);
- struct psr_cat_cbm *map;
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- if ( !psr_check_cbm(info->cbm_len, cbm) )
- return -EINVAL;
-
- if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
- type == PSR_CBM_TYPE_L3_DATA) )
- return -ENXIO;
-
- cos_max = info->cos_max;
- old_cos = d->arch.psr_cos_ids[socket];
- map = info->cos_to_cbm;
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- cbm_code = cbm;
- cbm_data = cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- cbm_code = cbm;
- cbm_data = map[old_cos].data;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- cbm_code = map[old_cos].code;
- cbm_data = cbm;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- return -EINVAL;
- }
-
- spin_lock(&info->cbm_lock);
- cos = find_cos(map, cos_max, cbm_code, cbm_data, cdp_enabled);
- if ( cos >= 0 )
- {
- if ( cos == old_cos )
- {
- spin_unlock(&info->cbm_lock);
- return 0;
- }
- }
- else
- {
- cos = pick_avail_cos(map, cos_max, old_cos);
- if ( cos < 0 )
- {
- spin_unlock(&info->cbm_lock);
- return cos;
- }
-
- /* We try to avoid writing MSR. */
- if ( (cdp_enabled &&
- (map[cos].code != cbm_code || map[cos].data != cbm_data)) ||
- (!cdp_enabled && map[cos].cbm != cbm_code) )
- {
- ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
- if ( ret )
- {
- spin_unlock(&info->cbm_lock);
- return ret;
- }
- map[cos].code = cbm_code;
- map[cos].data = cbm_data;
- }
- }
-
- map[cos].ref++;
- map[old_cos].ref--;
- spin_unlock(&info->cbm_lock);
-
- d->arch.psr_cos_ids[socket] = cos;
-
return 0;
}
-/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
-static void psr_free_cos(struct domain *d)
-{
- unsigned int socket;
- unsigned int cos;
- struct psr_cat_socket_info *info;
-
- if( !d->arch.psr_cos_ids )
- return;
-
- for_each_set_bit(socket, cat_socket_enable, nr_sockets)
- {
- if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
- continue;
-
- info = cat_socket_info + socket;
- spin_lock(&info->cbm_lock);
- info->cos_to_cbm[cos].ref--;
- spin_unlock(&info->cbm_lock);
- }
-
- xfree(d->arch.psr_cos_ids);
- d->arch.psr_cos_ids = NULL;
-}
-
int psr_domain_init(struct domain *d)
{
- if ( cat_socket_info )
- {
- d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
- if ( !d->arch.psr_cos_ids )
- return -ENOMEM;
- }
-
return 0;
}
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
- psr_free_cos(d);
-}
-
-static int cat_cpu_prepare(unsigned int cpu)
-{
- if ( !cat_socket_info )
- return 0;
-
- if ( temp_cos_to_cbm == NULL &&
- (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
- opt_cos_max + 1UL)) == NULL )
- return -ENOMEM;
-
- return 0;
-}
-
-static void cat_cpu_init(void)
-{
- unsigned int eax, ebx, ecx, edx;
- struct psr_cat_socket_info *info;
- unsigned int socket;
- unsigned int cpu = smp_processor_id();
- uint64_t val;
- const struct cpuinfo_x86 *c = cpu_data + cpu;
-
- if ( !cpu_has(c, X86_FEATURE_PQE) || c->cpuid_level < PSR_CPUID_LEVEL_CAT )
- return;
-
- socket = cpu_to_socket(cpu);
- if ( test_bit(socket, cat_socket_enable) )
- return;
-
- cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
- if ( ebx & PSR_RESOURCE_TYPE_L3 )
- {
- cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
- info = cat_socket_info + socket;
- info->cbm_len = (eax & 0x1f) + 1;
- info->cos_max = min(opt_cos_max, edx & 0xffff);
-
- info->cos_to_cbm = temp_cos_to_cbm;
- temp_cos_to_cbm = NULL;
- /* cos=0 is reserved as default cbm(all ones). */
- info->cos_to_cbm[0].cbm = (1ull << info->cbm_len) - 1;
-
- spin_lock_init(&info->cbm_lock);
-
- set_bit(socket, cat_socket_enable);
-
- if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
- cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
- {
- info->cos_to_cbm[0].code = (1ull << info->cbm_len) - 1;
- info->cos_to_cbm[0].data = (1ull << info->cbm_len) - 1;
-
- /* We only write mask1 since mask0 is always all ones by default. */
- wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->cbm_len) - 1);
-
- rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
- wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
-
- /* Cut half of cos_max when CDP is enabled. */
- info->cos_max >>= 1;
-
- set_bit(socket, cdp_socket_enable);
- }
- printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
- socket, info->cos_max, info->cbm_len,
- cdp_is_enabled(socket) ? "on" : "off");
- }
-}
-
-static void cat_cpu_fini(unsigned int cpu)
-{
- unsigned int socket = cpu_to_socket(cpu);
-
- if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
- {
- struct psr_cat_socket_info *info = cat_socket_info + socket;
-
- if ( info->cos_to_cbm )
- {
- xfree(info->cos_to_cbm);
- info->cos_to_cbm = NULL;
- }
-
- if ( cdp_is_enabled(socket) )
- clear_bit(socket, cdp_socket_enable);
-
- clear_bit(socket, cat_socket_enable);
- }
-}
-
-static void __init psr_cat_free(void)
-{
- xfree(cat_socket_enable);
- cat_socket_enable = NULL;
- xfree(cat_socket_info);
- cat_socket_info = NULL;
-}
-
-static void __init init_psr_cat(void)
-{
- if ( opt_cos_max < 1 )
- {
- printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
- return;
- }
-
- cat_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
- cat_socket_info = xzalloc_array(struct psr_cat_socket_info, nr_sockets);
- cdp_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
-
- if ( !cat_socket_enable || !cat_socket_info )
- psr_cat_free();
}
static int psr_cpu_prepare(unsigned int cpu)
{
- return cat_cpu_prepare(cpu);
+ return 0;
}
static void psr_cpu_init(void)
{
- if ( cat_socket_info )
- cat_cpu_init();
-
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
- if ( cat_socket_info )
- cat_cpu_fini(cpu);
+ return;
}
static int cpu_callback(
@@ -738,14 +282,10 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- if ( opt_psr & PSR_CAT )
- init_psr_cat();
-
- if ( psr_cpu_prepare(0) )
- psr_cat_free();
+ psr_cpu_prepare(0);
psr_cpu_init();
- if ( psr_cmt_enabled() || cat_socket_info )
+ if ( psr_cmt_enabled() )
register_cpu_notifier(&cpu_nfb);
return 0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (2 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 03/23] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-28 7:12 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
` (18 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
To construct an extendible framework, we need analyze PSR features
and abstract the common things and feature specific things. Then,
encapsulate them into different data structures.
By analyzing PSR features, we can get below map.
+------+------+------+
--------->| Dom0 | Dom1 | ... |
| +------+------+------+
| |
|Dom ID | cos_id of domain
| V
| +-----------------------------------------------------------------------------+
User --------->| PSR |
Socket ID | +--------------+---------------+---------------+ |
| | Socket0 Info | Socket 1 Info | ... | |
| +--------------+---------------+---------------+ |
| | cos_id=0 cos_id=1 ... |
| | +-----------------------+-----------------------+-----------+ |
| |->Ref : | ref 0 | ref 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L3 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L2 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------+-----------+-----------+-----------+-----------+ |
| |->CDP : | cos0 code | cos0 data | cos1 code | cos1 data | ... | |
| +-----------+-----------+-----------+-----------+-----------+ |
+-----------------------------------------------------------------------------+
So, we need define a socket info data structure, 'struct
psr_socket_info' to manage information per socket. It contains a
reference count array according to COS ID and a feature array to
manage all features enabled. Every entry of the reference count
array is used to record how many domains are using the COS registers
according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
Dom1 uses COS_ID=1 registers of both features to save CBM values, like
below.
+-------+-------+-------+-----+
| COS 0 | COS 1 | COS 2 | ... |
+-------+-------+-------+-----+
L3 CAT | 0x7ff | 0x1ff | ... | ... |
+-------+-------+-------+-----+
L2 CAT | 0xff | 0xff | ... | ... |
+-------+-------+-------+-----+
If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
That means, both Dom1 and Dom2 use same COS registers(ID=1) to keep same
L3/L2 values. So, the value of ref[1] is 2 which means 2 domains are using
COS_ID 1.
To manage a feature, we need define a feature node data structure,
'struct feat_node', to manage feature's specific HW info, and an array of all
COS registers values of this feature.
To manage feature properties, we need define a feature property data structure,
'struct feat_props', to manage common properties (callback functions - all
feature's specific behaviors are encapsulated into these callback functions,
and generic values - e.g. the cos_max), the feature independent values.
CDP is a special feature which uses two entries of the array
for one COS ID. So, the number of CDP COS registers is the half of L3
CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
it is enabled. CDP uses the COS registers array as below.
+-----------+-----------+-----------+-----------+-----------+
CDP cos_reg_val[] index: | 0 | 1 | 2 | 3 | ... |
+-----------+-----------+-----------+-----------+-----------+
value: | cos0 code | cos0 data | cos1 code | cos1 data | ... |
+-----------+-----------+-----------+-----------+-----------+
For more details, please refer SDM and patches to implement 'get value' and
'set value'.
This patch also implements the CPU init and free flow including L3 CAT
initialization and some resources free. It includes below flows:
1. presmp init:
- parse command line parameter.
- allocate socket info for every socket.
- allocate feature resource.
- initialize socket info, get feature info and add feature into feature
array per cpuid result.
- free resources allocated if error happens.
- register cpu notifier to handle cpu events.
2. cpu notifier:
- handle cpu online events, if initialization work has been done before,
do nothing.
- handle cpu offline events, if it is the last cpu offline, free some
socket resources.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- add comment for 'feat_init'.
(suggested by Jan Beulich)
- use 'ARRAY_SIZE()' to check array boundary.
(suggested by Jan Beulich)
- use 'XENLOG_WARNING' for error message.
(suggested by Jan Beulich)
- move 'type[]' assignment for feat_props object into its delcaration
to make the object be 'const'.
(suggested by Jan Beulich)
- remove "L3" and "L2" indication in printk.
(suggested by Jan Beulich)
- move position of 'ref_lock' definition.
- restore mask(0) MSR to default value.
(suggested by Jan Beulich)
v11:
- handle 'feat_init'.
- merge main data structures implementation into CPU init/free patch,
including commit messages and change history.
(suggested by Jan Beulich)
- remove MSR restore action which is unnecessary.
(suggested by Jan Beulich)
- move 'type[]' declaration into this patch.
(suggested by Jan Beulich)
- modify comment.
(suggested by Jan Beulich)
v10:
- remove initialization for 'PSR_SOCKET_L3_CAT'.
(suggested by Jan Beulich)
- rename 'feat_ops' to 'feat_props'.
(suggested by Jan Beulich)
- move 'cbm_len' to 'feat_props' because it is feature independent so far.
(suggested by Jan Beulich)
- move 'cos_max' to 'feat_props' because it is feature independent.
(suggested by Jan Beulich)
- move 'cos_num' to 'feat_props' because it is feature independent.
(suggested by Jan Beulich)
- remove union 'info' and struct 'psr_cat_hw_info'.
- remove 'get_cos_max' from 'feat_props'.
(suggested by Jan Beulich)
- remove 'feat_mask' from 'psr_socket_info' because we can use 'features[]'
to check if any feature is initialized.
(suggested by Jan Beulich)
- move 'ref_lock' above 'cos_ref'.
(suggested by Jan Beulich)
- adjust comments and commit message according to above changes.
- remove 'asm/x86_emulate.h' inclusion as it has been indirectly included.
(suggested by Jan Beulich)
- remove 'CAT_COS_NUM' as it is only used once.
(suggested by Jan Beulich)
- remove 'feat_mask'.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- remove 'get_cos_max' hook declaration.
(suggested by Jan Beulich)
- modify 'cat_default_val' implementation.
(suggested by Jan Beulich)
- modify 'psr_alloc_feat_enabled' implementation to make it simple.
(suggested by Jan Beulich)
- rename 'free_feature' to 'free_socket_resources' because it is executed
when socket is offline. It needs free resources related to the socket.
(suggested by Jan Beulich)
- define 'feat_init_done' to iterate feature array to check if any feature
has been initialized.
(suggested by Jan Beulich)
- input 'struct cpuid_leaf' pointer into 'cat_init_feature' to avoid memory
copy.
(suggested by Jan Beulich)
- modify 'cat_init_feature' to use switch and things related to above
changes.
(suggested by Jan Beulich)
- add an indentation for label.
(suggested by Jan Beulich)
v9:
- replace feature list to a feature pointer array.
(suggested by Roger Pau)
- add 'PSR_SOCKET_MAX_FEAT' in 'enum psr_feat_type' to know features
account.
(suggested by Roger Pau)
- move 'feat_ops' declaration into 'feat_node' structure.
(suggested by Roger Pau)
- directly use uninon for feature HW info and move its declaration into
'feat_node' structure.
(suggested by Roger Pau)
- remove 'enum psr_feat_type feature' declared in 'feat_ops' because it is
not useful after using feature pointer array.
(suggested by Roger Pau)
- rename 'l3_cat_info' to 'cat_info' to be used by all CAT/CDP features.
- remove 'nr_feat' which is only for a record.
(suggested by Jan Beulich)
- add 'cos_num' to record how many COS registers are used by a feature in
one time access.
(suggested by Jan Beulich)
- replace 'uint64_t' to 'uint32_t' for cbm value because SDM specifies the
max 32 bits for it.
(suggested by Jan Beulich)
- add commit message to explain the flows.
- handle cpu offline and online again case to read MSRs registers values
back and save them into cos array to make user can get real data.
- create a new patch about moving 'cpuid_count_leaf'.
(suggested by Wei Liu)
- modify comment to explain why not free some resource in 'free_feature'.
(suggested by Wei Liu)
- implement 'psr_alloc_feat_enabled' to check if allocation feature is
enabled in cmdline and some initialization work done.
(suggested by Wei Liu)
- implement 'cat_default_val' to set default value for CAT features.
(suggested by Wei Liu)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- implement a common 'cat_init_feature' to replace L3 CAT/L2 CAT specific
init functions.
(suggested by Roger Pau)
- modify comments for global feature node.
(suggested by Jan Beulich)
- remove unnecessary comments.
(suggested by Jan Beulich)
- remove unnecessary 'else'.
(suggested by Jan Beulich)
- remove 'nr_feat'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- check global flag with boot cpu operations.
(suggested by Jan Beulich)
- remove 'cpu_init_work' and move codes into 'psr_cpu_init'.
(suggested by Jan Beulich)
- remove 'cpu_fini_work' and move codes into 'psr_cpu_fini'.
(suggested by Jan Beulich)
- assign value for 'cos_num'.
(suggested by Jan Beulich)
- change about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- fix format issue.
(suggested by Konrad Rzeszutek Wilk)
- add comments to explain why we care about cpumask_empty when the last
cpu on socket is offline.
(suggested by Konrad Rzeszutek Wilk)
v7:
- sort inclusion files position.
(suggested by Wei Liu)
- initialize structure objects for avoiding surprise.
(suggested by Konrad Rzeszutek Wilk)
- fix typo.
(suggested by Konrad Rzeszutek Wilk)
- fix a logical mistake when handling the last cpu offline event.
(suggested by Konrad Rzeszutek Wilk)
v6:
- make commit message be clearer.
(suggested by Konrad Rzeszutek Wilk)
- fix wordings.
(suggested by Konrad Rzeszutek Wilk)
- add comments to explain relationship between 'feat_mask' and
'enum psr_feat_type'.
(suggested by Konrad Rzeszutek Wilk)
- use 'struct cpuid_leaf' introduced in Andrew's patch.
(suggested by Konrad Rzeszutek Wilk)
- add comments about cpu_add_remove_lock.
(suggested by Konrad Rzeszutek Wilk)
- change 'clear_bit' to '__clear_bit'.
(suggested by Konrad Rzeszutek Wilk)
- add 'ASSERT' check when setting 'feat_mask'.
(suggested by Konrad Rzeszutek Wilk)
- adjust 'printk' position to avoid odd spacing.
(suggested by Konrad Rzeszutek Wilk)
- add comment to explain usage of 'feat_l3_cat'.
(suggested by Konrad Rzeszutek Wilk)
- fix wording.
(suggested by Konrad Rzeszutek Wilk)
- move 'cpuid_count_leaf' helper function to 'asm-x86/processor.h'.
It cannot be moved to 'cpuid.h' which causes compilation error because
of header file loop reference.
(suggested by Andrew Cooper)
v5:
- remove section number.
(suggested by Jan Beulich)
- remove double blank.
(suggested by Jan Beulich)
- add comment to explain the reason to define 'feat_l3_cat'.
(suggested by Jan Beulich)
- use 'list_for_each_entry_safe'.
(suggested by Jan Beulich)
- remove codes to free 'feat_l3_cat' in 'free_feature' to avoid the need
for an allocation the next time a CPU comes online.
(suggested by Jan Beulich)
- define 'struct cpuid_leaf_regs' to encapsulate eax~edx.
(suggested by Jan Beulich)
- print feature info on a socket only when 'opt_cpu_info' is true.
(suggested by Jan Beulich)
- declare global variable 'l3_cat_ops' to 'static const'.
(suggested by Jan Beulich)
- use 'current_cpu_data'.
(suggested by Jan Beulich)
- rename 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- clear PQE feature bit when the maximum CPUID level is too low.
(suggested by Jan Beulich)
- directly call 'l3_cat_init_feature'. No need to make it a callback
function.
(suggested by Jan Beulich)
- remove local variable 'info'.
(suggested by Jan Beulich)
- move 'INIT_LIST_HEAD' into 'cpu_init_work' to be together with
spin_lock_init().
(suggested by Jan Beulich)
- remove 'cpu_prepare_work' and move its content into 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v4:
- create this patch because of removing all old CAT/CDP codes to make
implementation be more easily understood.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 278 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 272 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 96a8589..251746a 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -13,16 +13,112 @@
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
* more details.
*/
-#include <xen/init.h>
#include <xen/cpu.h>
#include <xen/err.h>
+#include <xen/init.h>
#include <xen/sched.h>
#include <asm/psr.h>
+/*
+ * Terminology:
+ * - CAT Cache Allocation Technology
+ * - CBM Capacity BitMasks
+ * - CDP Code and Data Prioritization
+ * - CMT Cache Monitoring Technology
+ * - COS/CLOS Class of Service. Also mean COS registers.
+ * - COS_MAX Max number of COS for the feature (minus 1)
+ * - MSRs Machine Specific Registers
+ * - PSR Intel Platform Shared Resource
+ */
+
#define PSR_CMT (1<<0)
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
+#define CAT_CBM_LEN_MASK 0x1f
+#define CAT_COS_MAX_MASK 0xffff
+
+/*
+ * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
+ * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
+ * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
+ *
+ * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for
+ * up to 64 L2 CAT COS. The COS_ID=[0,63].
+ *
+ * So, the maximum COS register count of one feature is 128.
+ */
+#define MAX_COS_REG_CNT 128
+
+/*
+ * Every PSR feature uses some COS registers for each COS ID, e.g. CDP uses 2
+ * COS registers (DATA and CODE) for one COS ID, but CAT uses 1 COS register.
+ * We use below macro as the max number of COS registers used by all features.
+ * So far, it is 2 which means CDP's COS registers number.
+ */
+#define PSR_MAX_COS_NUM 2
+
+enum psr_feat_type {
+ PSR_SOCKET_L3_CAT,
+ PSR_SOCKET_FEAT_NUM,
+};
+
+/*
+ * This structure represents one feature.
+ * cos_max - The max COS registers number got through CPUID.
+ * cbm_len - The length of CBM got through CPUID.
+ * cos_reg_val - Array to store the values of COS registers. One entry stores
+ * the value of one COS register.
+ * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
+ * For CDP, two entries correspond to one COS_ID. E.g.
+ * COS_ID=0 corresponds to cos_reg_val[0] (Data) and
+ * cos_reg_val[1] (Code).
+ */
+struct feat_node {
+ /* cos_max and cbm_len are common values for all features so far. */
+ unsigned int cos_max;
+ unsigned int cbm_len;
+ uint32_t cos_reg_val[MAX_COS_REG_CNT];
+};
+
+/*
+ * This structure defines feature specific values, e.g. cos_num.
+ *
+ * Array 'feat_props' is defined to save every feature's properties. We use
+ * 'enum psr_feat_type' as index.
+ */
+static const struct feat_props {
+ /*
+ * cos_num - COS registers number that feature uses for one COS ID.
+ * It is defined in SDM.
+ */
+ unsigned int cos_num;
+
+ /*
+ * An array to save all 'enum cbm_type' values of the feature. It is
+ * used with cos_num together to get/write a feature's COS registers
+ * values one by one.
+ */
+ enum cbm_type type[PSR_MAX_COS_NUM];
+} *feat_props[PSR_SOCKET_FEAT_NUM];
+
+/*
+ * PSR features are managed per socket. Below structure defines the members
+ * used to manage these features.
+ * feat_init - Indicate if features on a socket have been initialized.
+ * features - A feature node array used to manage all features enabled.
+ * ref_lock - A lock to protect cos_ref.
+ * cos_ref - A reference count array to record how many domains are using the
+ * COS ID. Every entry of cos_ref corresponds to one COS ID.
+ */
+struct psr_socket_info {
+ bool feat_init;
+ /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
+ struct feat_node *features[PSR_SOCKET_FEAT_NUM];
+ spinlock_t ref_lock;
+ unsigned int cos_ref[MAX_COS_REG_CNT];
+};
+
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -30,11 +126,97 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
+static struct psr_socket_info *__read_mostly socket_info;
+
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
+static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
+/*
+ * Declare global feature node for every feature to facilitate the feature
+ * array creation. It is used to transiently store a spare node.
+ */
+static struct feat_node *feat_l3_cat;
+
+/* Common functions */
+#define cat_default_val(len) (0xffffffff >> (32 - (len)))
+
+/*
+ * Use this function to check if any allocation feature has been enabled
+ * in cmdline.
+ */
+static bool psr_alloc_feat_enabled(void)
+{
+ return !!socket_info;
+}
+
+static void free_socket_resources(unsigned int socket)
+{
+ unsigned int i;
+ struct psr_socket_info *info = socket_info + socket;
+
+ if ( !info )
+ return;
+
+ /*
+ * Free resources of features. The global feature object, e.g. feat_l3_cat,
+ * may not be freed here if it is not added into array. It is simply being
+ * kept until the next CPU online attempt.
+ */
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ xfree(info->features[i]);
+ info->features[i] = NULL;
+ }
+
+ info->feat_init = false;
+}
+
+/* CAT common functions implementation. */
+static void cat_init_feature(const struct cpuid_leaf *regs,
+ struct feat_node *feat,
+ struct psr_socket_info *info,
+ enum psr_feat_type type)
+{
+ /* No valid value so do not enable feature. */
+ if ( !regs->a || !regs->d )
+ return;
+
+ feat->cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
+ feat->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
+
+ switch ( type )
+ {
+ case PSR_SOCKET_L3_CAT:
+ /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+ feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
+
+ wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
+
+ break;
+
+ default:
+ return;
+ }
+
+ /* Add this feature into array. */
+ info->features[type] = feat;
+
+ if ( !opt_cpu_info )
+ return;
+
+ printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
+}
+
+/* L3 CAT props */
+static const struct feat_props l3_cat_props = {
+ .cos_num = 1,
+ .type[0] = PSR_CBM_TYPE_L3,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -74,6 +256,9 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
+ if ( val_str && !strcmp(s, "cos_max") )
+ opt_cos_max = simple_strtoul(val_str, NULL, 0);
+
s = ss + 1;
} while ( ss );
}
@@ -229,19 +414,96 @@ void psr_domain_free(struct domain *d)
psr_free_rmid(d);
}
-static int psr_cpu_prepare(unsigned int cpu)
+static void __init init_psr(void)
{
+ if ( opt_cos_max < 1 )
+ {
+ printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
+ return;
+ }
+
+ socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
+
+ if ( !socket_info )
+ {
+ printk(XENLOG_WARNING "Failed to alloc socket_info!\n");
+ return;
+ }
+}
+
+static void __init psr_free(void)
+{
+ xfree(socket_info);
+ socket_info = NULL;
+}
+
+static int psr_cpu_prepare(void)
+{
+ if ( !psr_alloc_feat_enabled() )
+ return 0;
+
+ /* Malloc memory for the global feature node here. */
+ if ( feat_l3_cat == NULL &&
+ (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
static void psr_cpu_init(void)
{
+ struct psr_socket_info *info;
+ unsigned int socket, cpu = smp_processor_id();
+ struct feat_node *feat;
+ struct cpuid_leaf regs;
+
+ if ( !psr_alloc_feat_enabled() || !boot_cpu_has(X86_FEATURE_PQE) )
+ goto assoc_init;
+
+ if ( boot_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
+ {
+ setup_clear_cpu_cap(X86_FEATURE_PQE);
+ goto assoc_init;
+ }
+
+ socket = cpu_to_socket(cpu);
+ info = socket_info + socket;
+ if ( info->feat_init )
+ goto assoc_init;
+
+ spin_lock_init(&info->ref_lock);
+
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L3 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
+
+ feat = feat_l3_cat;
+ feat_l3_cat = NULL;
+ feat_props[PSR_SOCKET_L3_CAT] = &l3_cat_props;
+
+ cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CAT);
+
+ info->feat_init = true;
+ }
+
+ assoc_init:
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
- return;
+ unsigned int socket = cpu_to_socket(cpu);
+
+ if ( !psr_alloc_feat_enabled() )
+ return;
+
+ /*
+ * We only free when we are the last CPU in the socket. The socket_cpumask
+ * is cleared prior to this notification code by remove_siblinginfo().
+ */
+ if ( socket_cpumask[socket] && cpumask_empty(socket_cpumask[socket]) )
+ free_socket_resources(socket);
}
static int cpu_callback(
@@ -253,7 +515,7 @@ static int cpu_callback(
switch ( action )
{
case CPU_UP_PREPARE:
- rc = psr_cpu_prepare(cpu);
+ rc = psr_cpu_prepare();
break;
case CPU_STARTING:
psr_cpu_init();
@@ -282,10 +544,14 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- psr_cpu_prepare(0);
+ if ( opt_psr & PSR_CAT )
+ init_psr();
+
+ if ( psr_cpu_prepare() )
+ psr_free();
psr_cpu_init();
- if ( psr_cmt_enabled() )
+ if ( psr_cmt_enabled() || psr_alloc_feat_enabled() )
register_cpu_notifier(&cpu_nfb);
return 0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (3 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-28 7:13 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 06/23] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
` (17 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the Domain init/free and schedule flows.
- When domain init, its psr resource should be allocated.
- When domain free, its psr resource should be freed too.
- When domain is scheduled, its COS ID on the socket should be
set into ASSOC register to make corresponding COS MSR value
work.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- fix coding style issue.
(suggested by Jan Beulich)
- in 'get_max_cos_max', invert the condition and dropping 'continue'.
(suggested by Jan Beulich)
- modify 'psr_assoc_cos' to directly return 'reg' value.
(suggested by Jan Beulich)
- modify code style.
(suggested by Jan Beulich)
- do not break domain creation if 'psr_cos_ids' allocation fails.
(suggested by Jan Beulich)
v11:
- replace 'feat_init_done()' to 'feat_init' flag.
(suggested by Jan Beulich)
- adjust parameters positions when calling 'psr_assoc_cos'.
(suggested by Jan Beulich)
- add comment to explain why to check 'psr_cos_ids'.
v10:
- remove 'cat_get_cos_max' as 'cos_max' is a feature property now which
can be directly used.
(suggested by Jan Beulich)
- replace 'info->feat_mask' check to 'feat_init_done'.
(suggested by Jan Beulich)
v9:
- rename 'l3_cat_get_cos_max' to 'cat_get_cos_max' to cover all CAT/CDP
features.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- implement 'psr_alloc_cos' to match 'psr_free_cos'.
(suggested by Wei Liu)
- use 'psr_alloc_feat_enabled'.
(suggested by Wei Liu)
- fix coding style issue.
(suggested by Wei Liu)
- remove 'inline'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- remove 'psr_cos_ids' check in 'psr_free_cos'.
(suggested by Jan Beulich)
v6:
- change 'PSR_ASSOC_REG_POS' to 'PSR_ASSOC_REG_SHIFT'.
(suggested by Konrad Rzeszutek Wilk)
v5:
- rename 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- define 'PSR_ASSOC_REG_POS'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/domain.c | 3 +--
xen/arch/x86/psr.c | 67 ++++++++++++++++++++++++++++++++++++++++++++---
xen/include/asm-x86/psr.h | 2 +-
3 files changed, 65 insertions(+), 7 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 90e2b1f..a2abed1 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -646,8 +646,7 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags,
}
spin_lock_init(&d->arch.e820_lock);
- if ( (rc = psr_domain_init(d)) != 0 )
- goto fail;
+ psr_domain_init(d);
if ( is_hvm_domain(d) )
{
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 251746a..9fb0da8 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -50,6 +50,8 @@
*/
#define MAX_COS_REG_CNT 128
+#define PSR_ASSOC_REG_SHIFT 32
+
/*
* Every PSR feature uses some COS registers for each COS ID, e.g. CDP uses 2
* COS registers (DATA and CODE) for one COS ID, but CAT uses 1 COS register.
@@ -358,11 +360,37 @@ void psr_free_rmid(struct domain *d)
d->arch.psr_rmid = 0;
}
-static inline void psr_assoc_init(void)
+static unsigned int get_max_cos_max(const struct psr_socket_info *info)
+{
+ unsigned int cos_max = 0, i;
+
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ const struct feat_node *feat = info->features[i];
+
+ if ( feat )
+ cos_max = max(feat->cos_max, cos_max);
+ }
+
+ return cos_max;
+}
+
+static void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( psr_cmt_enabled() )
+ if ( psr_alloc_feat_enabled() )
+ {
+ unsigned int socket = cpu_to_socket(smp_processor_id());
+ const struct psr_socket_info *info = socket_info + socket;
+ unsigned int cos_max = get_max_cos_max(info);
+
+ if ( info->feat_init )
+ psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
+ PSR_ASSOC_REG_SHIFT;
+ }
+
+ if ( psr_cmt_enabled() || psra->cos_mask )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -371,6 +399,13 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
+static uint64_t psr_assoc_cos(uint64_t reg, unsigned int cos,
+ uint64_t cos_mask)
+{
+ return (reg & ~cos_mask) |
+ (((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
+}
+
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -379,6 +414,14 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
+ /* If domain's 'psr_cos_ids' is NULL, we set default value for it. */
+ if ( psra->cos_mask )
+ reg = psr_assoc_cos(reg,
+ (d->arch.psr_cos_ids ?
+ d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
+ 0),
+ psra->cos_mask);
+
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
@@ -404,14 +447,30 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
return 0;
}
-int psr_domain_init(struct domain *d)
+/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
+static void psr_free_cos(struct domain *d)
{
- return 0;
+ xfree(d->arch.psr_cos_ids);
+ d->arch.psr_cos_ids = NULL;
+}
+
+static void psr_alloc_cos(struct domain *d)
+{
+ d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
+ if ( !d->arch.psr_cos_ids )
+ printk(XENLOG_WARNING "Failed to alloc psr_cos_ids!\n");
+}
+
+void psr_domain_init(struct domain *d)
+{
+ if ( psr_alloc_feat_enabled() )
+ psr_alloc_cos(d);
}
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
+ psr_free_cos(d);
}
static void __init init_psr(void)
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 57f47e9..d146706 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -70,7 +70,7 @@ int psr_get_l3_cbm(struct domain *d, unsigned int socket,
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type);
-int psr_domain_init(struct domain *d);
+void psr_domain_init(struct domain *d);
void psr_domain_free(struct domain *d);
#endif /* __ASM_PSR_H__ */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 06/23] x86: refactor psr: L3 CAT: implement get hw info flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (4 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
` (16 subsequent siblings)
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements get HW info flow including L3 CAT callback
function.
It also changes sysctl interface to make it more general.
With this patch, 'psr-hwinfo' can work for L3 CAT.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v12:
- use 'ASSERT_UNREACHABLE()' to record bug.
(suggested by Jan Beulich)
v11:
- changes about 'cos_max' and 'cbm_len'.
(suggested by Jan Beulich)
- change about 'feat_props'.
(suggested by Jan Beulich)
- declare 'PSR_SOCKET_FEAT_UNKNOWN' to handle error case.
(suggested by Jan Beulich)
- use 'ASSERT' to check data.
(suggested by Jan Beulich)
- fix comment of 'get_feat_info'.
(suggested by Jan Beulich)
- fix a bug when checking 'feat_type'.
(suggested by Jan Beulich)
v10:
- remove 'PSR_SOCKET_UNKNOWN' and use 'ASSERT_UNREACHABLE()' to handle
this case.
(suggested by Jan Beulich)
- check 'feat_type'.
(suggested by Jan Beulich)
- adjust macros names and values to make them more appropriate.
(suggested by Jan Beulich)
- use 'feat_init_done'.
(suggested by Jan Beulich)
- changes about 'cbm_len'.
(suggested by Jan Beulich)
v9:
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- define 'PSR_INFO_SIZE'.
(suggested by Roger Pau)
- fix coding style issue.
(suggested by Roger Pau and Jan Beulich)
- use 'ARRAY_SIZE'.
(suggested by Roger Pau)
- rename 'l3_cat_get_feat_info' to 'cat_get_feat_info' to make it a common
function for both L3/L2 CAT.
(suggested by Roger Pau)
- move constant to the right of comparison.
(suggested by Wei Liu)
- remove wrong comment.
(suggested by Jan Beulich)
- rename macros used by psr_get_info to make them meaningful.
(suggested by Jan Beulich)
- remove assignment for 'PSR_SOCKET_UNKNOWN'.
(suggested by Jan Beulich)
- retain blank line after 'case XEN_SYSCTL_PSR_CAT_get_l3_info'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- move common data check into common function.
(suggested by Jan Beulich)
v6:
- fix coding style issue.
(suggested by Konrad Rzeszutek Wilk)
- define 'PSR_SOCKET_UNKNOWN' in 'psr_feat_type'.
(suggested by Konrad Rzeszutek Wilk)
- change '-ENOTSOCK' to 'ERANGE'.
(suggested by Konrad Rzeszutek Wilk)
- modify position of macros to remove odd spacing in psr.h.
(suggested by Konrad Rzeszutek Wilk)
v5:
- change 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
- modify parameter type to avoid fixed width type when there is no such
intention.
(suggested by Jan Beulich)
- use 'const' when it is possible.
(suggested by Jan Beulich)
- check feature type outside callback function.
(suggested by Jan Beulich)
- modify macros names to add prefix 'PSR_' and change 'CDP_FLAG' to
'PSR_FLAG'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 81 +++++++++++++++++++++++++++++++++++++++++++++--
xen/arch/x86/sysctl.c | 19 ++++++++---
xen/include/asm-x86/psr.h | 16 +++++++---
3 files changed, 104 insertions(+), 12 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 9fb0da8..1ccbd00 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -63,6 +63,7 @@
enum psr_feat_type {
PSR_SOCKET_L3_CAT,
PSR_SOCKET_FEAT_NUM,
+ PSR_SOCKET_FEAT_UNKNOWN,
};
/*
@@ -102,6 +103,10 @@ static const struct feat_props {
* values one by one.
*/
enum cbm_type type[PSR_MAX_COS_NUM];
+
+ /* get_feat_info is used to return feature HW info through sysctl. */
+ bool (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len);
} *feat_props[PSR_SOCKET_FEAT_NUM];
/*
@@ -176,6 +181,22 @@ static void free_socket_resources(unsigned int socket)
info->feat_init = false;
}
+static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
+{
+ enum psr_feat_type feat_type = PSR_SOCKET_FEAT_UNKNOWN;
+
+ switch ( type )
+ {
+ case PSR_CBM_TYPE_L3:
+ feat_type = PSR_SOCKET_L3_CAT;
+ break;
+ default:
+ ASSERT_UNREACHABLE();
+ }
+
+ return feat_type;
+}
+
/* CAT common functions implementation. */
static void cat_init_feature(const struct cpuid_leaf *regs,
struct feat_node *feat,
@@ -213,10 +234,24 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
}
+static bool cat_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len)
+{
+ if ( array_len != PSR_INFO_ARRAY_SIZE )
+ return false;
+
+ data[PSR_INFO_IDX_COS_MAX] = feat->cos_max;
+ data[PSR_INFO_IDX_CAT_CBM_LEN] = feat->cbm_len;
+ data[PSR_INFO_IDX_CAT_FLAG] = 0;
+
+ return true;
+}
+
/* L3 CAT props */
static const struct feat_props l3_cat_props = {
.cos_num = 1,
.type[0] = PSR_CBM_TYPE_L3,
+ .get_feat_info = cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -429,10 +464,50 @@ void psr_ctxt_switch_to(struct domain *d)
}
}
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags)
+static struct psr_socket_info *get_socket_info(unsigned int socket)
{
- return 0;
+ if ( !socket_info )
+ return ERR_PTR(-ENODEV);
+
+ if ( socket >= nr_sockets )
+ return ERR_PTR(-ERANGE);
+
+ if ( !socket_info[socket].feat_init )
+ return ERR_PTR(-ENOENT);
+
+ return socket_info + socket;
+}
+
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len)
+{
+ const struct psr_socket_info *info = get_socket_info(socket);
+ const struct feat_node *feat;
+ enum psr_feat_type feat_type;
+
+ ASSERT(data);
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( feat_type >= ARRAY_SIZE(info->features) )
+ return -ENOENT;
+
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ if ( !feat_props[feat_type] )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( feat_props[feat_type]->get_feat_info(feat, data, array_len) )
+ return 0;
+
+ return -EINVAL;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 2f7056e..c23270d 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -175,14 +175,25 @@ long arch_do_sysctl(
switch ( sysctl->u.psr_cat_op.cmd )
{
case XEN_SYSCTL_PSR_CAT_get_l3_info:
- ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
- &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
- &sysctl->u.psr_cat_op.u.l3_info.cos_max,
- &sysctl->u.psr_cat_op.u.l3_info.flags);
+ {
+ uint32_t data[PSR_INFO_ARRAY_SIZE];
+
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
+ if ( ret )
+ break;
+
+ sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ data[PSR_INFO_IDX_COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ data[PSR_INFO_IDX_CAT_CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.flags =
+ data[PSR_INFO_IDX_CAT_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
ret = -EFAULT;
break;
+ }
default:
ret = -EOPNOTSUPP;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index d146706..9625e90 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -19,20 +19,26 @@
#include <xen/types.h>
/* CAT cpuid level */
-#define PSR_CPUID_LEVEL_CAT 0x10
+#define PSR_CPUID_LEVEL_CAT 0x10
/* Resource Type Enumeration */
#define PSR_RESOURCE_TYPE_L3 0x2
/* L3 Monitoring Features */
-#define PSR_CMT_L3_OCCUPANCY 0x1
+#define PSR_CMT_L3_OCCUPANCY 0x1
/* CDP Capability */
-#define PSR_CAT_CDP_CAPABILITY (1u << 2)
+#define PSR_CAT_CDP_CAPABILITY (1u << 2)
/* L3 CDP Enable bit*/
#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
+/* Used by psr_get_info() */
+#define PSR_INFO_IDX_COS_MAX 0
+#define PSR_INFO_IDX_CAT_CBM_LEN 1
+#define PSR_INFO_IDX_CAT_FLAG 2
+#define PSR_INFO_ARRAY_SIZE 3
+
struct psr_cmt_l3 {
unsigned int features;
unsigned int upscaling_factor;
@@ -63,8 +69,8 @@ int psr_alloc_rmid(struct domain *d);
void psr_free_rmid(struct domain *d);
void psr_ctxt_switch_to(struct domain *d);
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags);
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len);
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (5 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 06/23] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-28 7:14 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
` (15 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
There is an interface in user space to show feature value of
domains.
This patch implements get value flow in hypervisor.
It also changes domctl interface to make it more general.
With this patch, 'psr-cat-show' can work for L3 CAT but not for
L3 code/data which is implemented in CDP related patches.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- remove 'psr_get_feat_and_type' and implement its functionality in
'psr_get_val'.
- use 'ASSERT_UNREACHABLE' to record error.
(suggested by Jan Beulich)
- return success only if the 'val' is set in 'psr_get_val'.
(suggested by Jan Beulich)
v11:
- declare a 'switch()' wide variable 'val32' in domctl.
(suggested by Jan Beulich)
- remove 'get_val' callback function which is replaced by generic codes.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- do not 'ASSERT' domain pointer.
(suggested by Jan Beulich)
- modify commit message.
v10:
- use an intermediate variable to get value and avoid cast in domctl.
(suggested by Jan Beulich)
- remove 'type' in 'get_val' parameters and will add it back when
implementing CDP.
(suggested by Jan Beulich)
- remove unnecessary variable and return error about 'info' in
'psr_get_feat'.
(suggested by Jan Beulich)
- use 'ASSERT' to check input parameter in 'psr_get_val'.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
v9:
- add commit message to explain there is an user space interface.
- rename 'l3_cat_get_val' to 'cat_get_val' to cover all L3/L2 CAT features.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- change parameter of 'psr_get'. Use 'psr_cos_ids' directly to replace
domain. Also declare it to 'const'.
(suggested by Jan Beulich)
- change code flow to remove 'psr_get' but add 'psr_get_feat' to make codes
more reasonable.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- move cos check into common function because this check is required by all
features.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v7:
- rename '__psr_get' to 'psr_get'.
(suggested by Wei Liu)
v6:
- modify commit message to make it clearer.
(suggested by Konrad Rzeszutek Wilk)
- remove one extra space in code.
(suggested by Konrad Rzeszutek Wilk)
- remove unnecessary comment.
(suggested by Konrad Rzeszutek Wilk)
- write a helper function to move get info and get val functions into
it. Because most codes of 'get_info' and 'get_val' are same.
(suggested by Konrad Rzeszutek Wilk)
v5:
- rename 'dat[]' to 'data[]'
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- check if feature type match in caller of feature callback function.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/domctl.c | 20 ++++++++++---------
xen/arch/x86/psr.c | 51 ++++++++++++++++++++++++++++++++++++++++++++---
xen/include/asm-x86/psr.h | 4 ++--
3 files changed, 61 insertions(+), 14 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 1220224..5b62c5c 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1408,6 +1408,8 @@ long arch_do_domctl(
case XEN_DOMCTL_psr_cat_op:
switch ( domctl->u.psr_cat_op.cmd )
{
+ uint32_t val32;
+
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
domctl->u.psr_cat_op.data,
@@ -1427,23 +1429,23 @@ long arch_do_domctl(
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L3);
+ domctl->u.psr_cat_op.data = val32;
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L3_CODE);
+ domctl->u.psr_cat_op.data = val32;
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L3_DATA);
+ domctl->u.psr_cat_op.data = val32;
copyback = 1;
break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 1ccbd00..7583f4d 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -510,10 +510,55 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
return -EINVAL;
}
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type)
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint32_t *val, enum cbm_type type)
{
- return 0;
+ const struct psr_socket_info *info = get_socket_info(socket);
+ const struct feat_node *feat;
+ enum psr_feat_type feat_type;
+ unsigned int cos, i;
+
+ ASSERT(val);
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( feat_type >= ARRAY_SIZE(info->features) )
+ return -ENOENT;
+
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ if ( !feat_props[feat_type] )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ cos = d->arch.psr_cos_ids[socket];
+ /*
+ * If input cos exceeds current feature's cos_max, we should return its
+ * default value which is stored in cos 0. This case only happens
+ * when more than two features enabled concurrently and at least one
+ * features's cos_max is bigger than others. When a domain's working cos
+ * id is bigger than some features' cos_max, HW automatically works as
+ * default value for those features which cos_max is smaller.
+ */
+ if ( cos > feat->cos_max )
+ cos = 0;
+
+ for ( i = 0; i < feat_props[feat_type]->cos_num; i++ )
+ {
+ if ( type == feat_props[feat_type]->type[i] )
+ {
+ *val = feat->cos_reg_val[cos * feat_props[feat_type]->cos_num + i];
+ return 0;
+ }
+ }
+
+ return -EINVAL;
}
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 9625e90..ac9118e 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -71,8 +71,8 @@ void psr_ctxt_switch_to(struct domain *d);
int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type);
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint32_t *val, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (6 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-28 7:14 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
` (14 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
As set value flow is the most complicated one in psr, it will be
divided to some patches to make things clearer. This patch
implements the set value framework to show a whole picture firstly.
It also changes domctl interface to make it more general.
To make the set value flow be general and can support multiple features
at same time, it includes below steps:
1. Get COS ID that current domain is using.
2. Gather a value array to store all features current value
into it and replace the current value of the feature which is
being set to the new input value.
3. Find if there is already a COS ID on which all features'
values are same as the array. Then, we can reuse this COS
ID.
4. If fail to find, we need pick an available COS ID. Only COS ID which ref
is 0 or 1 can be picked.
5. Write the feature's MSRs according to the COS ID.
6. Update ref according to COS ID.
7. Save the COS ID into current domain's psr_cos_ids[socket] so that we
can know which COS the domain is using on the socket.
8. Set dom_ids bit corresponding to the domain so that we can know the domain
has been set and the COS ID of the domain is valid.
So, some functions are abstracted and the callback functions will be
implemented in next patches.
Here is an example to understand the process. The CPU supports
two featuers, e.g. L3 CAT and L2 CAT. User wants to set L3 CAT
of Dom1 to 0x1ff.
1. At the initial time, the old_cos of Dom1 is 0. The COS registers values
are below at this time.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | 0x7ff | 0x7ff | ... |
-------------------------------
L2 CAT | 0xff | 0xff | 0xff | ... |
-------------------------------
2. Gather the value array and insert new value into it:
val[0]: 0x1ff
val[1]: 0xff
3. It cannot find a matching COS.
4. Pick COS 1 to store the value set.
5. Write the L3 CAT COS 1 registers. The COS registers values are
changed to below now.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | 0x1ff | ... | ... |
-------------------------------
L2 CAT | 0xff | 0xff | ... | ... |
-------------------------------
6. The ref[1] is increased to 1 because Dom1 is using it now.
7. Save 1 to Dom1's psr_cos_ids[socket].
8. Set the bit in 'dom_ids[]'.
Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
of Dom2 is 0 too. Repeat above flow.
The val array assembled is:
val[0]: 0x1ff
val[1]: 0xff
So, it can find a matching COS, COS 1. Then, it can reuse COS 1
for Dom2.
The ref[1] is increased to 2 now because both Dom1 and Dom2 are
using this COS ID. Set 1 to Dom2's psr_cos_ids[socket].
There is one thing need to emphasize that we need restore domain's COS ID to
0 when socket is offline. Otherwise, a wrong COS ID will be used when the
socket is online again. That may cause user see the wrong CBM shown. But it
takes much time to iterate all domains to restore COS ID to 0. So, we define
a 'dom_ids[]' to represents all domains, one bit corresponds to one domain.
If the bit is 0 when entering 'psr_ctxt_switch_to', that means this is the
first time the domain is switched to this socket or domain's COS ID has not
been set since the socket is online. So, the COS ID set to ASSOC register on
this socket should be default value, 0. If not, that means the domain's COS
ID has been set when the socket was online. So, this COS ID is valid and we
can directly use it. We restore the domain's COS ID to 0 if the bit
corresponding to the domain is 0 but the domain's COS ID is not 0 when
'psr_get_val' is called. This can avoid CPU serialization if restoring action
is exectued in 'psr_ctxt_switch_to'.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- remove the memebers position changes in 'psr_socket_info'.
(suggested by Jan Beulich)
- rename 'dom_ids' to 'dom_set'.
(suggested by Jan Beulich)
- call 'bitmap_zero' to clear bitmap.
(suggested by Jan Beulich)
- combine two if()-s in 'psr_ctxt_switch_to' and add comment.
(suggested by Jan Beulich)
- remove redundant check in 'psr_get_val'.
(suggested by Jan Beulich)
- use 'domain_lock()' to protect 'psr_cos_ids' so that the codes do not
depend on 'domctl_lock'.
(suggested by Jan Beulich)
- adjust codes to avoid cast in 'psr_set_val'.
(suggested by Jan Beulich)
- adjust codes to avoid duplication of error paths in 'psr_set_val'.
(suggested by Jan Beulich)
- clear the domain bit anyway in 'psr_free_cos'.
(suggested by Jan Beulich)
- use the default cos id when getting old_cos in 'psr_set_val'.
v11:
- define 'dom_ids[]' and implement related flows.
- restore domain cos id to 0 in 'psr_get_val'.
- rename 'write_psr_msr' to 'write_psr_msrs' and change its parameters to
handle value array the feature's all MSRs.
- fix coding style issue.
(suggested by Jan Beulich)
- do not need check 'cos' in ASSERT.
(suggested by Jan Beulich)
- rename 'insert_val_to_array' to 'insert_val_into_array'.
(suggested by Jan Beulich)
- remove 'ref_lock' from parameter list in 'find_cos' and 'pick_avail_cos'.
(suggested by Jan Beulich)
- remove ASSERT check to 'ref_lock' in 'find_cos' and 'pick_avail_cos'.
(suggested by Jan Beulich)
- fix a bug for checking 'feat_type'.
(suggested by Jan Beulich)
- move 'free_array' label.
(suggested by Jan Beulich)
- modify comments and commit message.
v10:
- restore domain cos id to 0 when socket is offline.
(suggested by Jan Beulich)
- check 'psr_cat_op.data' to make sure only lower 32 bits are valid.
(suggested by Jan Beulich)
- remove unnecessary fixed width type of parameters and variables.
(suggested by Jan Beulich)
- rename 'insert_new_val_to_array' to 'insert_val_to_array'.
(suggested by Jan Beulich)
- input 'ref_lock' pointer into functions to check if it has been locked.
(suggested by Jan Beulich)
- add comment to declare the set process is protected by 'domctl_lock'.
(suggested by Jan Beulich)
- check 'feat_type'.
(suggested by Jan Beulich)
- remove 'feat_mask'.
(suggested by Jan Beulich)
- remove unnecessary criteria of ASSERT.
(suggested by Jan Beulich)
- adjust flow of 'psr_set_val' to avoid 'goto' for successful cases.
(suggested by Jan Beulich)
- use ASSERT to check 'socket_info' in 'psr_free_cos'.
(suggested by Jan Beulich)
- remove unnecessary comment in 'psr_free_cos'.
(suggested by Jan Beulich)
v9:
- use goto style error handling in 'psr_set_val'.
(suggested by Wei Liu)
- use ASSERT for checking old_cos.
(suggested by Wei Liu and Jan Beulich)
- fix coding style issue.
(suggested by Wei Liu)
- rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
(suggested by Wei Liu)
- use 'spin_is_locked' to check ref_lock.
(suggested by Roger Pau)
- add an input parameter 'array_len' for 'write_psr_msr'.
- check 'socket_info' and 'psr_cos_ids' in this patch.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- fix commit message words.
(suggested by Jan Beulich)
- change 'assemble_val_array' to 'gather_val_array'.
(suggested by Jan Beulich)
- change 'set_new_val_to_array' to 'insert_new_val_to_array'.
(suggested by Jan Beulich)
- change parameter 'm' of 'insert_new_val_to_array' to 'new_val'.
(suggested by Jan Beulich)
- change 'write_psr_msr' to 'write_psr_msrs'.
(suggested by Jan Beulich)
- correct comments.
(suggested by Jan Beulich)
- remove unnecessary comments.
(suggested by Jan Beulich)
- adjust conditions after 'find_cos' to save a level of indentation.
(suggested by Jan Beulich)
- add 'ASSERT(!old_cos || ref[old_cos])'.
(suggested by Jan Beulich)
- move ASSERT() check into locked region.
(suggested by Jan Beulich)
- replace parameter '*val' to 'val[]' in some functions.
(suggested by Jan Beulich)
- change 'write_psr_msr' parameters to prepare to only set one new value
for one feature.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
- add explanation about context switch.
(suggested by Jan Beulich)
v5:
- modify commit message.
(suggested by Jan Beulich)
- return an error for all helper functions in set flow.
(suggested by Jan Beulich)
- remove unnecessary cast.
(suggested by Jan Beulich)
- divide 'get_old_set_new' to two functions, 'assemble_val_array' and
'set_new_val_to_array'.
(suggested by Jan Beulich)
- modify comments.
(suggested by Jan Beulich)
- adjust code format.
(suggested by Jan Beulich)
- change 'alloc_new_cos' to 'pick_avail_cos' to make name accurate.
(suggested by Jan Beulich)
- check feature type when entering 'psr_set_val'.
(suggested by Jan Beulich)
- use ASSERT to check ref.
(suggested by Jan Beulich)
- rename 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/domctl.c | 18 ++--
xen/arch/x86/psr.c | 255 ++++++++++++++++++++++++++++++++++++++++++++--
xen/include/asm-x86/psr.h | 4 +-
3 files changed, 257 insertions(+), 20 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 5b62c5c..124a1c6 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1411,21 +1411,21 @@ long arch_do_domctl(
uint32_t val32;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_CODE);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_DATA);
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 7583f4d..fb84223 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -117,6 +117,7 @@ static const struct feat_props {
* ref_lock - A lock to protect cos_ref.
* cos_ref - A reference count array to record how many domains are using the
* COS ID. Every entry of cos_ref corresponds to one COS ID.
+ * dom_set - A bitmap to indicate which domain's cos id has been set.
*/
struct psr_socket_info {
bool feat_init;
@@ -124,6 +125,8 @@ struct psr_socket_info {
struct feat_node *features[PSR_SOCKET_FEAT_NUM];
spinlock_t ref_lock;
unsigned int cos_ref[MAX_COS_REG_CNT];
+ /* Every bit corresponds to a domain. Index is domain_id. */
+ DECLARE_BITMAP(dom_set, DOMID_IDLE + 1);
};
struct psr_assoc {
@@ -179,6 +182,10 @@ static void free_socket_resources(unsigned int socket)
}
info->feat_init = false;
+
+ memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
+
+ bitmap_zero(info->dom_set, DOMID_IDLE + 1);
}
static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
@@ -449,13 +456,25 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
- /* If domain's 'psr_cos_ids' is NULL, we set default value for it. */
+ /*
+ * If the domain is not set in 'dom_set' bitmap, that means the domain's
+ * cos id is not valid. So, we have to use default value (0) to set ASSOC
+ * register. Furthermore, if domain's 'psr_cos_ids' is NULL, we need
+ * default value for it too (for case that the domain's psr_cos_ids is not
+ * successfully allocated).
+ */
if ( psra->cos_mask )
- reg = psr_assoc_cos(reg,
- (d->arch.psr_cos_ids ?
- d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
- 0),
- psra->cos_mask);
+ {
+ unsigned int socket = cpu_to_socket(smp_processor_id());
+ struct psr_socket_info *info = socket_info + socket;
+ unsigned int cos = 0;
+
+ if ( likely(test_bit(d->domain_id, info->dom_set)) &&
+ d->arch.psr_cos_ids )
+ cos = d->arch.psr_cos_ids[socket];
+
+ reg = psr_assoc_cos(reg, cos, psra->cos_mask);
+ }
if ( reg != psra->val )
{
@@ -537,7 +556,16 @@ int psr_get_val(struct domain *d, unsigned int socket,
return -ENOENT;
}
+ domain_lock(d);
+ if ( !test_bit(d->domain_id, socket_info[socket].dom_set) )
+ {
+ d->arch.psr_cos_ids[socket] = 0;
+ set_bit(d->domain_id, socket_info[socket].dom_set);
+ }
+
cos = d->arch.psr_cos_ids[socket];
+ domain_unlock(d);
+
/*
* If input cos exceeds current feature's cos_max, we should return its
* default value which is stored in cos 0. This case only happens
@@ -561,15 +589,224 @@ int psr_get_val(struct domain *d, unsigned int socket,
return -EINVAL;
}
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type)
+/* Set value functions */
+static unsigned int get_cos_num(const struct psr_socket_info *info)
{
return 0;
}
-/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
+static int gather_val_array(uint32_t val[],
+ unsigned int array_len,
+ const struct psr_socket_info *info,
+ unsigned int old_cos)
+{
+ return -EINVAL;
+}
+
+static int insert_val_into_array(uint32_t val[],
+ unsigned int array_len,
+ const struct psr_socket_info *info,
+ enum psr_feat_type feat_type,
+ enum cbm_type type,
+ uint32_t new_val)
+{
+ return -EINVAL;
+}
+
+static int find_cos(const uint32_t val[], unsigned int array_len,
+ enum psr_feat_type feat_type,
+ const struct psr_socket_info *info)
+{
+ return -ENOENT;
+}
+
+static int pick_avail_cos(const struct psr_socket_info *info,
+ const uint32_t val[], unsigned int array_len,
+ unsigned int old_cos,
+ enum psr_feat_type feat_type)
+{
+ return -ENOENT;
+}
+
+static int write_psr_msrs(unsigned int socket, unsigned int cos,
+ uint32_t val[], unsigned int array_len,
+ enum psr_feat_type feat_type)
+{
+ return -ENOENT;
+}
+
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t new_val, enum cbm_type type)
+{
+ unsigned int old_cos, array_len;
+ int cos, ret;
+ unsigned int *ref;
+ uint32_t *val_array, val;
+ struct psr_socket_info *info = get_socket_info(socket);
+ enum psr_feat_type feat_type;
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ val = new_val;
+ if ( new_val != val )
+ return -EINVAL;
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( feat_type >= ARRAY_SIZE(info->features) ||
+ !info->features[feat_type] )
+ return -ENOENT;
+
+ /*
+ * Step 0:
+ * old_cos means the COS ID current domain is using. By default, it is 0.
+ *
+ * For every COS ID, there is a reference count to record how many domains
+ * are using the COS register corresponding to this COS ID.
+ * - If ref[old_cos] is 0, that means this COS is not used by any domain.
+ * - If ref[old_cos] is 1, that means this COS is only used by current
+ * domain.
+ * - If ref[old_cos] is more than 1, that mean multiple domains are using
+ * this COS.
+ */
+ domain_lock(d);
+ if ( !test_bit(d->domain_id, info->dom_set) )
+ {
+ d->arch.psr_cos_ids[socket] = 0;
+ set_bit(d->domain_id, info->dom_set);
+ }
+
+ old_cos = d->arch.psr_cos_ids[socket];
+ domain_unlock(d);
+
+ ASSERT(old_cos < MAX_COS_REG_CNT);
+
+ ref = info->cos_ref;
+
+ /*
+ * Step 1:
+ * Gather a value array to store all features cos_reg_val[old_cos].
+ * And, set the input new val into array according to the feature's
+ * position in array.
+ */
+ array_len = get_cos_num(info);
+ val_array = xzalloc_array(uint32_t, array_len);
+ if ( !val_array )
+ return -ENOMEM;
+
+ if ( (ret = gather_val_array(val_array, array_len, info, old_cos)) != 0 )
+ goto free_array;
+
+ if ( (ret = insert_val_into_array(val_array, array_len, info,
+ feat_type, type, val)) != 0 )
+ goto free_array;
+
+ spin_lock(&info->ref_lock);
+
+ /*
+ * Step 2:
+ * Try to find if there is already a COS ID on which all features' values
+ * are same as the array. Then, we can reuse this COS ID.
+ */
+ cos = find_cos(val_array, array_len, feat_type, info);
+ if ( cos == old_cos )
+ {
+ ret = 0;
+ goto unlock_free_array;
+ }
+
+ /*
+ * Step 3:
+ * If fail to find, we need pick an available COS ID.
+ * In fact, only COS ID which ref is 1 or 0 can be picked for current
+ * domain. If old_cos is not 0 and its ref==1, that means only current
+ * domain is using this old_cos ID. So, this old_cos ID certainly can
+ * be reused by current domain. Ref==0 means there is no any domain
+ * using this COS ID. So it can be used for current domain too.
+ */
+ if ( cos < 0 )
+ {
+ cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
+ if ( cos < 0 )
+ {
+ ret = cos;
+ goto unlock_free_array;
+ }
+
+ /*
+ * Step 4:
+ * Write the feature's MSRs according to the COS ID.
+ */
+ ret = write_psr_msrs(socket, cos, val_array, array_len, feat_type);
+ if ( ret )
+ goto unlock_free_array;
+ }
+
+ /*
+ * Step 5:
+ * Find the COS ID (find_cos result is '>= 0' or an available COS ID is
+ * picked, then update ref according to COS ID.
+ */
+ ref[cos]++;
+ ASSERT(!cos || ref[cos]);
+ ASSERT(!old_cos || ref[old_cos]);
+ ref[old_cos]--;
+ spin_unlock(&info->ref_lock);
+
+ /*
+ * Step 6:
+ * Save the COS ID into current domain's psr_cos_ids[] so that we can know
+ * which COS the domain is using on the socket. One domain can only use
+ * one COS ID at same time on each socket.
+ */
+ domain_lock(d);
+ d->arch.psr_cos_ids[socket] = cos;
+ domain_unlock(d);
+
+ /*
+ * Step 7:
+ * Then, set the dom_set bit which corresponds to domain_id to mark this
+ * domain has been set and the COS ID of the domain is valid.
+ */
+ set_bit(d->domain_id, info->dom_set);
+
+ goto free_array;
+
+ unlock_free_array:
+ spin_unlock(&info->ref_lock);
+
+ free_array:
+ xfree(val_array);
+ return ret;
+}
+
static void psr_free_cos(struct domain *d)
{
+ unsigned int socket, cos;
+
+ ASSERT(socket_info);
+
+ if ( !d->arch.psr_cos_ids )
+ return;
+
+ /* Domain is destroyed so its cos_ref should be decreased. */
+ for ( socket = 0; socket < nr_sockets; socket++ )
+ {
+ struct psr_socket_info *info = socket_info + socket;
+
+ clear_bit(d->domain_id, info->dom_set);
+
+ /* cos 0 is default one which does not need be handled. */
+ cos = d->arch.psr_cos_ids[socket];
+ if ( cos == 0 )
+ continue;
+
+ spin_lock(&info->ref_lock);
+ ASSERT(info->cos_ref[cos]);
+ info->cos_ref[cos]--;
+ spin_unlock(&info->ref_lock);
+ }
+
xfree(d->arch.psr_cos_ids);
d->arch.psr_cos_ids = NULL;
}
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index ac9118e..15b9a25 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -73,8 +73,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
int psr_get_val(struct domain *d, unsigned int socket,
uint32_t *val, enum cbm_type type);
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type);
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t val, enum cbm_type type);
void psr_domain_init(struct domain *d);
void psr_domain_free(struct domain *d);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (7 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-29 17:56 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
` (13 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Only can one COS ID be used by one domain at one time. That means all enabled
features' COS registers at this COS ID are valid for this domain at that time.
When user updates a feature's value, we need make sure all other features'
values are not affected. So, we firstly need gather an array which contains
all features current values and replace the setting feature's value in array
to new value.
Then, we can try to find if there is a COS ID on which all features' COS
registers values are same as the array. If we can find, we just use this COS
ID. If fail to find, we need pick a new COS ID.
This patch implements value array assembling flow.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- use 'ARRAY_SIZE' to calculate array boundary.
(suggested by Jan Beulich)
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exist but props does not exist.
(suggested by Jan Beulich)
- return 0 only when value is set in 'insert_val_into_array'.
(suggested by Jan Beulich)
v11:
- changes about 'feat_props'.
(suggested by Jan Beulich)
- changes about 'get_val'. It is replaced by generic codes.
(suggested by Jan Beulich)
v10:
- remove 'get_old_val' to directly call 'get_val' to get needed val.
(suggested by Jan Beulich)
- move 'psr_check_cbm' into 'insert_val_to_array'.
(suggested by Jan Beulich)
- change type of 'cbm' in 'psr_check_cbm' to 'unsigned long'.
(suggested by Jan Beulich)
- remove 'set_new_val' as it can be handled in generic process.
- changes related to 'feat_props'.
(suggested by Jan Beulich)
- adjust flow in 'gather_val_array' to avoid array cross.
(suggested by Jan Beulich)
- adjust flow in 'insert_val_to_array' to avoid array cross.
(suggested by Jan Beulich)
v9:
- add comments about boundary checking.
(suggested by Wei Liu)
- rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
(suggested by Wei Liu)
- rename 'l3_cat_get_cos_num' to 'cat_get_cos_num' to cover all L3/L2 CAT
features.
(suggested by Roger Pau)
- rename 'l3_cat_get_old_val' to 'cat_get_old_val' to cover all L3/L2 CAT
features and reuse cat_get_val in it.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- replace 'm' to 'new_val'.
(suggested by Jan Beulich)
- move cos check outside callback function.
(suggested by Jan Beulich)
- remove 'get_cos_num' callback function.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v6:
- change 'assemble_val_array' to 'combine_val_array'.
(suggested by Konrad Rzeszutek Wilk)
- check return value of 'get_old_val'.
(suggested by Konrad Rzeszutek Wilk)
- replace some 'EINVAL' to 'ENOSPC'.
(suggested by Konrad Rzeszutek Wilk)
v5:
- modify comments according to changes of codes.
(suggested by Jan Beulich)
- change 'bool_t' to 'bool'.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 128 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 126 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index fb84223..70e7c41 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -204,6 +204,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
return feat_type;
}
+static bool psr_check_cbm(unsigned int cbm_len, unsigned long cbm)
+{
+ unsigned int first_bit, zero_bit;
+
+ /* Set bits should only in the range of [0, cbm_len]. */
+ if ( cbm & (~0ul << cbm_len) )
+ return false;
+
+ /* At least one bit need to be set. */
+ if ( cbm == 0 )
+ return false;
+
+ first_bit = find_first_bit(&cbm, cbm_len);
+ zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
+
+ /* Set bits should be contiguous. */
+ if ( zero_bit < cbm_len &&
+ find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
+ return false;
+
+ return true;
+}
+
/* CAT common functions implementation. */
static void cat_init_feature(const struct cpuid_leaf *regs,
struct feat_node *feat,
@@ -592,7 +615,14 @@ int psr_get_val(struct domain *d, unsigned int socket,
/* Set value functions */
static unsigned int get_cos_num(const struct psr_socket_info *info)
{
- return 0;
+ unsigned int num = 0, i;
+
+ /* Get all features total amount. */
+ for ( i = 0; i < ARRAY_SIZE(feat_props); i++ )
+ if ( feat_props[i] )
+ num += feat_props[i]->cos_num;
+
+ return num;
}
static int gather_val_array(uint32_t val[],
@@ -600,7 +630,47 @@ static int gather_val_array(uint32_t val[],
const struct psr_socket_info *info,
unsigned int old_cos)
{
- return -EINVAL;
+ unsigned int i;
+
+ if ( !val )
+ return -EINVAL;
+
+ /* Get all features current values according to old_cos. */
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ unsigned int cos = old_cos, j;
+ const struct feat_node *feat = info->features[i];
+ const struct feat_props *props = feat_props[i];
+
+ if ( !feat )
+ continue;
+
+ if ( !props )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( array_len < props->cos_num )
+ return -ENOSPC;
+
+ /*
+ * If old_cos exceeds current feature's cos_max, we should get
+ * default value. So assign cos to 0 which stores default value.
+ */
+ if ( cos > feat->cos_max )
+ cos = 0;
+
+ /* Value getting order is same as feature array. */
+ for ( j = 0; j < props->cos_num; j++ )
+ val[j] = feat->cos_reg_val[cos * props->cos_num + j];
+
+ array_len -= props->cos_num;
+
+ val += props->cos_num;
+ }
+
+ return 0;
}
static int insert_val_into_array(uint32_t val[],
@@ -610,6 +680,60 @@ static int insert_val_into_array(uint32_t val[],
enum cbm_type type,
uint32_t new_val)
{
+ const struct feat_node *feat;
+ const struct feat_props *props;
+ unsigned int i;
+
+ ASSERT(feat_type < PSR_SOCKET_FEAT_NUM);
+
+ /* Insert new value into array according to feature's position in array. */
+ for ( i = 0; i < feat_type; i++ )
+ {
+ if ( !info->features[i] )
+ continue;
+
+ props = feat_props[i];
+ if ( !props )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( array_len <= props->cos_num )
+ return -ENOSPC;
+
+ array_len -= props->cos_num;
+
+ val += props->cos_num;
+ }
+
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ props = feat_props[feat_type];
+ if ( !props )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( array_len < props->cos_num )
+ return -ENOSPC;
+
+ if ( !psr_check_cbm(feat->cbm_len, new_val) )
+ return -EINVAL;
+
+ /* Value setting position is same as feature array. */
+ for ( i = 0; i < props->cos_num; i++ )
+ {
+ if ( type == props->type[i] )
+ {
+ val[i] = new_val;
+ return 0;
+ }
+ }
+
return -EINVAL;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (8 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-29 17:57 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
` (12 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Continue from patch:
'x86: refactor psr: L3 CAT: set value: assemble features value array'
We can try to find if there is a COS ID on which all features' COS registers
values are same as the array assembled before.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- use 'ARRAY_SIZE' to calculate array boundary.
(suggested by Jan Beulich)
- move same type variables declaration to one line.
(suggested by Jan Beulich)
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exists but props does not exist.
(suggested by Jan Beulich)
- move 'i' definition into the loop to limit its scope.
(suggested by Jan Beulich)
v11:
- move 'compare_val' implementation from CDP patch to this.
(suggested by Jan Beulich)
- remove 'rc' in 'compare_val' to make codes clearer.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- fix bug.
(suggested by Jan Beulich)
- remove unnecessary variable initialization.
(suggested by Jan Beulich)
- remove unnecessary comment.
(suggested by Jan Beulich)
- adjust array check place prior to using the array.
(suggested by Jan Beulich)
- add comment.
v10:
- remove 'compare_val' hook and its CAT implementation. Make its
functionality be generic in 'find_cos' flow.
(suggested by Jan Beulich)
- changes related to 'props'.
(suggested by Jan Beulich)
- rename 'val_array' to 'val_ptr'.
(suggested by Jan Beulich)
- rename 'find' to 'found'.
(suggested by Jan Beulich)
- move some variables declaration and initialization into loop.
(suggested by Jan Beulich)
- adjust codes positions.
(suggested by Jan Beulich)
v9:
- modify comments of 'compare_val' to be same as current implementation.
(suggested by Wei Liu)
- fix indentation issue.
(suggested by Wei Liu)
- rename 'l3_cat_compare_val' to 'cat_compare_val' to cover all L3/L2 CAT
features.
(suggested by Roger Pau)
- remove parameter 'found' from 'cat_compare_val' and modify the return
values to let caller know if the id is found or not.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- replace 'get_cos_num' to 'feat->cos_num'.
(suggested by Jan Beulich)
- directly use 'cos_reg_val[0]' as default value.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v5:
- modify commit message to provide exact patch name to continue from.
(suggested by Jan Beulich)
- remove 'get_cos_max_from_type' because it can be replaced by
'get_cos_max'.
- move type check out from callback functions to caller.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- modify comments according to changes of codes.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 101 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 70e7c41..942ae63 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -737,10 +737,111 @@ static int insert_val_into_array(uint32_t val[],
return -EINVAL;
}
+static int compare_val(const uint32_t val[],
+ const struct feat_node *feat,
+ const struct feat_props *props,
+ unsigned int cos)
+{
+ unsigned int i;
+
+ for ( i = 0; i < props->cos_num; i++ )
+ {
+ uint32_t feat_val;
+
+ /* If cos is bigger than cos_max, we need compare default value. */
+ if ( cos > feat->cos_max )
+ {
+ /*
+ * COS ID 0 always stores the default value.
+ * For CDP:
+ * - DATA default value stored in cos_reg_val[0];
+ * - CODE default value stored in cos_reg_val[1].
+ */
+ feat_val = feat->cos_reg_val[i];
+
+ /*
+ * If cos is bigger than feature's cos_max, the val should be
+ * default value. Otherwise, it fails to find a COS ID. So we
+ * have to exit find flow.
+ */
+ if ( val[i] != feat_val )
+ return -EINVAL;
+ }
+ else
+ {
+ feat_val = feat->cos_reg_val[cos * props->cos_num + i];
+ if ( val[i] != feat_val )
+ return 0;
+ }
+ }
+
+ return 1;
+}
+
static int find_cos(const uint32_t val[], unsigned int array_len,
enum psr_feat_type feat_type,
const struct psr_socket_info *info)
{
+ unsigned int cos, cos_max;
+ const unsigned int *ref = info->cos_ref;
+ const struct feat_node *feat;
+
+ /* cos_max is the one of the feature which is being set. */
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ cos_max = feat->cos_max;
+
+ for ( cos = 0; cos <= cos_max; cos++ )
+ {
+ const uint32_t *val_ptr = val;
+ unsigned int len = array_len, i;
+ int rc = 0;
+
+ if ( cos && !ref[cos] )
+ continue;
+
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ const struct feat_props *props = feat_props[i];
+
+ feat = info->features[i];
+ if ( !feat )
+ continue;
+
+ if ( !props )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( len < props->cos_num )
+ return -ENOSPC;
+
+ /*
+ * Compare value according to feature array order.
+ * We must follow this order because value array is assembled
+ * as this order.
+ */
+ rc = compare_val(val_ptr, feat, props, cos);
+ if ( rc < 0 )
+ return rc;
+
+ /* If fail to match, go to next cos to compare. */
+ if ( !rc )
+ break;
+
+ len -= props->cos_num;
+
+ val_ptr += props->cos_num;
+ }
+
+ /* For this COS ID all entries in the values array do match. Use it. */
+ if ( rc )
+ return cos;
+ }
+
return -ENOENT;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (9 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-29 17:59 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
` (11 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Continue from previous patch:
'x86: refactor psr: L3 CAT: set value: implement cos finding flow.'
If fail to find a COS ID, we need pick a new COS ID for domain. Only COS ID
that ref[COS_ID] is 1 or 0 can be picked to input a new set feature values.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exists but props does not exist.
(suggested by Jan Beulich)
- declare same type vairables in one line.
(suggested by Jan Beulich)
- use 'ARRAY_SIZE' to calculate array boundary.
(suggested by Jan Beulich)
v11:
- remove unnecessary variable initialization.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- replace 'get_val' calling to generic codes.
(suggested by Jan Beulich)
v10:
- remove 'fits_cos_max' hook and CAT implementation. Move the process into
generic flow.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- adjust codes positions.
(suggested by Jan Beulich)
v9:
- modify return value of 'pick_avail_cos' to make it more accurate.
- rename 'l3_cat_fits_cos_max' to 'cat_fits_cos_max' to cover L3/L2 CAT
features.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- fix comment.
(suggested by Wei Liu)
- directly use 'cos_reg_val[0]' as default value.
(suggested by Jan Beulich)
- replace 'get_cos_num' to 'feat->cos_num'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v5:
- modify commit message to provide exact patch name to continue from.
(suggested by Jan Beulich)
- change 'exceeds_cos_max' to 'fits_cos_max' to be accurate.
(suggested by Jan Beulich)
- modify comments according to changes of codes.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- move type check out from callback functions to caller.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- modify code format.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 86 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 942ae63..5cbc389 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -845,12 +845,97 @@ static int find_cos(const uint32_t val[], unsigned int array_len,
return -ENOENT;
}
+static bool fits_cos_max(const uint32_t val[],
+ uint32_t array_len,
+ const struct psr_socket_info *info,
+ unsigned int cos)
+{
+ unsigned int i;
+
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ const struct feat_node *feat = info->features[i];
+ const struct feat_props *props = feat_props[i];
+
+ if ( !feat )
+ continue;
+
+ if ( !props )
+ {
+ ASSERT_UNREACHABLE();
+ return false;
+ }
+
+ if ( array_len < props->cos_num )
+ return false;
+
+ if ( cos > feat->cos_max )
+ {
+ unsigned int j;
+
+ for ( j = 0; j < props->cos_num; j++ )
+ {
+ /*
+ * Get default value which cos id is 0.
+ * For CDP:
+ * - DATA default value stored in cos_reg_val[0];
+ * - CODE default value stored in cos_reg_val[1].
+ */
+ uint32_t default_val = feat->cos_reg_val[j];
+
+ if ( val[j] != default_val )
+ return false;
+ }
+ }
+
+ array_len -= props->cos_num;
+
+ val += props->cos_num;
+ }
+
+ return true;
+}
+
static int pick_avail_cos(const struct psr_socket_info *info,
const uint32_t val[], unsigned int array_len,
unsigned int old_cos,
enum psr_feat_type feat_type)
{
- return -ENOENT;
+ unsigned int cos, cos_max = 0;
+ const struct feat_node *feat;
+ const unsigned int *ref = info->cos_ref;
+
+ /* cos_max is the one of the feature which is being set. */
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ cos_max = feat->cos_max;
+ if ( !cos_max )
+ return -ENOENT;
+
+ /* We cannot use id 0 because it stores the default values. */
+ if ( old_cos && ref[old_cos] == 1 &&
+ fits_cos_max(val, array_len, info, old_cos) )
+ return old_cos;
+
+ /* Find an unused one other than cos0. */
+ for ( cos = 1; cos <= cos_max; cos++ )
+ {
+ /*
+ * ref is 0 means this COS is not used by other domain and
+ * can be used for current setting.
+ */
+ if ( !ref[cos] )
+ {
+ if ( !fits_cos_max(val, array_len, info, cos) )
+ break;
+
+ return cos;
+ }
+ }
+
+ return -EOVERFLOW;
}
static int write_psr_msrs(unsigned int socket, unsigned int cos,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (10 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-29 18:00 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
` (10 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Continue from previous patch:
'x86: refactor psr: L3 CAT: set value: implement cos id picking flow.'
We have got the feature value and COS ID to set. Then, we write MSRs of the
designated feature.
Till now, set value process is completed.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- declare same type varaibles in one line.
(suggested by Jan Beulich)
- replace 'feat_type' to 'props' in 'struct cos_write_info'.
(suggested by Jan Beulich)
- assign the 'cos_num' to a local variable.
(suggested by Jan Beulich)
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exists but props does not exist.
(suggested by Jan Beulich)
v11:
- rename 'write_psr_msr' to 'write_psr_msrs'.
- rename 'do_write_psr_msr' to 'do_write_psr_msrs'.
- change parameters and codes of 'write_psr_msrs' to handle value array.
- add 'feat_type' in 'struct cos_write_info' to handle props array.
- in 'do_write_psr_msrs', write value array into msrs according to
'props->type[i]'.
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
- move check from 'do_write_psr_msrs' to 'write_psr_msrs'.
(suggested by Jan Beulich)
- change about 'cos_max'.
(suggested by Jan Beulich)
- change about 'feat_props'.
(suggested by Jan Beulich)
v10:
- remove 'type' from 'write_msr' parameter list. Will add it back when
implementing CDP.
(suggested by Jan Beulich)
- remove unnecessary casts.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
v9:
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- add 'array_len' in 'struct cos_write_info' and check if val array
exceeds it.
- modify 'write_psr_msr' flow only to set one value a time. No need to
set whole feature array values.
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- modify 'write_msr' callback function to 'void' because we have to set
all features' cbm. When input cos exceeds some features' cos_max, just
skip them but not break the iteration.
v5:
- modify commit message to provide exact patch name to continue from.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- move type check out from callback functions to caller.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- correct code format.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 91 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5cbc389..81d9a78 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -107,6 +107,9 @@ static const struct feat_props {
/* get_feat_info is used to return feature HW info through sysctl. */
bool (*get_feat_info)(const struct feat_node *feat,
uint32_t data[], unsigned int array_len);
+
+ /* write_msr is used to write out feature MSR register. */
+ void (*write_msr)(unsigned int cos, uint32_t val, enum cbm_type type);
} *feat_props[PSR_SOCKET_FEAT_NUM];
/*
@@ -278,10 +281,16 @@ static bool cat_get_feat_info(const struct feat_node *feat,
}
/* L3 CAT props */
+static void l3_cat_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
+{
+ wrmsrl(MSR_IA32_PSR_L3_MASK(cos), val);
+}
+
static const struct feat_props l3_cat_props = {
.cos_num = 1,
.type[0] = PSR_CBM_TYPE_L3,
.get_feat_info = cat_get_feat_info,
+ .write_msr = l3_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -938,11 +947,92 @@ static int pick_avail_cos(const struct psr_socket_info *info,
return -EOVERFLOW;
}
+static unsigned int get_socket_cpu(unsigned int socket)
+{
+ if ( likely(socket < nr_sockets) )
+ return cpumask_any(socket_cpumask[socket]);
+
+ return nr_cpu_ids;
+}
+
+struct cos_write_info
+{
+ unsigned int cos;
+ struct feat_node *feature;
+ uint32_t *val;
+ const struct feat_props *props;
+};
+
+static void do_write_psr_msrs(void *data)
+{
+ struct cos_write_info *info = data;
+ struct feat_node *feat = info->feature;
+ const struct feat_props *props = info->props;
+ unsigned int i, cos = info->cos, cos_num = props->cos_num;
+
+ for ( i = 0; i < cos_num; i++ )
+ {
+ if ( feat->cos_reg_val[cos * cos_num + i] != info->val[i] )
+ {
+ feat->cos_reg_val[cos * cos_num + i] = info->val[i];
+ props->write_msr(cos, info->val[i], props->type[i]);
+ }
+ }
+}
+
static int write_psr_msrs(unsigned int socket, unsigned int cos,
uint32_t val[], unsigned int array_len,
enum psr_feat_type feat_type)
{
- return -ENOENT;
+ unsigned int i;
+ struct psr_socket_info *info = get_socket_info(socket);
+ struct cos_write_info data =
+ {
+ .cos = cos,
+ .feature = info->features[feat_type],
+ .props = feat_props[feat_type],
+ };
+
+ if ( cos > info->features[feat_type]->cos_max )
+ return -EINVAL;
+
+ /* Skip to the feature's value head. */
+ for ( i = 0; i < feat_type; i++ )
+ {
+ if ( !info->features[i] )
+ continue;
+
+ if ( !feat_props[i] )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( array_len <= feat_props[feat_type]->cos_num )
+ return -ENOSPC;
+
+ array_len -= feat_props[feat_type]->cos_num;
+
+ val += feat_props[feat_type]->cos_num;
+ }
+
+ if ( array_len < feat_props[feat_type]->cos_num )
+ return -ENOSPC;
+
+ data.val = val;
+
+ if ( socket == cpu_to_socket(smp_processor_id()) )
+ do_write_psr_msrs(&data);
+ else
+ {
+ unsigned int cpu = get_socket_cpu(socket);
+
+ if ( cpu >= nr_cpu_ids )
+ return -ENOTSOCK;
+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msrs, &data, 1);
+ }
+
+ return 0;
}
int psr_set_val(struct domain *d, unsigned int socket,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (11 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-30 6:40 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
` (9 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the CPU init flow for CDP. The flow is almost
same as L3 CAT.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- move 'type[]' assignment into l3_cdp_props declaration to make it be
'const'.
(suggested by Jan Beulich)
- remove "L2 CAT" indication in printk.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- change 'val' type to uint64_t.
(suggested by Jan Beulich)
- use 1ull.
(suggested by Jan Beulich)
- restore mask(0) MSR to default value.
(suggested by Jan Beulich)
v11:
- changes about 'feat_props'.
(suggested by Jan Beulich)
- remove MSR restore action which is unnecessary.
(suggested by Jan Beulich)
- modify commit message.
v10:
- fix comment.
(suggested by Jan Beulich)
- use swith in 'cat_init_feature' to handle different feature types.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- restore MSRs to default value when cpu online.
(suggested by Jan Beulich)
- remove feat_mask.
(suggested by Jan Beulich)
v9:
- modify commit message to describe flow clearer.
- handle cpu offline and online again case to read MSRs registers values
back and save them into cos array to make user can get real data.
- modify error handling process in 'psr_cpu_prepare' to reduce redundant
codes.
- modify 'get_cdp_data' and 'get_cdp_code' to make them standard.
(suggested by Roger Pau and Jan Beulich)
- encapsulate CDP operations into 'cat_init_feature' to reduce redundant
codes.
(suggested by Roger Pau)
- reuse 'cat_get_cos_max' for CDP.
(suggested by Roger Pau)
- handle 'PSR_CDP' in psr_presmp_init to make init work can be done when
there is only 'psr=cdp' in cmdline.
- remove unnecessary comment.
(suggested by Jan Beulich)
- move CDP related codes in 'cpu_init_work' into 'psr_cpu_init'.
(suggested by Jan Beulich)
- add codes to handle CDP's 'cos_num'.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- do not free resources when allocation fails in 'psr_cpu_prepare'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v7:
- initialize 'l3_cdp'.
(suggested by Konrad Rzeszutek Wilk)
v6:
- use 'cpuid_leaf'.
(suggested by Konrad Rzeszutek Wilk and Jan Beulich)
v5:
- remove codes to free 'feat_l3_cdp' in 'free_feature'.
(suggested by Jan Beulich)
- encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
(suggested by Jan Beulich)
- print socket info when 'opt_cpu_info' is true.
(suggested by Jan Beulich)
- rename 'l3_cdp_get_max_cos_max' to 'l3_cdp_get_cos_max'.
(suggested by Jan Beulich)
- rename 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
- move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 67 insertions(+), 7 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 81d9a78..814f0e1 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -62,6 +62,7 @@
enum psr_feat_type {
PSR_SOCKET_L3_CAT,
+ PSR_SOCKET_L3_CDP,
PSR_SOCKET_FEAT_NUM,
PSR_SOCKET_FEAT_UNKNOWN,
};
@@ -152,11 +153,28 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
* array creation. It is used to transiently store a spare node.
*/
static struct feat_node *feat_l3_cat;
+static struct feat_node *feat_l3_cdp;
/* Common functions */
#define cat_default_val(len) (0xffffffff >> (32 - (len)))
/*
+ * get_cdp_data - get DATA COS register value from input COS ID.
+ * @feat: the feature node.
+ * @cos: the COS ID.
+ */
+#define get_cdp_data(feat, cos) \
+ ((feat)->cos_reg_val[(cos) * 2])
+
+/*
+ * get_cdp_code - get CODE COS register value from input COS ID.
+ * @feat: the feature node.
+ * @cos: the COS ID.
+ */
+#define get_cdp_code(feat, cos) \
+ ((feat)->cos_reg_val[(cos) * 2 + 1])
+
+/*
* Use this function to check if any allocation feature has been enabled
* in cmdline.
*/
@@ -253,6 +271,26 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
break;
+ case PSR_SOCKET_L3_CDP:
+ {
+ uint64_t val;
+
+ /* Cut half of cos_max when CDP is enabled. */
+ feat->cos_max >>= 1;
+
+ wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
+ wrmsrl(MSR_IA32_PSR_L3_MASK(1), cat_default_val(feat->cbm_len));
+ rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
+ wrmsrl(MSR_IA32_PSR_L3_QOS_CFG,
+ val | (1ull << PSR_L3_QOS_CDP_ENABLE_BIT));
+
+ /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+ get_cdp_code(feat, 0) = cat_default_val(feat->cbm_len);
+ get_cdp_data(feat, 0) = cat_default_val(feat->cbm_len);
+
+ break;
+ }
+
default:
return;
}
@@ -263,7 +301,8 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
if ( !opt_cpu_info )
return;
- printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ ((type == PSR_SOCKET_L3_CDP) ? "CDP" : "L3 CAT"),
cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
}
@@ -293,6 +332,13 @@ static const struct feat_props l3_cat_props = {
.write_msr = l3_cat_write_msr,
};
+/* L3 CDP props */
+static const struct feat_props l3_cdp_props = {
+ .cos_num = 2,
+ .type[0] = PSR_CBM_TYPE_L3_DATA,
+ .type[1] = PSR_CBM_TYPE_L3_CODE,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1263,6 +1309,10 @@ static int psr_cpu_prepare(void)
(feat_l3_cat = xzalloc(struct feat_node)) == NULL )
return -ENOMEM;
+ if ( feat_l3_cdp == NULL &&
+ (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
@@ -1294,11 +1344,21 @@ static void psr_cpu_init(void)
{
cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
- feat = feat_l3_cat;
- feat_l3_cat = NULL;
- feat_props[PSR_SOCKET_L3_CAT] = &l3_cat_props;
-
- cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CAT);
+ if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
+ !info->features[PSR_SOCKET_L3_CDP] )
+ {
+ feat = feat_l3_cdp;
+ feat_l3_cdp = NULL;
+ feat_props[PSR_SOCKET_L3_CDP] = &l3_cdp_props;
+ cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CDP);
+ }
+ else
+ {
+ feat = feat_l3_cat;
+ feat_l3_cat = NULL;
+ feat_props[PSR_SOCKET_L3_CAT] = &l3_cat_props;
+ cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CAT);
+ }
info->feat_init = true;
}
@@ -1360,7 +1420,7 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- if ( opt_psr & PSR_CAT )
+ if ( opt_psr & (PSR_CAT | PSR_CDP) )
init_psr();
if ( psr_cpu_prepare() )
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 14/23] x86: refactor psr: CDP: implement get hw info flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (12 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-30 6:41 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
` (8 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements get HW info flow for CDP including L3 CDP callback
function. The flow is almost same as L3 CAT.
With this patch, 'psr-hwinfo' can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- remove input parameter 'info' in 'psr_cbm_type_to_feat_type'. Use
'feat_props' array to check.
(suggested by Jan Beulich)
v11:
- modify 'psr_get_info' flow to make it simple to cover CDP case.
v10:
- update renamed macros used by psr_get_info.
(suggested by Jan Beulich)
- change 'psr_get_info' flow to cover CDP case to make codes in sysctl
more simple.
(suggested by Jan Beulich)
- remove sysctl redundant codes after applying above changes.
(suggested by Jan Beulich)
v9:
- modify commit message to explain flow more clearly.
- reuse 'cat_get_feat_info' for CDP to reduce redundant codes.
(suggested by Roger Pau)
- fix coding style issues.
(suggested by Wei Liu and Roger Pau)
- rename macros used by psr_get_info to make them meaningful.
(suggested by Jan Beulich)
v5:
- rename 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
- remove type check in callback function.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 814f0e1..aee6e3e 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -217,7 +217,21 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
{
case PSR_CBM_TYPE_L3:
feat_type = PSR_SOCKET_L3_CAT;
+
+ /*
+ * If type is L3 CAT but we cannot find it in feat_props array,
+ * try CDP.
+ */
+ if ( !feat_props[feat_type] )
+ feat_type = PSR_SOCKET_L3_CDP;
+
break;
+
+ case PSR_CBM_TYPE_L3_DATA:
+ case PSR_CBM_TYPE_L3_CODE:
+ feat_type = PSR_SOCKET_L3_CDP;
+ break;
+
default:
ASSERT_UNREACHABLE();
}
@@ -333,10 +347,22 @@ static const struct feat_props l3_cat_props = {
};
/* L3 CDP props */
+static bool l3_cdp_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], uint32_t array_len)
+{
+ if ( !cat_get_feat_info(feat, data, array_len) )
+ return false;
+
+ data[PSR_INFO_IDX_CAT_FLAG] |= XEN_SYSCTL_PSR_CAT_L3_CDP;
+
+ return true;
+}
+
static const struct feat_props l3_cdp_props = {
.cos_num = 2,
.type[0] = PSR_CBM_TYPE_L3_DATA,
.type[1] = PSR_CBM_TYPE_L3_CODE,
+ .get_feat_info = l3_cdp_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (13 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-30 6:42 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
` (7 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L3 CDP set value related callback function.
With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- add comment to explain how to deal with the case that user set new val
for both DATA and CODE at same time.
- add parameter for 'psr_cbm_type_to_feat_type' to return the feature type
according to it.
- use the feature type returned by 'psr_cbm_type_to_feat_type' to check
if we need insert the new value into all items of the feature value array.
- use conditional expression for wrmsrl.
(suggested by Jan Beulich)
v11:
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
- changes about setting both CDP DATA and CODE at same time.
- move 'type[]' declaration into previous patch which introduced 'cos_num'.
(suggested by Jan Beulich)
- changes about 'type[]'.
(suggested by Jan Beulich)
- move 'compare_val' to previous patch.
(suggested by Jan Beulich)
- changes about 'get_val' which has been replace by generic codes.
(suggested by Jan Beulich)
- remove 'restore_default_val' which is unnecessary now.
(suggested by Jan Beulich)
v10:
- remove 'l3_cdp_get_old_val' and use 'l3_cdp_get_val' to replace it.
(suggested by Jan Beulich)
- remove 'l3_cdp_set_new_val'.
- modify 'insert_val_to_array' flow to handle multiple COSs case.
(suggested by Jan Beulich)
- remove 'l3_cdp_compare_val' and implement a generic function
'comapre_val'.
(suggested by Jan Beulich)
- remove 'l3_cdp_fits_cos_max'.
(suggested by Jan Beulich)
- introduce macro 'PSR_MAX_COS_NUM'.
- introduce a new member in 'feat_props', 'type[PSR_MAX_COS_NUM]' to record
all 'cbm_type' the feature has.
(suggested by Jan Beulich)
- modify 'gather_val_array' flow to handle multiple COSs case.
(suggested by Jan Beulich)
- modify 'find_cos' flow and implement 'compare_val' to handle multiple
COSs case.
(suggested by Jan Beulich)
- modify 'fits_cos_max' flow to handle multiple COSs case.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- remove cast in 'l3_cdp_write_msr'.
(suggested by Jan Beulich)
- implement 'compare_val' function to compare if feature values are what
we expect in finding flow.
- implement 'restore_default_val' function to restore feature's COS values
to default if the feature has multiple COSs. It is called when the COS
ID is reduced to 0.
v9:
- add comment to explain why CDP uses 2 COSs.
(suggested by Wei Liu)
- use 'cat_default_val'.
(suggested by Wei Liu)
- remove 'l3_cdp_get_cos_num' because we can directly get cos_num from
feat_node now.
(suggested by Jan Beulich)
- remove cos checking because it has been moved to common function.
(suggested by Jan Beulich)
- l3_cdp_set_new_val parameter 'm' is changed to 'new_val'.
(suggested by Jan Beulich)
- directly use get_cdp_data(feat, 0) and get_cdp_code(feat, 0) to get
default value.
(suggested by Jan Beulich)
- modify 'l3_cdp_write_msr' flow to write value into register according
to input type.
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- modify 'l3_cdp_write_msr' type to 'void'.
v5:
- remove type check in callback function.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- remove 'l3_cdp_get_cos_max_from_type'.
- rename 'l3_cdp_exceeds_cos_max' to 'l3_cdp_fits_cos_max'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 36 +++++++++++++++++++++++++++---------
1 file changed, 27 insertions(+), 9 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index aee6e3e..91b2122 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -209,7 +209,8 @@ static void free_socket_resources(unsigned int socket)
bitmap_zero(info->dom_set, DOMID_IDLE + 1);
}
-static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
+static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type,
+ bool strict)
{
enum psr_feat_type feat_type = PSR_SOCKET_FEAT_UNKNOWN;
@@ -222,7 +223,7 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
* If type is L3 CAT but we cannot find it in feat_props array,
* try CDP.
*/
- if ( !feat_props[feat_type] )
+ if ( !feat_props[feat_type] && !strict )
feat_type = PSR_SOCKET_L3_CDP;
break;
@@ -358,11 +359,20 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
return true;
}
+static void l3_cdp_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
+{
+ wrmsrl(((type == PSR_CBM_TYPE_L3_DATA) ?
+ MSR_IA32_PSR_L3_MASK_DATA(cos) :
+ MSR_IA32_PSR_L3_MASK_CODE(cos)),
+ val);
+}
+
static const struct feat_props l3_cdp_props = {
.cos_num = 2,
.type[0] = PSR_CBM_TYPE_L3_DATA,
.type[1] = PSR_CBM_TYPE_L3_CODE,
.get_feat_info = l3_cdp_get_feat_info,
+ .write_msr = l3_cdp_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -613,7 +623,7 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
if ( IS_ERR(info) )
return PTR_ERR(info);
- feat_type = psr_cbm_type_to_feat_type(type);
+ feat_type = psr_cbm_type_to_feat_type(type, false);
if ( feat_type >= ARRAY_SIZE(info->features) )
return -ENOENT;
@@ -646,7 +656,7 @@ int psr_get_val(struct domain *d, unsigned int socket,
if ( IS_ERR(info) )
return PTR_ERR(info);
- feat_type = psr_cbm_type_to_feat_type(type);
+ feat_type = psr_cbm_type_to_feat_type(type, false);
if ( feat_type >= ARRAY_SIZE(info->features) )
return -ENOENT;
@@ -764,6 +774,7 @@ static int insert_val_into_array(uint32_t val[],
const struct feat_node *feat;
const struct feat_props *props;
unsigned int i;
+ int ret = -EINVAL;
ASSERT(feat_type < PSR_SOCKET_FEAT_NUM);
@@ -805,17 +816,24 @@ static int insert_val_into_array(uint32_t val[],
if ( !psr_check_cbm(feat->cbm_len, new_val) )
return -EINVAL;
- /* Value setting position is same as feature array. */
+ /*
+ * Value setting position is same as feature array.
+ * For CDP, user may set both DATA and CODE to same value. For such case,
+ * user input 'PSR_CBM_TYPE_L3' as type. The strict feature type of
+ * 'PSR_CBM_TYPE_L3' is L3 CAT. So, we should set new_val to both of DATA
+ * and CODE under such case.
+ */
for ( i = 0; i < props->cos_num; i++ )
{
- if ( type == props->type[i] )
+ if ( type == props->type[i] ||
+ feat_type != psr_cbm_type_to_feat_type(type, true) )
{
val[i] = new_val;
- return 0;
+ ret = 0;
}
}
- return -EINVAL;
+ return ret;
}
static int compare_val(const uint32_t val[],
@@ -1124,7 +1142,7 @@ int psr_set_val(struct domain *d, unsigned int socket,
if ( new_val != val )
return -EINVAL;
- feat_type = psr_cbm_type_to_feat_type(type);
+ feat_type = psr_cbm_type_to_feat_type(type, false);
if ( feat_type >= ARRAY_SIZE(info->features) ||
!info->features[feat_type] )
return -ENOENT;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 16/23] x86: L2 CAT: implement CPU init flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (14 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-30 6:58 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 17/23] x86: L2 CAT: implement get hw info flow Yi Sun
` (6 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the CPU init flow for L2 CAT.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v12:
- move 'type[]' assignment into l2_cat_props declaration to make it be
'const'.
(suggested by Jan Beulich)
- add "L2 CAT" indicator in printk.
(suggested by Jan Beulich)
- restore mask(0) MSR to default value.
(suggested by Jan Beulich)
v11:
- move l2 cat 'type[]' assignement into 'psr_cpu_init'.
- remove COS MSR restore action in 'cpu_init_feature'.
- set 'feat_init' to true after CPU init.
- modify commit message.
v10:
- implement L2 CAT case in 'cat_init_feature'.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- introduce 'PSR_CBM_TYPE_L2'.
v9:
- modify error handling process in 'psr_cpu_prepare' to reduce redundant
codes.
- reuse 'cat_init_feature' and 'cat_get_cos_max' for L2 CAT to reduce
redundant codes.
(suggested by Roger Pau)
- remove unnecessary comment.
(suggested by Jan Beulich)
- move L2 CAT related codes from 'cpu_init_work' into 'psr_cpu_init'.
(suggested by Jan Beulich)
- do not free resource when allocation fails in 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v7:
- initialize 'l2_cat'.
(suggested by Konrad Rzeszutek Wilk)
v6:
- use 'struct cpuid_leaf'.
(suggested by Konrad Rzeszutek Wilk and Jan Beulich)
v5:
- remove 'feat_l2_cat' free in 'free_feature'.
(suggested by Jan Beulich)
- encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
(suggested by Jan Beulich)
- print socket info when 'opt_cpu_info' is true.
(suggested by Jan Beulich)
- rename 'l2_cat_get_max_cos_max' to 'l2_cat_get_cos_max'.
(suggested by Jan Beulich)
- rename 'dat[]' to 'data[]'
(suggested by Jan Beulich)
- move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 34 ++++++++++++++++++++++++++++++++--
xen/include/asm-x86/msr-index.h | 1 +
xen/include/asm-x86/psr.h | 2 ++
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 91b2122..60202b2 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -63,6 +63,7 @@
enum psr_feat_type {
PSR_SOCKET_L3_CAT,
PSR_SOCKET_L3_CDP,
+ PSR_SOCKET_L2_CAT,
PSR_SOCKET_FEAT_NUM,
PSR_SOCKET_FEAT_UNKNOWN,
};
@@ -154,6 +155,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
*/
static struct feat_node *feat_l3_cat;
static struct feat_node *feat_l3_cdp;
+static struct feat_node *feat_l2_cat;
/* Common functions */
#define cat_default_val(len) (0xffffffff >> (32 - (len)))
@@ -279,10 +281,14 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
switch ( type )
{
case PSR_SOCKET_L3_CAT:
+ case PSR_SOCKET_L2_CAT:
/* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
- wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
+ if ( type == PSR_SOCKET_L3_CAT )
+ wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
+ else
+ wrmsrl(MSR_IA32_PSR_L2_MASK(0), cat_default_val(feat->cbm_len));
break;
@@ -317,7 +323,8 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
return;
printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
- ((type == PSR_SOCKET_L3_CDP) ? "CDP" : "L3 CAT"),
+ ((type == PSR_SOCKET_L3_CDP) ? "CDP" :
+ ((type == PSR_SOCKET_L3_CAT) ? "L3 CAT": "L2 CAT")),
cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
}
@@ -375,6 +382,12 @@ static const struct feat_props l3_cdp_props = {
.write_msr = l3_cdp_write_msr,
};
+/* L2 CAT props */
+static const struct feat_props l2_cat_props = {
+ .cos_num = 1,
+ .type[0] = PSR_CBM_TYPE_L2,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1357,6 +1370,10 @@ static int psr_cpu_prepare(void)
(feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
return -ENOMEM;
+ if ( feat_l2_cat == NULL &&
+ (feat_l2_cat = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
@@ -1407,6 +1424,19 @@ static void psr_cpu_init(void)
info->feat_init = true;
}
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L2 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, ®s);
+
+ feat = feat_l2_cat;
+ feat_l2_cat = NULL;
+ feat_props[PSR_SOCKET_L2_CAT] = &l2_cat_props;
+ cat_init_feature(®s, feat, info, PSR_SOCKET_L2_CAT);
+
+ info->feat_init = true;
+ }
+
assoc_init:
psr_assoc_init();
}
diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h
index 771e750..6c49c6d 100644
--- a/xen/include/asm-x86/msr-index.h
+++ b/xen/include/asm-x86/msr-index.h
@@ -345,6 +345,7 @@
#define MSR_IA32_PSR_L3_MASK(n) (0x00000c90 + (n))
#define MSR_IA32_PSR_L3_MASK_CODE(n) (0x00000c90 + (n) * 2 + 1)
#define MSR_IA32_PSR_L3_MASK_DATA(n) (0x00000c90 + (n) * 2)
+#define MSR_IA32_PSR_L2_MASK(n) (0x00000d10 + (n))
/* Intel Model 6 */
#define MSR_P6_PERFCTR(n) (0x000000c1 + (n))
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 15b9a25..276fdd6 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -23,6 +23,7 @@
/* Resource Type Enumeration */
#define PSR_RESOURCE_TYPE_L3 0x2
+#define PSR_RESOURCE_TYPE_L2 0x4
/* L3 Monitoring Features */
#define PSR_CMT_L3_OCCUPANCY 0x1
@@ -56,6 +57,7 @@ enum cbm_type {
PSR_CBM_TYPE_L3,
PSR_CBM_TYPE_L3_CODE,
PSR_CBM_TYPE_L3_DATA,
+ PSR_CBM_TYPE_L2,
};
extern struct psr_cmt *psr_cmt;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 17/23] x86: L2 CAT: implement get hw info flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (15 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-30 6:59 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 18/23] x86: L2 CAT: implement get value flow Yi Sun
` (5 subsequent siblings)
22 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L2 CAT get HW info flow and interface in sysctl.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v10:
- modify macro name according to previous patch change.
(suggested by Jan Beulich)
- modify commit message.
v9:
- reuse 'cat_get_feat_info' for L2 CAT to reduce redundant codes.
(suggested by Roger Pau)
- modify sysctl implementation of L2 CAT to input data[3] to use
'cat_get_feat_info'.
(suggested by Roger Pau)
- modify macros names to newly defined ones.
(suggested by Jan Beulich)
- remove 'l2_info' to reuse 'l3_info'.
(suggested by Jan Beulich)
- modify macro name according to previous patch change.
(suggested by Jan Beulich)
v5:
- rename 'dat[]' to 'data[]'
(suggested by Jan Beulich)
- remove type check in callback function.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
---
xen/arch/x86/psr.c | 5 +++++
xen/arch/x86/sysctl.c | 23 +++++++++++++++++++++--
xen/include/public/sysctl.h | 1 +
3 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 60202b2..22de787 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -235,6 +235,10 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type,
feat_type = PSR_SOCKET_L3_CDP;
break;
+ case PSR_CBM_TYPE_L2:
+ feat_type = PSR_SOCKET_L2_CAT;
+ break;
+
default:
ASSERT_UNREACHABLE();
}
@@ -386,6 +390,7 @@ static const struct feat_props l3_cdp_props = {
static const struct feat_props l2_cat_props = {
.cos_num = 1,
.type[0] = PSR_CBM_TYPE_L2,
+ .get_feat_info = cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index c23270d..5e91755 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -174,10 +174,10 @@ long arch_do_sysctl(
case XEN_SYSCTL_psr_cat_op:
switch ( sysctl->u.psr_cat_op.cmd )
{
- case XEN_SYSCTL_PSR_CAT_get_l3_info:
- {
uint32_t data[PSR_INFO_ARRAY_SIZE];
+ case XEN_SYSCTL_PSR_CAT_get_l3_info:
+ {
ret = psr_get_info(sysctl->u.psr_cat_op.target,
PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
if ( ret )
@@ -195,6 +195,25 @@ long arch_do_sysctl(
break;
}
+ case XEN_SYSCTL_PSR_CAT_get_l2_info:
+ {
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L2, data, ARRAY_SIZE(data));
+ if ( ret )
+ break;
+
+ sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ data[PSR_INFO_IDX_COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ data[PSR_INFO_IDX_CAT_CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.flags =
+ data[PSR_INFO_IDX_CAT_FLAG];
+
+ if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
+ ret = -EFAULT;
+ break;
+ }
+
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index ee76a66..4c76d3a 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
#define XEN_SYSCTL_PSR_CAT_get_l3_info 0
+#define XEN_SYSCTL_PSR_CAT_get_l2_info 1
struct xen_sysctl_psr_cat_op {
uint32_t cmd; /* IN: XEN_SYSCTL_PSR_CAT_* */
uint32_t target; /* IN */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 18/23] x86: L2 CAT: implement get value flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (16 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 17/23] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 19/23] x86: L2 CAT: implement set " Yi Sun
` (4 subsequent siblings)
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L2 CAT get value interface in domctl.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v11:
- remove "get_val' assignment because it has been replaced by generic
codes.
(suggested by Jan Beulich)
v10:
- remove cast in domctl.
(suggested by Jan Beulich)
v9:
- reuse 'cat_get_val' for L2 CAT to reduce redundant codes
(suggested by Roger Pau)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v5:
- remove type check in callback function.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
---
xen/arch/x86/domctl.c | 7 +++++++
xen/include/public/domctl.h | 1 +
2 files changed, 8 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 124a1c6..2970118 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1449,6 +1449,13 @@ long arch_do_domctl(
copyback = 1;
break;
+ case XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM:
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L2);
+ domctl->u.psr_cat_op.data = val32;
+ copyback = 1;
+ break;
+
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 9e3ce21..16c08c0 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1132,6 +1132,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
uint64_t data; /* IN/OUT */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 19/23] x86: L2 CAT: implement set value flow.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (17 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 18/23] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 20/23] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
` (3 subsequent siblings)
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L2 CAT set value related callback function
and domctl interface.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v11:
- remove 'domctl->u.psr_cat_op.data' check because it has been moved into
'psr_set_val'.
(suggested by Jan Beulich)
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
v10:
- check input data and remove cast in domctl.
(suggested by Jan Beulich)
- remove some hooks assignment due to previous patches changes.
(suggested by Jan Beulich)
- remove cast in 'l2_cat_write_msr'.
(suggested by Jan Beulich)
- remove 'return in 'l2_cat_write_msr'.
(suggested by Jan Beulich)
v9:
- reuse some CAT common functions for L2 CAT to reduce redundant codes.
(suggested by Roger Pau)
- remove parameter 'found' from 'cat_compare_val' and modify the return
values to let caller know if the id is found or not. These things are
done in patch "x86: refactor psr: set value: implement cos finding flow."
(suggested by Roger Pau and Dario Faggioli)
- remove 'get_cos_num' related codes.
(suggested by Jan Beulich)
- modify 'l2_cat_write_msr' according to previous patch change.
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- modify 'l2_cat_write_msr' to 'void'.
v5:
- remove type check in callback function.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- remove 'l2_cat_get_cos_max_from_type'.
(suggested by Jan Beulich)
- rename 'l2_cat_exceeds_cos_max' to 'l2_cat_fits_cos_max'.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
---
xen/arch/x86/domctl.c | 6 ++++++
xen/arch/x86/psr.c | 6 ++++++
xen/include/public/domctl.h | 1 +
3 files changed, 13 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 2970118..b5a892a 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1428,6 +1428,12 @@ long arch_do_domctl(
PSR_CBM_TYPE_L3_DATA);
break;
+ case XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM:
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L2);
+ break;
+
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
ret = psr_get_val(d, domctl->u.psr_cat_op.target,
&val32, PSR_CBM_TYPE_L3);
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 22de787..ac7697b 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -387,10 +387,16 @@ static const struct feat_props l3_cdp_props = {
};
/* L2 CAT props */
+static void l2_cat_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
+{
+ wrmsrl(MSR_IA32_PSR_L2_MASK(cos), val);
+}
+
static const struct feat_props l2_cat_props = {
.cos_num = 1,
.type[0] = PSR_CBM_TYPE_L2,
.get_feat_info = cat_get_feat_info,
+ .write_msr = l2_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 16c08c0..a06c6c5 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1132,6 +1132,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM 6
#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 20/23] tools: L2 CAT: support get HW info for L2 CAT.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (18 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 19/23] x86: L2 CAT: implement set " Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 21/23] tools: L2 CAT: support show cbm " Yi Sun
` (2 subsequent siblings)
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements xl/xc changes to support get HW info
for L2 CAT.
'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
info.
Example(on machine which only supports L2 CAT):
Cache Monitoring Technology (CMT):
Enabled : 0
Cache Allocation Technology (CAT): L2
Socket ID : 0
Maximum COS : 3
CBM length : 8
Default CBM : 0xff
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v12:
- acked by Wei for tools side changes.
v11:
- acked by Jan for Hypervisor side changes.
v10:
- change macros names according to previous changes.
(suggested by Jan Beulich)
v9:
- add some cases to handle return error no.
- move xl_cmdimpl.c codes into xl/xl_psr.c.
- change 'l3_info' to 'cat_info' to cover both L3 and L2 CAT.
v6:
- adjust '{' position for 'switch'.
(suggested by Wei Liu)
- modify commit message to remove error log.
(suggested by Dario Faggioli)
v5:
- modify commit message to remove error log.
(suggested by Wei Liu and Jan Beulich)
- replace unnecessary 'return' to 'break'.
(suggested by Wei Liu)
- restore 'libxl_psr_cat_get_l3_info' to keep interface backward compatible
but change codes in it to call new function to get hw info.
(suggested by Wei Liu)
- add 'L2_CBM' into 'psr_cbm_type' because it is interface change which
should be in same patch with new 'LIBXL_HAVE_' macro.
(suggested by Wei Liu)
- addjust logs sentence to make unnecessary error logs not show.
(suggested by Wei Liu and Jan Beulich)
v4:
- create this patch to help reviewers better understand the codes.
---
---
tools/libxc/include/xenctrl.h | 6 ++---
tools/libxc/xc_psr.c | 39 +++++++++++++++++++++++---------
tools/libxl/libxl.h | 9 ++++++++
tools/libxl/libxl_psr.c | 28 ++++++++++++++++++-----
tools/libxl/libxl_types.idl | 1 +
tools/xl/xl_psr.c | 52 +++++++++++++++++++++++++++++++++----------
xen/arch/x86/sysctl.c | 12 +++++-----
xen/include/public/sysctl.h | 2 +-
8 files changed, 111 insertions(+), 38 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 01f8dfe..3fbcd70 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2468,9 +2468,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
xc_psr_cat_type type, uint32_t target,
uint64_t *data);
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled);
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len,
+ bool *cdp_enabled);
int xc_get_cpu_levelling_caps(xc_interface *xch, uint32_t *caps);
int xc_get_cpu_featureset(xc_interface *xch, uint32_t index,
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 43b3286..84a08c4 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -317,24 +317,41 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
return rc;
}
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled)
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len, bool *cdp_enabled)
{
- int rc;
+ int rc = -1;
DECLARE_SYSCTL;
sysctl.cmd = XEN_SYSCTL_psr_cat_op;
- sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
sysctl.u.psr_cat_op.target = socket;
- rc = xc_sysctl(xch, &sysctl);
- if ( !rc )
+ switch ( lvl )
{
- *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
- *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
- *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
- XEN_SYSCTL_PSR_CAT_L3_CDP;
+ case 2:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l2_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.cat_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.cat_info.cbm_len;
+ *cdp_enabled = false;
+ }
+ break;
+ case 3:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.cat_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.cat_info.cbm_len;
+ *cdp_enabled = sysctl.u.psr_cat_op.u.cat_info.flags &
+ XEN_SYSCTL_PSR_CAT_L3_CDP;
+ }
+ break;
+ default:
+ errno = EOPNOTSUPP;
+ break;
}
return rc;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index a402236..92da2ae 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -916,6 +916,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, const libxl_mac *src);
* If this is defined, the Code and Data Prioritization feature is supported.
*/
#define LIBXL_HAVE_PSR_CDP 1
+
+/*
+ * LIBXL_HAVE_PSR_L2_CAT
+ *
+ * If this is defined, the L2 Cache Allocation Technology feature is supported.
+ */
+#define LIBXL_HAVE_PSR_L2_CAT 1
#endif
/*
@@ -2194,6 +2201,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
* On success, the function returns an array of elements in 'info',
* and the length in 'nr'.
*/
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl);
int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
int *nr);
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr);
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index ec5c79d..f55ba1e 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -91,6 +91,15 @@ static void libxl__psr_cat_log_err_msg(libxl__gc *gc, int err)
case ENXIO:
msg = "Unable to set code or data CBM when CDP is disabled";
break;
+ case EINVAL:
+ msg = "Invalid input or some internal values are not expected";
+ break;
+ case ERANGE:
+ msg = "Socket number is wrong";
+ break;
+ case ENOSPC:
+ msg = "Value array exceeds the range";
+ break;
default:
libxl__psr_log_err_msg(gc, err);
@@ -352,8 +361,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
return rc;
}
-int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
- int *nr)
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl)
{
GC_INIT(ctx);
int rc;
@@ -380,9 +389,8 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
libxl_for_each_set_bit(socketid, socketmap) {
ptr[i].id = socketid;
- if (xc_psr_cat_get_l3_info(ctx->xch, socketid, &ptr[i].cos_max,
- &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
- libxl__psr_cat_log_err_msg(gc, errno);
+ if (xc_psr_cat_get_info(ctx->xch, socketid, lvl, &ptr[i].cos_max,
+ &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
rc = ERROR_FAIL;
free(ptr);
goto out;
@@ -398,6 +406,16 @@ out:
return rc;
}
+int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr)
+{
+ int rc;
+
+ rc = libxl_psr_cat_get_info(ctx, info, nr, 3);
+
+ return rc;
+}
+
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr)
{
int i;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index d970284..2776b6c 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -960,6 +960,7 @@ libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
(1, "L3_CBM"),
(2, "L3_CBM_CODE"),
(3, "L3_CBM_DATA"),
+ (4, "L2_CBM"),
])
libxl_psr_cat_info = Struct("psr_cat_info", [
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index c061b29..271b88f 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -294,21 +294,19 @@ int main_psr_cmt_show(int argc, char **argv)
}
#endif
-#ifdef LIBXL_HAVE_PSR_CAT
-static int psr_cat_hwinfo(void)
+#if defined(LIBXL_HAVE_PSR_CAT) || defined(LIBXL_HAVE_PSR_L2_CAT)
+static int psr_l3_cat_hwinfo(void)
{
- int rc;
- int i, nr;
+ int rc, nr;
+ unsigned int i;
uint32_t l3_cache_size;
libxl_psr_cat_info *info;
- printf("Cache Allocation Technology (CAT):\n");
-
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
- if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (rc)
return rc;
- }
+
+ printf("Cache Allocation Technology (CAT):\n");
for (i = 0; i < nr; i++) {
rc = libxl_psr_cmt_get_l3_cache_size(ctx, info[i].id, &l3_cache_size);
@@ -417,7 +415,7 @@ static int psr_cat_show(uint32_t domid)
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
if (rc) {
fprintf(stderr, "Failed to get cat info\n");
return rc;
@@ -434,6 +432,32 @@ out:
return rc;
}
+static int psr_l2_cat_hwinfo(void)
+{
+ int rc;
+ unsigned int i;
+ int nr;
+ libxl_psr_cat_info *info;
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 2);
+ if (rc)
+ return rc;
+
+ printf("Cache Allocation Technology (CAT): L2\n");
+
+ for (i = 0; i < nr; i++) {
+ /* There is no CMT on L2 cache so far. */
+ printf("%-16s: %u\n", "Socket ID", info[i].id);
+ printf("%-16s: %u\n", "Maximum COS", info[i].cos_max);
+ printf("%-16s: %u\n", "CBM length", info[i].cbm_len);
+ printf("%-16s: %#llx\n", "Default CBM",
+ (1ull << info[i].cbm_len) - 1);
+ }
+
+ libxl_psr_cat_info_list_free(info, nr);
+ return rc;
+}
+
int main_psr_cat_cbm_set(int argc, char **argv)
{
uint32_t domid;
@@ -551,7 +575,11 @@ int main_psr_hwinfo(int argc, char **argv)
ret = psr_cmt_hwinfo();
if (!ret && (all || cat))
- ret = psr_cat_hwinfo();
+ ret = psr_l3_cat_hwinfo();
+
+ /* L2 CAT is independent of CMT and L3 CAT */
+ if (all || cat)
+ ret = psr_l2_cat_hwinfo();
return ret;
}
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 5e91755..14c12b7 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -183,11 +183,11 @@ long arch_do_sysctl(
if ( ret )
break;
- sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ sysctl->u.psr_cat_op.u.cat_info.cos_max =
data[PSR_INFO_IDX_COS_MAX];
- sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ sysctl->u.psr_cat_op.u.cat_info.cbm_len =
data[PSR_INFO_IDX_CAT_CBM_LEN];
- sysctl->u.psr_cat_op.u.l3_info.flags =
+ sysctl->u.psr_cat_op.u.cat_info.flags =
data[PSR_INFO_IDX_CAT_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
@@ -202,11 +202,11 @@ long arch_do_sysctl(
if ( ret )
break;
- sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ sysctl->u.psr_cat_op.u.cat_info.cos_max =
data[PSR_INFO_IDX_COS_MAX];
- sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ sysctl->u.psr_cat_op.u.cat_info.cbm_len =
data[PSR_INFO_IDX_CAT_CBM_LEN];
- sysctl->u.psr_cat_op.u.l3_info.flags =
+ sysctl->u.psr_cat_op.u.cat_info.flags =
data[PSR_INFO_IDX_CAT_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 4c76d3a..9e51af6 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -754,7 +754,7 @@ struct xen_sysctl_psr_cat_op {
uint32_t cos_max; /* OUT: Maximum COS */
#define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
uint32_t flags; /* OUT: CAT flags */
- } l3_info;
+ } cat_info;
} u;
};
typedef struct xen_sysctl_psr_cat_op xen_sysctl_psr_cat_op_t;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 21/23] tools: L2 CAT: support show cbm for L2 CAT.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (19 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 20/23] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 22/23] tools: L2 CAT: support set " Yi Sun
2017-06-14 1:12 ` [PATCH v12 23/23] docs: add L2 CAT description in docs Yi Sun
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements changes in xl/xc changes to support
showing CBM of L2 CAT.
The new level option is introduced to original CAT showing
command in order to show CBM for specified level CAT.
- 'xl psr-cat-show' is updated to show CBM of a domain
according to input cache level.
Examples:
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v9:
- move xl_cmdimpl.c changes into xl/xl_psr.c.
- move xl_cmdtable.c changes into xl/xl_cmdtable.c.
v6:
- check if input level is correct.
- adjust '{' postion for 'if'.
(suggested by Wei Liu)
v5:
- remove 'L2_CBM' in idl because it has been moved to patch 21:
"tools: L2 CAT: support get HW info for L2 CAT".
(suggested by Wei Liu)
v4:
- create this patch because of codes architecture change.
---
---
tools/libxc/include/xenctrl.h | 1 +
tools/libxc/xc_psr.c | 3 ++
tools/xl/xl_cmdtable.c | 3 +-
tools/xl/xl_psr.c | 85 +++++++++++++++++++++++++++++--------------
4 files changed, 63 insertions(+), 29 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 3fbcd70..0952f75 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2444,6 +2444,7 @@ enum xc_psr_cat_type {
XC_PSR_CAT_L3_CBM = 1,
XC_PSR_CAT_L3_CBM_CODE = 2,
XC_PSR_CAT_L3_CBM_DATA = 3,
+ XC_PSR_CAT_L2_CBM = 4,
};
typedef enum xc_psr_cat_type xc_psr_cat_type;
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 84a08c4..04f5927 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -299,6 +299,9 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 30eb93c..539cb64 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -557,7 +557,8 @@ struct cmd_spec cmd_table[] = {
{ "psr-cat-show",
&main_psr_cat_show, 0, 1,
"Show Cache Allocation Technology information",
- "<Domain>",
+ "[options] <Domain>",
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
},
#endif
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index 271b88f..575f4a0 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -342,7 +342,7 @@ static void psr_cat_print_one_domain_cbm_type(uint32_t domid, uint32_t socketid,
}
static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
char *domain_name;
@@ -350,27 +350,38 @@ static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
printf("%5d%25s", domid, domain_name);
free(domain_name);
- if (!cdp_enabled) {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM);
- } else {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ switch (lvl) {
+ case 3:
+ if (!cdp_enabled) {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM);
+ } else {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ }
+ break;
+ case 2:
psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ LIBXL_PSR_CBM_TYPE_L2_CBM);
+ break;
+ default:
+ printf("Input lvl %d is wrong!", lvl);
+ break;
}
printf("\n");
}
static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
int i, nr_domains;
libxl_dominfo *list;
if (domid != INVALID_DOMID) {
- psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled, lvl);
return 0;
}
@@ -380,49 +391,59 @@ static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
}
for (i = 0; i < nr_domains; i++)
- psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled, lvl);
libxl_dominfo_list_free(list, nr_domains);
return 0;
}
-static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
+static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info,
+ unsigned int lvl)
{
int rc;
uint32_t l3_cache_size;
- rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
- if (rc) {
- fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
- info->id);
- return -1;
+ printf("%-16s: %u\n", "Socket ID", info->id);
+
+ /* So far, CMT only supports L3 cache. */
+ if (lvl == 3) {
+ rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
+ if (rc) {
+ fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
+ info->id);
+ return -1;
+ }
+ printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
}
- printf("%-16s: %u\n", "Socket ID", info->id);
- printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
printf("%-16s: %#llx\n", "Default CBM", (1ull << info->cbm_len) - 1);
if (info->cdp_enabled)
printf("%5s%25s%16s%16s\n", "ID", "NAME", "CBM (code)", "CBM (data)");
else
printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
- return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled);
+ return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled, lvl);
}
-static int psr_cat_show(uint32_t domid)
+static int psr_cat_show(uint32_t domid, unsigned int lvl)
{
int i, nr;
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (lvl != 2 && lvl != 3) {
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
+ }
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, lvl);
if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ fprintf(stderr, "Failed to get %s cat info\n", (lvl == 3)?"L3":"L2");
return rc;
}
for (i = 0; i < nr; i++) {
- rc = psr_cat_print_socket(domid, info + i);
+ rc = psr_cat_print_socket(domid, info + i, lvl);
if (rc)
goto out;
}
@@ -533,11 +554,19 @@ int main_psr_cat_cbm_set(int argc, char **argv)
int main_psr_cat_show(int argc, char **argv)
{
- int opt;
+ int opt = 0;
uint32_t domid;
+ unsigned int lvl = 3;
- SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cat-show", 0) {
- /* No options */
+ static struct option opts[] = {
+ {"level", 1, 0, 'l'},
+ COMMON_LONG_OPTS
+ };
+
+ SWITCH_FOREACH_OPT(opt, "l:", opts, "psr-cat-show", 0) {
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
if (optind >= argc)
@@ -549,7 +578,7 @@ int main_psr_cat_show(int argc, char **argv)
return 2;
}
- return psr_cat_show(domid);
+ return psr_cat_show(domid, lvl);
}
int main_psr_hwinfo(int argc, char **argv)
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 22/23] tools: L2 CAT: support set cbm for L2 CAT.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (20 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 21/23] tools: L2 CAT: support show cbm " Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 23/23] docs: add L2 CAT description in docs Yi Sun
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the xl/xc changes to support set CBM
for L2 CAT.
The new level option is introduced to original CAT setting
command in order to set CBM for specified level CAT.
- 'xl psr-cat-set' is updated to set cache capacity bitmasks(CBM)
for a domain according to input cache level.
root@:~$ xl psr-cat-set -l2 1 0x7f
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v11:
- revert change in v9 to handle the case to set both CODE and DATA for CDP
at same time. This must be handled in hypervisor.
v10:
- fix comments.
(suggested by Wei Liu)
v9:
- handle the case to set both CODE and DATA for CDP at same time.
For such case, user does not input '-c' or '-d' to set CDP cbm.
- move xl_cmdimpl.c changes into xl/xl_psr.c.
- move xl_cmdtable.c changes into xl/xl_cmdtable.c.
v6:
- rename 'psr-cat-cbm-set' to 'psr-cat-set'.
(suggested by Kevin Tian)
- return 'EXIT_FAILURE' for error case.
(suggested by Dario Faggioli)
- print error info when input level is wrong.
v4:
- create this patch because of codes architecture change.
---
---
tools/libxc/xc_psr.c | 3 +++
tools/xl/xl_cmdtable.c | 3 ++-
tools/xl/xl_psr.c | 33 +++++++++++++++++++++++----------
3 files changed, 28 insertions(+), 11 deletions(-)
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 04f5927..039b920 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -266,6 +266,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 539cb64..2c71a9f 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -546,11 +546,12 @@ struct cmd_spec cmd_table[] = {
},
#endif
#ifdef LIBXL_HAVE_PSR_CAT
- { "psr-cat-cbm-set",
+ { "psr-cat-set",
&main_psr_cat_cbm_set, 0, 1,
"Set cache capacity bitmasks(CBM) for a domain",
"[options] <Domain> <CBM>",
"-s <socket> Specify the socket to process, otherwise all sockets are processed\n"
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
"-c Set code CBM if CDP is supported\n"
"-d Set data CBM if CDP is supported\n"
},
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index 575f4a0..7309d4f 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -490,19 +490,21 @@ int main_psr_cat_cbm_set(int argc, char **argv)
char *value;
libxl_string_list socket_list;
unsigned long start, end;
- int i, j, len;
+ unsigned int i, j, len;
+ unsigned int lvl = 3;
static struct option opts[] = {
{"socket", 1, 0, 's'},
{"data", 0, 0, 'd'},
{"code", 0, 0, 'c'},
+ {"level", 1, 0, 'l'},
COMMON_LONG_OPTS
};
libxl_socket_bitmap_alloc(ctx, &target_map, 0);
libxl_bitmap_set_none(&target_map);
- SWITCH_FOREACH_OPT(opt, "s:cd", opts, "psr-cat-cbm-set", 2) {
+ SWITCH_FOREACH_OPT(opt, "s:l:cd", opts, "psr-cat-set", 2) {
case 's':
trim(isspace, optarg, &value);
split_string_into_string_list(value, ",", &socket_list);
@@ -522,24 +524,35 @@ int main_psr_cat_cbm_set(int argc, char **argv)
case 'c':
opt_code = 1;
break;
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
- if (opt_data && opt_code) {
- fprintf(stderr, "Cannot handle -c and -d at the same time\n");
- return -1;
- } else if (opt_data) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
- } else if (opt_code) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ if (lvl == 2)
+ type = LIBXL_PSR_CBM_TYPE_L2_CBM;
+ else if (lvl == 3) {
+ if (opt_data && opt_code) {
+ fprintf(stderr, "Cannot handle -c and -d at the same time\n");
+ return EXIT_FAILURE;
+ } else if (opt_data) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
+ } else if (opt_code) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ } else {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ }
} else {
type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
}
if (libxl_bitmap_is_empty(&target_map))
libxl_bitmap_set_any(&target_map);
if (argc != optind + 2) {
- help("psr-cat-cbm-set");
+ help("psr-cat-set");
return 2;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v12 23/23] docs: add L2 CAT description in docs.
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (21 preceding siblings ...)
2017-06-14 1:12 ` [PATCH v12 22/23] tools: L2 CAT: support set " Yi Sun
@ 2017-06-14 1:12 ` Yi Sun
22 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-14 1:12 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch adds L2 CAT description in related documents.
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
docs/man/xl.pod.1.in | 25 ++++++++++++++++++++++---
docs/misc/xl-psr.markdown | 18 ++++++++++++------
2 files changed, 34 insertions(+), 9 deletions(-)
diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 78bf884..db951c0 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -1716,6 +1716,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
a domain is attached to the monitoring service, monitoring data can be shown
for any of these monitoring types.
+There is no cache monitoring and memory bandwidth monitoring on L2 cache so
+far.
+
=over 4
=item B<psr-cmt-attach> [I<domain-id>]
@@ -1740,7 +1743,7 @@ monitor types are:
Intel Broadwell and later server platforms offer capabilities to configure and
make use of the Cache Allocation Technology (CAT) mechanisms, which enable more
-cache resources (i.e. L3 cache) to be made available for high priority
+cache resources (i.e. L3/L2 cache) to be made available for high priority
applications. In the Xen implementation, CAT is used to control cache allocation
on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
(CBM) for the domain.
@@ -1750,7 +1753,7 @@ Intel Broadwell and later server platforms also offer Code/Data Prioritization
applications. CDP is used on a per VM basis in the Xen implementation. To
specify code or data CBM for the domain, CDP feature must be enabled and CBM
type options need to be specified when setting CBM, and the type options (code
-and data) are mutually exclusive.
+and data) are mutually exclusive. There is no CDP support on L2 so far.
=over 4
@@ -1767,6 +1770,11 @@ B<OPTIONS>
Specify the socket to process, otherwise all sockets are processed.
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
=item B<-c>, B<--code>
Set code CBM when CDP is enabled.
@@ -1777,10 +1785,21 @@ Set data CBM when CDP is enabled.
=back
-=item B<psr-cat-show> [I<domain-id>]
+=item B<psr-cat-show> [I<OPTIONS>] [I<domain-id>]
Show CAT settings for a certain domain or all domains.
+B<OPTIONS>
+
+=over 4
+
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
+=back
+
=back
=head1 IGNORED FOR COMPATIBILITY WITH XM
diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
index c3c1e8e..04dd957 100644
--- a/docs/misc/xl-psr.markdown
+++ b/docs/misc/xl-psr.markdown
@@ -70,7 +70,7 @@ total-mem-bandwidth instead of cache-occupancy). E.g. after a `xl psr-cmt-attach
Cache Allocation Technology (CAT) is a new feature available on Intel
Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
-partition cache allocation (i.e. L3 cache) based on application priority or
+partition cache allocation (i.e. L3/L2 cache) based on application priority or
Class of Service (COS). Each COS is configured using capacity bitmasks (CBM)
which represent cache capacity and indicate the degree of overlap and
isolation between classes. System cache resource is divided into numbers of
@@ -107,7 +107,7 @@ System CAT information such as maximum COS and CBM length can be obtained by:
The simplest way to change a domain's CBM from its default is running:
-`xl psr-cat-cbm-set [OPTIONS] <domid> <cbm>`
+`xl psr-cat-set [OPTIONS] <domid> <cbm>`
where cbm is a number to represent the corresponding cache subset can be used.
A cbm is valid only when:
@@ -119,13 +119,19 @@ A cbm is valid only when:
In a multi-socket system, the same cbm will be set on each socket by default.
Per socket cbm can be specified with the `--socket SOCKET` option.
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
+
Setting the CBM may not be successful if insufficient COS is available. In
such case unused COS(es) may be freed by setting CBM of all related domains to
its default value(all-ones).
Per domain CBM settings can be shown by:
-`xl psr-cat-show`
+`xl psr-cat-show [OPTIONS] <domid>`
+
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
## Code and Data Prioritization (CDP)
@@ -172,13 +178,13 @@ options is invalid.
Example:
Setting code CBM for a domain:
-`xl psr-cat-cbm-set -c <domid> <cbm>`
+`xl psr-cat-set -c <domid> <cbm>`
Setting data CBM for a domain:
-`xl psr-cat-cbm-set -d <domid> <cbm>`
+`xl psr-cat-set -d <domid> <cbm>`
Setting the same code and data CBM for a domain:
-`xl psr-cat-cbm-set <domid> <cbm>`
+`xl psr-cat-set <domid> <cbm>`
## Reference
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 67+ messages in thread
* Re: [PATCH v12 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-06-14 1:12 ` [PATCH v12 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
@ 2017-06-28 7:12 ` Jan Beulich
2017-06-28 9:07 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-28 7:12 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -13,16 +13,112 @@
> * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
> * more details.
> */
> -#include <xen/init.h>
> #include <xen/cpu.h>
> #include <xen/err.h>
> +#include <xen/init.h>
> #include <xen/sched.h>
> #include <asm/psr.h>
>
> +/*
> + * Terminology:
> + * - CAT Cache Allocation Technology
> + * - CBM Capacity BitMasks
> + * - CDP Code and Data Prioritization
> + * - CMT Cache Monitoring Technology
> + * - COS/CLOS Class of Service. Also mean COS registers.
> + * - COS_MAX Max number of COS for the feature (minus 1)
> + * - MSRs Machine Specific Registers
> + * - PSR Intel Platform Shared Resource
> + */
> +
> #define PSR_CMT (1<<0)
> #define PSR_CAT (1<<1)
> #define PSR_CDP (1<<2)
>
> +#define CAT_CBM_LEN_MASK 0x1f
> +#define CAT_COS_MAX_MASK 0xffff
> +
> +/*
> + * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
> + * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
> + * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
> + *
> + * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for
> + * up to 64 L2 CAT COS. The COS_ID=[0,63].
> + *
> + * So, the maximum COS register count of one feature is 128.
> + */
> +#define MAX_COS_REG_CNT 128
> +
> +/*
> + * Every PSR feature uses some COS registers for each COS ID, e.g. CDP uses 2
> + * COS registers (DATA and CODE) for one COS ID, but CAT uses 1 COS register.
> + * We use below macro as the max number of COS registers used by all features.
> + * So far, it is 2 which means CDP's COS registers number.
> + */
> +#define PSR_MAX_COS_NUM 2
> +
> +enum psr_feat_type {
> + PSR_SOCKET_L3_CAT,
> + PSR_SOCKET_FEAT_NUM,
> +};
For identifiers going into a header, PSR_ and psr_ disambiguation prefixes
are certainly necessary. For everything being declared / defined for just this
one file this isn't really necessary imo (the SOCKET_ part above I'd then also
be uncertain about). The main thing, however, is the inconsistency here: Above
you have MAX_COS_REG_CNT and PSR_MAX_COS_NUM. I would really prefer if both
prefix and suffix wise these were consistent.
> +static void cat_init_feature(const struct cpuid_leaf *regs,
> + struct feat_node *feat,
> + struct psr_socket_info *info,
> + enum psr_feat_type type)
> +{
> + /* No valid value so do not enable feature. */
> + if ( !regs->a || !regs->d )
> + return;
> +
> + feat->cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
> + feat->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
> +
> + switch ( type )
> + {
> + case PSR_SOCKET_L3_CAT:
> + /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> + feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
The word "reserved" in the comment is a little unfortunate - if there was
anything reserved in a register, I'd expect the respective parts to either
not be writable, or to fault upon write attempts. However, I think you mean
that you reserve it for the described purpose. So perhaps "We reserve ..."?
Also please have a blank before the opeing paren.
With all of the suggestion taken care of
Reviewed-by: Jan Beulich <jbeulich@suse.com>
With at least the comment adjusted (and considering how late I am with the
other suggestions)
Acked-by: Jan Beulich <jbeulich@suse.com>
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows.
2017-06-14 1:12 ` [PATCH v12 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
@ 2017-06-28 7:13 ` Jan Beulich
0 siblings, 0 replies; 67+ messages in thread
From: Jan Beulich @ 2017-06-28 7:13 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> @@ -379,6 +414,14 @@ void psr_ctxt_switch_to(struct domain *d)
> if ( psr_cmt_enabled() )
> psr_assoc_rmid(®, d->arch.psr_rmid);
>
> + /* If domain's 'psr_cos_ids' is NULL, we set default value for it. */
> + if ( psra->cos_mask )
> + reg = psr_assoc_cos(reg,
> + (d->arch.psr_cos_ids ?
> + d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
> + 0),
> + psra->cos_mask);
The common way of indenting such longish function calls is to have the
arguments be 4 positions deeper indented than the function name (following
our general 4-space indentation model). Another alternative would be to use
e.g. a "socket" helper variable just to shrink line length.
With this taken care of either way, and preferably with the naming aspect
mentioned in the previous patch also extended to here (I won't mention this
again on later patches, but I'll imply it to apply there too)
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-06-14 1:12 ` [PATCH v12 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
@ 2017-06-28 7:14 ` Jan Beulich
0 siblings, 0 replies; 67+ messages in thread
From: Jan Beulich @ 2017-06-28 7:14 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> There is an interface in user space to show feature value of
> domains.
>
> This patch implements get value flow in hypervisor.
>
> It also changes domctl interface to make it more general.
>
> With this patch, 'psr-cat-show' can work for L3 CAT but not for
> L3 code/data which is implemented in CDP related patches.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-14 1:12 ` [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
@ 2017-06-28 7:14 ` Jan Beulich
2017-06-28 9:09 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-28 7:14 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> @@ -179,6 +182,10 @@ static void free_socket_resources(unsigned int socket)
> }
>
> info->feat_init = false;
> +
> + memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
> +
> + bitmap_zero(info->dom_set, DOMID_IDLE + 1);
> }
I can see the point of the latter (as you add the new structure member), but
if the former is necessary, shouldn't it have been done in an earlier patch?
Or should be field only be introduced here?
> @@ -537,7 +556,16 @@ int psr_get_val(struct domain *d, unsigned int socket,
> return -ENOENT;
> }
>
> + domain_lock(d);
> + if ( !test_bit(d->domain_id, socket_info[socket].dom_set) )
> + {
> + d->arch.psr_cos_ids[socket] = 0;
> + set_bit(d->domain_id, socket_info[socket].dom_set);
> + }
Any reason not to use test_and_set_bit() here? I.e. is this on any hot path?
Or wait - I think it's even wrong to split the test from the set, as the lock
doesn't protect dom_set[].
> +int psr_set_val(struct domain *d, unsigned int socket,
> + uint64_t new_val, enum cbm_type type)
> +{
> + unsigned int old_cos, array_len;
> + int cos, ret;
> + unsigned int *ref;
> + uint32_t *val_array, val;
> + struct psr_socket_info *info = get_socket_info(socket);
> + enum psr_feat_type feat_type;
> +
> + if ( IS_ERR(info) )
> + return PTR_ERR(info);
> +
> + val = new_val;
> + if ( new_val != val )
> + return -EINVAL;
> +
> + feat_type = psr_cbm_type_to_feat_type(type);
> + if ( feat_type >= ARRAY_SIZE(info->features) ||
> + !info->features[feat_type] )
> + return -ENOENT;
> +
> + /*
> + * Step 0:
> + * old_cos means the COS ID current domain is using. By default, it is 0.
> + *
> + * For every COS ID, there is a reference count to record how many domains
> + * are using the COS register corresponding to this COS ID.
> + * - If ref[old_cos] is 0, that means this COS is not used by any domain.
> + * - If ref[old_cos] is 1, that means this COS is only used by current
> + * domain.
> + * - If ref[old_cos] is more than 1, that mean multiple domains are using
> + * this COS.
> + */
> + domain_lock(d);
> + if ( !test_bit(d->domain_id, info->dom_set) )
> + {
> + d->arch.psr_cos_ids[socket] = 0;
> + set_bit(d->domain_id, info->dom_set);
> + }
Same here.
> + old_cos = d->arch.psr_cos_ids[socket];
> + domain_unlock(d);
> +
> + ASSERT(old_cos < MAX_COS_REG_CNT);
> +
> + ref = info->cos_ref;
> +
> + /*
> + * Step 1:
> + * Gather a value array to store all features cos_reg_val[old_cos].
> + * And, set the input new val into array according to the feature's
> + * position in array.
> + */
> + array_len = get_cos_num(info);
> + val_array = xzalloc_array(uint32_t, array_len);
> + if ( !val_array )
> + return -ENOMEM;
> +
> + if ( (ret = gather_val_array(val_array, array_len, info, old_cos)) != 0 )
> + goto free_array;
> +
> + if ( (ret = insert_val_into_array(val_array, array_len, info,
> + feat_type, type, val)) != 0 )
> + goto free_array;
> +
> + spin_lock(&info->ref_lock);
> +
> + /*
> + * Step 2:
> + * Try to find if there is already a COS ID on which all features' values
> + * are same as the array. Then, we can reuse this COS ID.
> + */
> + cos = find_cos(val_array, array_len, feat_type, info);
> + if ( cos == old_cos )
> + {
> + ret = 0;
> + goto unlock_free_array;
> + }
> +
> + /*
> + * Step 3:
> + * If fail to find, we need pick an available COS ID.
> + * In fact, only COS ID which ref is 1 or 0 can be picked for current
> + * domain. If old_cos is not 0 and its ref==1, that means only current
> + * domain is using this old_cos ID. So, this old_cos ID certainly can
> + * be reused by current domain. Ref==0 means there is no any domain
> + * using this COS ID. So it can be used for current domain too.
> + */
> + if ( cos < 0 )
> + {
> + cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
> + if ( cos < 0 )
> + {
> + ret = cos;
> + goto unlock_free_array;
> + }
> +
> + /*
> + * Step 4:
> + * Write the feature's MSRs according to the COS ID.
> + */
> + ret = write_psr_msrs(socket, cos, val_array, array_len, feat_type);
> + if ( ret )
> + goto unlock_free_array;
> + }
> +
> + /*
> + * Step 5:
> + * Find the COS ID (find_cos result is '>= 0' or an available COS ID is
> + * picked, then update ref according to COS ID.
> + */
> + ref[cos]++;
> + ASSERT(!cos || ref[cos]);
> + ASSERT(!old_cos || ref[old_cos]);
> + ref[old_cos]--;
> + spin_unlock(&info->ref_lock);
> +
> + /*
> + * Step 6:
> + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> + * which COS the domain is using on the socket. One domain can only use
> + * one COS ID at same time on each socket.
> + */
> + domain_lock(d);
> + d->arch.psr_cos_ids[socket] = cos;
> + domain_unlock(d);
> +
> + /*
> + * Step 7:
> + * Then, set the dom_set bit which corresponds to domain_id to mark this
> + * domain has been set and the COS ID of the domain is valid.
> + */
> + set_bit(d->domain_id, info->dom_set);
With the way things are being done above, doesn't this belong in the
domain_lock()-ed region?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-06-28 7:12 ` Jan Beulich
@ 2017-06-28 9:07 ` Yi Sun
0 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-28 9:07 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-28 01:12:23, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -13,16 +13,112 @@
> > * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
> > * more details.
> > */
> > -#include <xen/init.h>
> > #include <xen/cpu.h>
> > #include <xen/err.h>
> > +#include <xen/init.h>
> > #include <xen/sched.h>
> > #include <asm/psr.h>
> >
> > +/*
> > + * Terminology:
> > + * - CAT Cache Allocation Technology
> > + * - CBM Capacity BitMasks
> > + * - CDP Code and Data Prioritization
> > + * - CMT Cache Monitoring Technology
> > + * - COS/CLOS Class of Service. Also mean COS registers.
> > + * - COS_MAX Max number of COS for the feature (minus 1)
> > + * - MSRs Machine Specific Registers
> > + * - PSR Intel Platform Shared Resource
> > + */
> > +
> > #define PSR_CMT (1<<0)
> > #define PSR_CAT (1<<1)
> > #define PSR_CDP (1<<2)
> >
> > +#define CAT_CBM_LEN_MASK 0x1f
> > +#define CAT_COS_MAX_MASK 0xffff
> > +
> > +/*
> > + * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
> > + * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
> > + * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
> > + *
> > + * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for
> > + * up to 64 L2 CAT COS. The COS_ID=[0,63].
> > + *
> > + * So, the maximum COS register count of one feature is 128.
> > + */
> > +#define MAX_COS_REG_CNT 128
> > +
> > +/*
> > + * Every PSR feature uses some COS registers for each COS ID, e.g. CDP uses 2
> > + * COS registers (DATA and CODE) for one COS ID, but CAT uses 1 COS register.
> > + * We use below macro as the max number of COS registers used by all features.
> > + * So far, it is 2 which means CDP's COS registers number.
> > + */
> > +#define PSR_MAX_COS_NUM 2
> > +
> > +enum psr_feat_type {
> > + PSR_SOCKET_L3_CAT,
> > + PSR_SOCKET_FEAT_NUM,
> > +};
>
> For identifiers going into a header, PSR_ and psr_ disambiguation prefixes
> are certainly necessary. For everything being declared / defined for just this
> one file this isn't really necessary imo (the SOCKET_ part above I'd then also
> be uncertain about). The main thing, however, is the inconsistency here: Above
> you have MAX_COS_REG_CNT and PSR_MAX_COS_NUM. I would really prefer if both
> prefix and suffix wise these were consistent.
>
> > +static void cat_init_feature(const struct cpuid_leaf *regs,
> > + struct feat_node *feat,
> > + struct psr_socket_info *info,
> > + enum psr_feat_type type)
> > +{
> > + /* No valid value so do not enable feature. */
> > + if ( !regs->a || !regs->d )
> > + return;
> > +
> > + feat->cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
> > + feat->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
> > +
> > + switch ( type )
> > + {
> > + case PSR_SOCKET_L3_CAT:
> > + /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> > + feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
>
> The word "reserved" in the comment is a little unfortunate - if there was
> anything reserved in a register, I'd expect the respective parts to either
> not be writable, or to fault upon write attempts. However, I think you mean
> that you reserve it for the described purpose. So perhaps "We reserve ..."?
> Also please have a blank before the opeing paren.
>
> With all of the suggestion taken care of
> Reviewed-by: Jan Beulich <jbeulich@suse.com>
>
Thank you! I will modify macros, enum identifiers added since this patch to
make them be consistent.
> With at least the comment adjusted (and considering how late I am with the
> other suggestions)
> Acked-by: Jan Beulich <jbeulich@suse.com>
>
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-28 7:14 ` Jan Beulich
@ 2017-06-28 9:09 ` Yi Sun
2017-06-28 11:43 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-28 9:09 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-28 01:14:59, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> > @@ -179,6 +182,10 @@ static void free_socket_resources(unsigned int socket)
> > }
> >
> > info->feat_init = false;
> > +
> > + memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
> > +
> > + bitmap_zero(info->dom_set, DOMID_IDLE + 1);
> > }
>
> I can see the point of the latter (as you add the new structure member), but
> if the former is necessary, shouldn't it have been done in an earlier patch?
> Or should be field only be introduced here?
>
You are right. I will move the cos_ref clearing into earlier patch.
> > @@ -537,7 +556,16 @@ int psr_get_val(struct domain *d, unsigned int socket,
> > return -ENOENT;
> > }
> >
> > + domain_lock(d);
> > + if ( !test_bit(d->domain_id, socket_info[socket].dom_set) )
> > + {
> > + d->arch.psr_cos_ids[socket] = 0;
> > + set_bit(d->domain_id, socket_info[socket].dom_set);
> > + }
>
> Any reason not to use test_and_set_bit() here? I.e. is this on any hot path?
> Or wait - I think it's even wrong to split the test from the set, as the lock
> doesn't protect dom_set[].
>
Will change it to test_and_set_bit.
> > +int psr_set_val(struct domain *d, unsigned int socket,
> > + uint64_t new_val, enum cbm_type type)
> > +{
> > + unsigned int old_cos, array_len;
> > + int cos, ret;
> > + unsigned int *ref;
> > + uint32_t *val_array, val;
> > + struct psr_socket_info *info = get_socket_info(socket);
> > + enum psr_feat_type feat_type;
> > +
> > + if ( IS_ERR(info) )
> > + return PTR_ERR(info);
> > +
> > + val = new_val;
> > + if ( new_val != val )
> > + return -EINVAL;
> > +
> > + feat_type = psr_cbm_type_to_feat_type(type);
> > + if ( feat_type >= ARRAY_SIZE(info->features) ||
> > + !info->features[feat_type] )
> > + return -ENOENT;
> > +
> > + /*
> > + * Step 0:
> > + * old_cos means the COS ID current domain is using. By default, it is 0.
> > + *
> > + * For every COS ID, there is a reference count to record how many domains
> > + * are using the COS register corresponding to this COS ID.
> > + * - If ref[old_cos] is 0, that means this COS is not used by any domain.
> > + * - If ref[old_cos] is 1, that means this COS is only used by current
> > + * domain.
> > + * - If ref[old_cos] is more than 1, that mean multiple domains are using
> > + * this COS.
> > + */
> > + domain_lock(d);
> > + if ( !test_bit(d->domain_id, info->dom_set) )
> > + {
> > + d->arch.psr_cos_ids[socket] = 0;
> > + set_bit(d->domain_id, info->dom_set);
> > + }
>
> Same here.
>
> > + old_cos = d->arch.psr_cos_ids[socket];
> > + domain_unlock(d);
> > +
> > + ASSERT(old_cos < MAX_COS_REG_CNT);
> > +
> > + ref = info->cos_ref;
> > +
> > + /*
> > + * Step 1:
> > + * Gather a value array to store all features cos_reg_val[old_cos].
> > + * And, set the input new val into array according to the feature's
> > + * position in array.
> > + */
> > + array_len = get_cos_num(info);
> > + val_array = xzalloc_array(uint32_t, array_len);
> > + if ( !val_array )
> > + return -ENOMEM;
> > +
> > + if ( (ret = gather_val_array(val_array, array_len, info, old_cos)) != 0 )
> > + goto free_array;
> > +
> > + if ( (ret = insert_val_into_array(val_array, array_len, info,
> > + feat_type, type, val)) != 0 )
> > + goto free_array;
> > +
> > + spin_lock(&info->ref_lock);
> > +
> > + /*
> > + * Step 2:
> > + * Try to find if there is already a COS ID on which all features' values
> > + * are same as the array. Then, we can reuse this COS ID.
> > + */
> > + cos = find_cos(val_array, array_len, feat_type, info);
> > + if ( cos == old_cos )
> > + {
> > + ret = 0;
> > + goto unlock_free_array;
> > + }
> > +
> > + /*
> > + * Step 3:
> > + * If fail to find, we need pick an available COS ID.
> > + * In fact, only COS ID which ref is 1 or 0 can be picked for current
> > + * domain. If old_cos is not 0 and its ref==1, that means only current
> > + * domain is using this old_cos ID. So, this old_cos ID certainly can
> > + * be reused by current domain. Ref==0 means there is no any domain
> > + * using this COS ID. So it can be used for current domain too.
> > + */
> > + if ( cos < 0 )
> > + {
> > + cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
> > + if ( cos < 0 )
> > + {
> > + ret = cos;
> > + goto unlock_free_array;
> > + }
> > +
> > + /*
> > + * Step 4:
> > + * Write the feature's MSRs according to the COS ID.
> > + */
> > + ret = write_psr_msrs(socket, cos, val_array, array_len, feat_type);
> > + if ( ret )
> > + goto unlock_free_array;
> > + }
> > +
> > + /*
> > + * Step 5:
> > + * Find the COS ID (find_cos result is '>= 0' or an available COS ID is
> > + * picked, then update ref according to COS ID.
> > + */
> > + ref[cos]++;
> > + ASSERT(!cos || ref[cos]);
> > + ASSERT(!old_cos || ref[old_cos]);
> > + ref[old_cos]--;
> > + spin_unlock(&info->ref_lock);
> > +
> > + /*
> > + * Step 6:
> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> > + * which COS the domain is using on the socket. One domain can only use
> > + * one COS ID at same time on each socket.
> > + */
> > + domain_lock(d);
> > + d->arch.psr_cos_ids[socket] = cos;
> > + domain_unlock(d);
> > +
> > + /*
> > + * Step 7:
> > + * Then, set the dom_set bit which corresponds to domain_id to mark this
> > + * domain has been set and the COS ID of the domain is valid.
> > + */
> > + set_bit(d->domain_id, info->dom_set);
>
> With the way things are being done above, doesn't this belong in the
> domain_lock()-ed region?
>
Yes, should be. Thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-28 9:09 ` Yi Sun
@ 2017-06-28 11:43 ` Jan Beulich
2017-06-29 5:12 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-28 11:43 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/28/17 11:10 AM >>>
>On 17-06-28 01:14:59, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
>> > @@ -537,7 +556,16 @@ int psr_get_val(struct domain *d, unsigned int socket,
>> > return -ENOENT;
>> > }
>> >
>> > + domain_lock(d);
>> > + if ( !test_bit(d->domain_id, socket_info[socket].dom_set) )
>> > + {
>> > + d->arch.psr_cos_ids[socket] = 0;
>> > + set_bit(d->domain_id, socket_info[socket].dom_set);
>> > + }
>>
>> Any reason not to use test_and_set_bit() here? I.e. is this on any hot path?
>> Or wait - I think it's even wrong to split the test from the set, as the lock
>> doesn't protect dom_set[].
With the last sentence here (which I had added after having written all of the
rest of the reply, I'm afraid I've managed to confuse you:
>>>
>>Will change it to test_and_set_bit.
>...
>> > + /*
>> > + * Step 6:
>> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
>> > + * which COS the domain is using on the socket. One domain can only use
>> > + * one COS ID at same time on each socket.
>> > + */
>> > + domain_lock(d);
>> > + d->arch.psr_cos_ids[socket] = cos;
>> > + domain_unlock(d);
>> > +
>> > + /*
>> > + * Step 7:
>> > + * Then, set the dom_set bit which corresponds to domain_id to mark this
>> > + * domain has been set and the COS ID of the domain is valid.
>> > + */
>> > + set_bit(d->domain_id, info->dom_set);
>>
>> With the way things are being done above, doesn't this belong in the
>> domain_lock()-ed region?
I should have deleted this, since - as said above - the lock doesn't guard
against anything dom_set[]-wise. So ...
>Yes, should be. Thanks!
... I think you rather shouldn't do this. Instead you may want to consider whether
the other domain_lock()-ed regions couldn't be further shrunk.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-28 11:43 ` Jan Beulich
@ 2017-06-29 5:12 ` Yi Sun
2017-06-29 6:24 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-29 5:12 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-28 05:43:58, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/28/17 11:10 AM >>>
> >On 17-06-28 01:14:59, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> >> > @@ -537,7 +556,16 @@ int psr_get_val(struct domain *d, unsigned int socket,
> >> > return -ENOENT;
> >> > }
> >> >
> >> > + domain_lock(d);
> >> > + if ( !test_bit(d->domain_id, socket_info[socket].dom_set) )
> >> > + {
> >> > + d->arch.psr_cos_ids[socket] = 0;
> >> > + set_bit(d->domain_id, socket_info[socket].dom_set);
> >> > + }
> >>
> >> Any reason not to use test_and_set_bit() here? I.e. is this on any hot path?
> >> Or wait - I think it's even wrong to split the test from the set, as the lock
> >> doesn't protect dom_set[].
>
> With the last sentence here (which I had added after having written all of the
> rest of the reply, I'm afraid I've managed to confuse you:
>
> >>>
> >>Will change it to test_and_set_bit.
> >...
> >> > + /*
> >> > + * Step 6:
> >> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> >> > + * which COS the domain is using on the socket. One domain can only use
> >> > + * one COS ID at same time on each socket.
> >> > + */
> >> > + domain_lock(d);
> >> > + d->arch.psr_cos_ids[socket] = cos;
> >> > + domain_unlock(d);
> >> > +
> >> > + /*
> >> > + * Step 7:
> >> > + * Then, set the dom_set bit which corresponds to domain_id to mark this
> >> > + * domain has been set and the COS ID of the domain is valid.
> >> > + */
> >> > + set_bit(d->domain_id, info->dom_set);
> >>
> >> With the way things are being done above, doesn't this belong in the
> >> domain_lock()-ed region?
>
> I should have deleted this, since - as said above - the lock doesn't guard
> against anything dom_set[]-wise. So ...
>
> >Yes, should be. Thanks!
>
> ... I think you rather shouldn't do this. Instead you may want to consider whether
> the other domain_lock()-ed regions couldn't be further shrunk.
>
I want to confirm below two points with you:
1. remove this 'set_bit' here if above 'test_bit' is replaced to
'test_and_set_bit'.
2. For the 'be further shrunk', I think the 'domain_lock' above 'set_bit' can be
removed if 'test_and_set_bit' is used.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-29 5:12 ` Yi Sun
@ 2017-06-29 6:24 ` Jan Beulich
2017-06-29 7:21 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-29 6:24 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/29/17 7:12 AM >>>
>On 17-06-28 05:43:58, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/28/17 11:10 AM >>>
>> >On 17-06-28 01:14:59, Jan Beulich wrote:
>> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
>> >> > @@ -537,7 +556,16 @@ int psr_get_val(struct domain *d, unsigned int socket,
>> >> > return -ENOENT;
>> >> > }
>> >> >
>> >> > + domain_lock(d);
>> >> > + if ( !test_bit(d->domain_id, socket_info[socket].dom_set) )
>> >> > + {
>> >> > + d->arch.psr_cos_ids[socket] = 0;
>> >> > + set_bit(d->domain_id, socket_info[socket].dom_set);
>> >> > + }
>> >>
>> >> Any reason not to use test_and_set_bit() here? I.e. is this on any hot path?
>> >> Or wait - I think it's even wrong to split the test from the set, as the lock
>> >> doesn't protect dom_set[].
>>
>> With the last sentence here (which I had added after having written all of the
>> rest of the reply, I'm afraid I've managed to confuse you:
>>
>> >>>
>> >>Will change it to test_and_set_bit.
>> >...
>> >> > + /*
>> >> > + * Step 6:
>> >> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
>> >> > + * which COS the domain is using on the socket. One domain can only use
>> >> > + * one COS ID at same time on each socket.
>> >> > + */
>> >> > + domain_lock(d);
>> >> > + d->arch.psr_cos_ids[socket] = cos;
>> >> > + domain_unlock(d);
>> >> > +
>> >> > + /*
>> >> > + * Step 7:
>> >> > + * Then, set the dom_set bit which corresponds to domain_id to mark this
>> >> > + * domain has been set and the COS ID of the domain is valid.
>> >> > + */
>> >> > + set_bit(d->domain_id, info->dom_set);
>> >>
>> >> With the way things are being done above, doesn't this belong in the
>> >> domain_lock()-ed region?
>>
>> I should have deleted this, since - as said above - the lock doesn't guard
>> against anything dom_set[]-wise. So ...
>>
>> >Yes, should be. Thanks!
>>
>> ... I think you rather shouldn't do this. Instead you may want to consider whether
>> the other domain_lock()-ed regions couldn't be further shrunk.
>>
>I want to confirm below two points with you:
>1. remove this 'set_bit' here if above 'test_bit' is replaced to
>'test_and_set_bit'.
I don't think so, at least not for the ones still visible in context here. I've only
suggested to convert test/set pairs into test_and_set. The one at step 7 doesn't
have a test next to it, so either it was redundant with some other set (in which
case it should indeed be dropped), or it needs to stay as is.
>2. For the 'be further shrunk', I think the 'domain_lock' above 'set_bit' can be
>removed if 'test_and_set_bit' is used.
I don't think it can be removed altogether, but I think it could be moved into the
body of the if().
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-29 6:24 ` Jan Beulich
@ 2017-06-29 7:21 ` Yi Sun
0 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-29 7:21 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-29 00:24:50, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/29/17 7:12 AM >>>
> >On 17-06-28 05:43:58, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/28/17 11:10 AM >>>
> >> >On 17-06-28 01:14:59, Jan Beulich wrote:
> >> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> >> >> > @@ -537,7 +556,16 @@ int psr_get_val(struct domain *d, unsigned int socket,
> >> >> > return -ENOENT;
> >> >> > }
> >> >> >
> >> >> > + domain_lock(d);
> >> >> > + if ( !test_bit(d->domain_id, socket_info[socket].dom_set) )
> >> >> > + {
> >> >> > + d->arch.psr_cos_ids[socket] = 0;
> >> >> > + set_bit(d->domain_id, socket_info[socket].dom_set);
> >> >> > + }
> >> >>
> >> >> Any reason not to use test_and_set_bit() here? I.e. is this on any hot path?
> >> >> Or wait - I think it's even wrong to split the test from the set, as the lock
> >> >> doesn't protect dom_set[].
> >>
> >> With the last sentence here (which I had added after having written all of the
> >> rest of the reply, I'm afraid I've managed to confuse you:
> >>
> >> >>>
> >> >>Will change it to test_and_set_bit.
> >> >...
> >> >> > + /*
> >> >> > + * Step 6:
> >> >> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> >> >> > + * which COS the domain is using on the socket. One domain can only use
> >> >> > + * one COS ID at same time on each socket.
> >> >> > + */
> >> >> > + domain_lock(d);
> >> >> > + d->arch.psr_cos_ids[socket] = cos;
> >> >> > + domain_unlock(d);
> >> >> > +
> >> >> > + /*
> >> >> > + * Step 7:
> >> >> > + * Then, set the dom_set bit which corresponds to domain_id to mark this
> >> >> > + * domain has been set and the COS ID of the domain is valid.
> >> >> > + */
> >> >> > + set_bit(d->domain_id, info->dom_set);
> >> >>
> >> >> With the way things are being done above, doesn't this belong in the
> >> >> domain_lock()-ed region?
> >>
> >> I should have deleted this, since - as said above - the lock doesn't guard
> >> against anything dom_set[]-wise. So ...
> >>
> >> >Yes, should be. Thanks!
> >>
> >> ... I think you rather shouldn't do this. Instead you may want to consider whether
> >> the other domain_lock()-ed regions couldn't be further shrunk.
> >>
> >I want to confirm below two points with you:
> >1. remove this 'set_bit' here if above 'test_bit' is replaced to
> >'test_and_set_bit'.
>
> I don't think so, at least not for the ones still visible in context here. I've only
> suggested to convert test/set pairs into test_and_set. The one at step 7 doesn't
> have a test next to it, so either it was redundant with some other set (in which
> case it should indeed be dropped), or it needs to stay as is.
>
For the 'set_bit' at Step 7, it is redundant because the bit has been set anyway
when entering 'psr_set_val' if we use 'test_and_set_bit' there. So, I think we
can drop Step 7.
> >2. For the 'be further shrunk', I think the 'domain_lock' above 'set_bit' can be
> >removed if 'test_and_set_bit' is used.
>
> I don't think it can be removed altogether, but I think it could be moved into the
> body of the if().
>
I think we still need to keep the lock protection range in current codes. We
need the lock to protect the action to get 'cos id' too.
There is a scenario to explain this: if the domain's bit in dom_set has been
cleared, 'psr_get_val' and 'psr_set_val' are called almost at same time, but
'psr_get_val' is a little bit eariler. It sets dom_set bit to 1 firstly. At
that time, 'psr_set_val' checks the bit and finds it has been set, it goes to
next instruction to get old_cos. But the 'psr_get_val' may not restore the
cos id to 0 yet. So, the old_cos got in 'psr_set_val' is wrong. To avoid
this, I think should use current codes to protect the whole range.
psr_get_val() psr_set_val()
//old bit is 0, enter statement.
if (!test_and_set_bit())
{
//old bit is 1, skip statement.
if (!test_and_set_bit())
old_cos = d->arch.psr_cos_ids[socket];
domain_lock();
d->arch.psr_cos_ids[socket] = 0;
domain_unlock();
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array.
2017-06-14 1:12 ` [PATCH v12 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
@ 2017-06-29 17:56 ` Jan Beulich
0 siblings, 0 replies; 67+ messages in thread
From: Jan Beulich @ 2017-06-29 17:56 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> @@ -592,7 +615,14 @@ int psr_get_val(struct domain *d, unsigned int socket,
> /* Set value functions */
> static unsigned int get_cos_num(const struct psr_socket_info *info)
> {
> - return 0;
> + unsigned int num = 0, i;
> +
> + /* Get all features total amount. */
> + for ( i = 0; i < ARRAY_SIZE(feat_props); i++ )
> + if ( feat_props[i] )
> + num += feat_props[i]->cos_num;
> +
> + return num;
> }
The function parameter isn't being used afaics (neither by later patches), so
please drop it (in the earlier patch introducing the function).
> @@ -600,7 +630,47 @@ static int gather_val_array(uint32_t val[],
> const struct psr_socket_info *info,
> unsigned int old_cos)
> {
> - return -EINVAL;
> + unsigned int i;
> +
> + if ( !val )
> + return -EINVAL;
> +
> + /* Get all features current values according to old_cos. */
> + for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
> + {
> + unsigned int cos = old_cos, j;
> + const struct feat_node *feat = info->features[i];
> + const struct feat_props *props = feat_props[i];
> +
> + if ( !feat )
> + continue;
> +
> + if ( !props )
> + {
> + ASSERT_UNREACHABLE();
> + return -ENOENT;
> + }
> +
> + if ( array_len < props->cos_num )
> + return -ENOSPC;
> +
> + /*
> + * If old_cos exceeds current feature's cos_max, we should get
> + * default value. So assign cos to 0 which stores default value.
> + */
> + if ( cos > feat->cos_max )
> + cos = 0;
> +
> + /* Value getting order is same as feature array. */
> + for ( j = 0; j < props->cos_num; j++ )
> + val[j] = feat->cos_reg_val[cos * props->cos_num + j];
> +
> + array_len -= props->cos_num;
> +
> + val += props->cos_num;
I don't think there should be a blank line between these two ones.
> @@ -610,6 +680,60 @@ static int insert_val_into_array(uint32_t val[],
> enum cbm_type type,
> uint32_t new_val)
> {
> + const struct feat_node *feat;
> + const struct feat_props *props;
> + unsigned int i;
> +
> + ASSERT(feat_type < PSR_SOCKET_FEAT_NUM);
> +
> + /* Insert new value into array according to feature's position in array. */
> + for ( i = 0; i < feat_type; i++ )
> + {
> + if ( !info->features[i] )
> + continue;
> +
> + props = feat_props[i];
> + if ( !props )
> + {
> + ASSERT_UNREACHABLE();
> + return -ENOENT;
How about you use "continue" here and drop the info->features[i] check above?
In any event
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow.
2017-06-14 1:12 ` [PATCH v12 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
@ 2017-06-29 17:57 ` Jan Beulich
0 siblings, 0 replies; 67+ messages in thread
From: Jan Beulich @ 2017-06-29 17:57 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> static int find_cos(const uint32_t val[], unsigned int array_len,
> enum psr_feat_type feat_type,
> const struct psr_socket_info *info)
> {
> + unsigned int cos, cos_max;
> + const unsigned int *ref = info->cos_ref;
> + const struct feat_node *feat;
> +
> + /* cos_max is the one of the feature which is being set. */
> + feat = info->features[feat_type];
> + if ( !feat )
> + return -ENOENT;
> +
> + cos_max = feat->cos_max;
> +
> + for ( cos = 0; cos <= cos_max; cos++ )
> + {
> + const uint32_t *val_ptr = val;
> + unsigned int len = array_len, i;
> + int rc = 0;
> +
> + if ( cos && !ref[cos] )
> + continue;
> +
> + for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
> + {
> + const struct feat_props *props = feat_props[i];
> +
> + feat = info->features[i];
> + if ( !feat )
> + continue;
> +
> + if ( !props )
> + {
> + ASSERT_UNREACHABLE();
> + return -ENOENT;
> + }
> +
> + if ( len < props->cos_num )
> + return -ENOSPC;
> +
> + /*
> + * Compare value according to feature array order.
> + * We must follow this order because value array is assembled
> + * as this order.
> + */
> + rc = compare_val(val_ptr, feat, props, cos);
> + if ( rc < 0 )
> + return rc;
> +
> + /* If fail to match, go to next cos to compare. */
> + if ( !rc )
> + break;
> +
> + len -= props->cos_num;
> +
> + val_ptr += props->cos_num;
Unnecessary blank line again between these two. With that once again
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
2017-06-14 1:12 ` [PATCH v12 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
@ 2017-06-29 17:59 ` Jan Beulich
0 siblings, 0 replies; 67+ messages in thread
From: Jan Beulich @ 2017-06-29 17:59 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -845,12 +845,97 @@ static int find_cos(const uint32_t val[], unsigned int array_len,
> return -ENOENT;
> }
>
> +static bool fits_cos_max(const uint32_t val[],
> + uint32_t array_len,
> + const struct psr_socket_info *info,
> + unsigned int cos)
> +{
> + unsigned int i;
> +
> + for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
> + {
> + const struct feat_node *feat = info->features[i];
> + const struct feat_props *props = feat_props[i];
> +
> + if ( !feat )
> + continue;
> +
> + if ( !props )
> + {
> + ASSERT_UNREACHABLE();
> + return false;
> + }
> +
> + if ( array_len < props->cos_num )
> + return false;
> +
> + if ( cos > feat->cos_max )
> + {
> + unsigned int j;
> +
> + for ( j = 0; j < props->cos_num; j++ )
> + {
> + /*
> + * Get default value which cos id is 0.
"Get default value, the COS ID of which is zero" perhaps?
> + * For CDP:
> + * - DATA default value stored in cos_reg_val[0];
> + * - CODE default value stored in cos_reg_val[1].
Strictly speaking this doesn't belong into this patch, as you're not at CDP
yet. In fact I wonder whether this couldn't be dropped altogether.
> + */
> + uint32_t default_val = feat->cos_reg_val[j];
> +
> + if ( val[j] != default_val )
> + return false;
> + }
> + }
> +
> + array_len -= props->cos_num;
> +
> + val += props->cos_num;
Well, you already know my comment here by now, I guess.
In any case again
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-06-14 1:12 ` [PATCH v12 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
@ 2017-06-29 18:00 ` Jan Beulich
2017-06-30 5:45 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-29 18:00 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> +struct cos_write_info
> +{
> + unsigned int cos;
> + struct feat_node *feature;
> + uint32_t *val;
const?
> static int write_psr_msrs(unsigned int socket, unsigned int cos,
> uint32_t val[], unsigned int array_len,
> enum psr_feat_type feat_type)
> {
> - return -ENOENT;
> + unsigned int i;
> + struct psr_socket_info *info = get_socket_info(socket);
> + struct cos_write_info data =
> + {
> + .cos = cos,
> + .feature = info->features[feat_type],
> + .props = feat_props[feat_type],
> + };
> +
> + if ( cos > info->features[feat_type]->cos_max )
> + return -EINVAL;
> +
> + /* Skip to the feature's value head. */
> + for ( i = 0; i < feat_type; i++ )
> + {
> + if ( !info->features[i] )
> + continue;
> +
> + if ( !feat_props[i] )
> + {
> + ASSERT_UNREACHABLE();
> + return -ENOENT;
> + }
> +
> + if ( array_len <= feat_props[feat_type]->cos_num )
> + return -ENOSPC;
> +
> + array_len -= feat_props[feat_type]->cos_num;
> +
> + val += feat_props[feat_type]->cos_num;
Well, you guess it. But additionally - doesn't the array index in all three
cases above need to be i? If so, please also check other patches (including
earlier ones, where I then may have overlooked this). It is anyway worth to
consider making this skip-prior-features loop a helper function, as this isn't
the first time this occurs. Otoh this would involve quite a bit of passing
return values via pointers, so maybe that wouldn't be too efficient. And I
guess macroizing this may end up looking a little clumsy / convoluted.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-06-29 18:00 ` Jan Beulich
@ 2017-06-30 5:45 ` Yi Sun
2017-06-30 6:45 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-30 5:45 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-29 12:00:12, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> > +struct cos_write_info
> > +{
> > + unsigned int cos;
> > + struct feat_node *feature;
> > + uint32_t *val;
>
> const?
>
The member of feature, 'cos_reg_val', will be written in 'do_write_psr_msrs'.
So, I cannot use const here.
> > static int write_psr_msrs(unsigned int socket, unsigned int cos,
> > uint32_t val[], unsigned int array_len,
> > enum psr_feat_type feat_type)
> > {
> > - return -ENOENT;
> > + unsigned int i;
> > + struct psr_socket_info *info = get_socket_info(socket);
> > + struct cos_write_info data =
> > + {
> > + .cos = cos,
> > + .feature = info->features[feat_type],
> > + .props = feat_props[feat_type],
> > + };
> > +
> > + if ( cos > info->features[feat_type]->cos_max )
> > + return -EINVAL;
> > +
> > + /* Skip to the feature's value head. */
> > + for ( i = 0; i < feat_type; i++ )
> > + {
> > + if ( !info->features[i] )
> > + continue;
> > +
> > + if ( !feat_props[i] )
> > + {
> > + ASSERT_UNREACHABLE();
> > + return -ENOENT;
> > + }
> > +
> > + if ( array_len <= feat_props[feat_type]->cos_num )
> > + return -ENOSPC;
> > +
> > + array_len -= feat_props[feat_type]->cos_num;
> > +
> > + val += feat_props[feat_type]->cos_num;
>
> Well, you guess it. But additionally - doesn't the array index in all three
> cases above need to be i? If so, please also check other patches (including
Very sorry for this obvious error!
> earlier ones, where I then may have overlooked this). It is anyway worth to
> consider making this skip-prior-features loop a helper function, as this isn't
> the first time this occurs. Otoh this would involve quite a bit of passing
> return values via pointers, so maybe that wouldn't be too efficient. And I
> guess macroizing this may end up looking a little clumsy / convoluted.
>
Will implement 'skip-prior-features' function to do this. Thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-14 1:12 ` [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
@ 2017-06-30 6:40 ` Jan Beulich
2017-06-30 6:59 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 6:40 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> @@ -253,6 +271,26 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
>
> break;
>
> + case PSR_SOCKET_L3_CDP:
> + {
> + uint64_t val;
> +
> + /* Cut half of cos_max when CDP is enabled. */
> + feat->cos_max >>= 1;
I'm afraid this is off by one in the unusual but possible case of cos_max
being an even number.
> + wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
> + wrmsrl(MSR_IA32_PSR_L3_MASK(1), cat_default_val(feat->cbm_len));
> + rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
> + wrmsrl(MSR_IA32_PSR_L3_QOS_CFG,
> + val | (1ull << PSR_L3_QOS_CDP_ENABLE_BIT));
> +
> + /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
Along the lines of a comment to an earlier patch, please add a blank ahead of
the opeing paren.
> + get_cdp_code(feat, 0) = cat_default_val(feat->cbm_len);
> + get_cdp_data(feat, 0) = cat_default_val(feat->cbm_len);
Wouldn't you better do this prior to enabling CDP?
> @@ -1294,11 +1344,21 @@ static void psr_cpu_init(void)
> {
> cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
>
> - feat = feat_l3_cat;
> - feat_l3_cat = NULL;
> - feat_props[PSR_SOCKET_L3_CAT] = &l3_cat_props;
> -
> - cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CAT);
> + if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
> + !info->features[PSR_SOCKET_L3_CDP] )
Doesn't this last check mean you'd set up CAT in case would come here for
the 2nd CPU on a socket? In the end the check is simplky pointless afaict,
due to psr_cpu_init() calling cat_init_feature() only if ->feat_init is still
false. But please remove it as being potentially confusing (and inconsistent
with the else branch).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 14/23] x86: refactor psr: CDP: implement get hw info flow.
2017-06-14 1:12 ` [PATCH v12 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
@ 2017-06-30 6:41 ` Jan Beulich
0 siblings, 0 replies; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 6:41 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -217,7 +217,21 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> {
> case PSR_CBM_TYPE_L3:
> feat_type = PSR_SOCKET_L3_CAT;
> +
> + /*
> + * If type is L3 CAT but we cannot find it in feat_props array,
> + * try CDP.
> + */
> + if ( !feat_props[feat_type] )
> + feat_type = PSR_SOCKET_L3_CDP;
> +
> break;
> +
> + case PSR_CBM_TYPE_L3_DATA:
> + case PSR_CBM_TYPE_L3_CODE:
> + feat_type = PSR_SOCKET_L3_CDP;
> + break;
I think the CAT part here would better be done as a fall-through to the CDP
cases, but I won't insist on it, i.e.
Reviewed-by: Jan Beulich <jbeulich@suse.com>
either way.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-14 1:12 ` [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
@ 2017-06-30 6:42 ` Jan Beulich
2017-06-30 7:22 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 6:42 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> This patch implements L3 CDP set value related callback function.
>
> With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> v12:
> - add comment to explain how to deal with the case that user set new val
> for both DATA and CODE at same time.
> - add parameter for 'psr_cbm_type_to_feat_type' to return the feature type
> according to it.
> - use the feature type returned by 'psr_cbm_type_to_feat_type' to check
> if we need insert the new value into all items of the feature value array.
> - use conditional expression for wrmsrl.
> (suggested by Jan Beulich)
> ---
> xen/arch/x86/psr.c | 36 +++++++++++++++++++++++++++---------
> 1 file changed, 27 insertions(+), 9 deletions(-)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index aee6e3e..91b2122 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -209,7 +209,8 @@ static void free_socket_resources(unsigned int socket)
> bitmap_zero(info->dom_set, DOMID_IDLE + 1);
> }
>
> -static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> +static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type,
> + bool strict)
> {
> enum psr_feat_type feat_type = PSR_SOCKET_FEAT_UNKNOWN;
>
> @@ -222,7 +223,7 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> * If type is L3 CAT but we cannot find it in feat_props array,
> * try CDP.
> */
> - if ( !feat_props[feat_type] )
> + if ( !feat_props[feat_type] && !strict )
> feat_type = PSR_SOCKET_L3_CDP;
>
> break;
> @@ -358,11 +359,20 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
> return true;
> }
>
> +static void l3_cdp_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
> +{
> + wrmsrl(((type == PSR_CBM_TYPE_L3_DATA) ?
> + MSR_IA32_PSR_L3_MASK_DATA(cos) :
> + MSR_IA32_PSR_L3_MASK_CODE(cos)),
> + val);
> +}
> +
> static const struct feat_props l3_cdp_props = {
> .cos_num = 2,
> .type[0] = PSR_CBM_TYPE_L3_DATA,
> .type[1] = PSR_CBM_TYPE_L3_CODE,
> .get_feat_info = l3_cdp_get_feat_info,
> + .write_msr = l3_cdp_write_msr,
Adding this hook only now means the earlier CDP patches must not be applied on
their own. You should state this prominently (in the patch introducing
l3_cdp_props) for whoever is going to eventually commit (parts of) this series.
> @@ -805,17 +816,24 @@ static int insert_val_into_array(uint32_t val[],
> if ( !psr_check_cbm(feat->cbm_len, new_val) )
> return -EINVAL;
>
> - /* Value setting position is same as feature array. */
> + /*
> + * Value setting position is same as feature array.
> + * For CDP, user may set both DATA and CODE to same value. For such case,
> + * user input 'PSR_CBM_TYPE_L3' as type. The strict feature type of
> + * 'PSR_CBM_TYPE_L3' is L3 CAT. So, we should set new_val to both of DATA
> + * and CODE under such case.
> + */
> for ( i = 0; i < props->cos_num; i++ )
> {
> - if ( type == props->type[i] )
> + if ( type == props->type[i] ||
> + feat_type != psr_cbm_type_to_feat_type(type, true) )
While I think it is correct (at least up to the L2 CAT additions), it still
seems fragile to me to use != here (effectively allowing any other type to
come back). Couldn't props gain a field indicating the permitted alternative
type?
> {
> val[i] = new_val;
> - return 0;
> + ret = 0;
> }
Wouldn't it be better to return -EINVAL in a to be added else branch here
and ...
> }
>
> - return -EINVAL;
> + return ret;
> }
... to return zero here?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-06-30 5:45 ` Yi Sun
@ 2017-06-30 6:45 ` Jan Beulich
2017-06-30 7:08 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 6:45 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 7:46 AM >>>
>On 17-06-29 12:00:12, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
>> > +struct cos_write_info
>> > +{
>> > + unsigned int cos;
>> > + struct feat_node *feature;
>> > + uint32_t *val;
>>
>> const?
>>
>The member of feature, 'cos_reg_val', will be written in 'do_write_psr_msrs'.
>So, I cannot use const here.
cos_reg_val is a member of struct feat_node, not struct cos_write_info. Note also
the difference in field names.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 16/23] x86: L2 CAT: implement CPU init flow.
2017-06-14 1:12 ` [PATCH v12 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
@ 2017-06-30 6:58 ` Jan Beulich
2017-06-30 7:27 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 6:58 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> @@ -279,10 +281,14 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
> switch ( type )
> {
> case PSR_SOCKET_L3_CAT:
> + case PSR_SOCKET_L2_CAT:
> /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
>
> - wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
> + if ( type == PSR_SOCKET_L3_CAT )
> + wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
> + else
> + wrmsrl(MSR_IA32_PSR_L2_MASK(0), cat_default_val(feat->cbm_len));
Once again I think a conditional expression would yield in easier to read code,
as that would make even more obvious that the second argument is the same for
both cases.
> @@ -317,7 +323,8 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
> return;
>
> printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
> - ((type == PSR_SOCKET_L3_CDP) ? "CDP" : "L3 CAT"),
> + ((type == PSR_SOCKET_L3_CDP) ? "CDP" :
> + ((type == PSR_SOCKET_L3_CAT) ? "L3 CAT": "L2 CAT")),
At this point it would probably be better to have a static const lookup array
for the types, or for this descriptive string to be passed into the function.
> @@ -375,6 +382,12 @@ static const struct feat_props l3_cdp_props = {
> .write_msr = l3_cdp_write_msr,
> };
>
> +/* L2 CAT props */
> +static const struct feat_props l2_cat_props = {
> + .cos_num = 1,
> + .type[0] = PSR_CBM_TYPE_L2,
> +};
Same remark as for CDP regarding the NULL function pointers left around here
until the later patches populate them.
> @@ -1407,6 +1424,19 @@ static void psr_cpu_init(void)
> info->feat_init = true;
> }
>
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> + if ( regs.b & PSR_RESOURCE_TYPE_L2 )
> + {
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, ®s);
> +
> + feat = feat_l2_cat;
> + feat_l2_cat = NULL;
> + feat_props[PSR_SOCKET_L2_CAT] = &l2_cat_props;
> + cat_init_feature(®s, feat, info, PSR_SOCKET_L2_CAT);
> +
> + info->feat_init = true;
This recurring setting of feat_init starts looking suspicious here. Why can't
this be done once at the end of the function, outside of any if()-s?
> --- a/xen/include/asm-x86/psr.h
> +++ b/xen/include/asm-x86/psr.h
> @@ -23,6 +23,7 @@
>
> /* Resource Type Enumeration */
> #define PSR_RESOURCE_TYPE_L3 0x2
> +#define PSR_RESOURCE_TYPE_L2 0x4
These are used in psr.c only afaics, so shouldn't be put in a header.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 17/23] x86: L2 CAT: implement get hw info flow.
2017-06-14 1:12 ` [PATCH v12 17/23] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-06-30 6:59 ` Jan Beulich
0 siblings, 0 replies; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 6:59 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> This patch implements L2 CAT get HW info flow and interface in sysctl.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-30 6:40 ` Jan Beulich
@ 2017-06-30 6:59 ` Yi Sun
2017-06-30 7:33 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-30 6:59 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-30 00:40:35, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> > @@ -253,6 +271,26 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
> >
> > break;
> >
> > + case PSR_SOCKET_L3_CDP:
> > + {
> > + uint64_t val;
> > +
> > + /* Cut half of cos_max when CDP is enabled. */
> > + feat->cos_max >>= 1;
>
> I'm afraid this is off by one in the unusual but possible case of cos_max
> being an even number.
>
This accords to spec:
"For CDP operations, COS_MAX_CDP is equal to (CPUID.(EAX=10H, ECX=1):EDX.COS_MAX_CAT >>1)."
HW should make sure it is even number.
> > + wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
> > + wrmsrl(MSR_IA32_PSR_L3_MASK(1), cat_default_val(feat->cbm_len));
> > + rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
> > + wrmsrl(MSR_IA32_PSR_L3_QOS_CFG,
> > + val | (1ull << PSR_L3_QOS_CDP_ENABLE_BIT));
> > +
> > + /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
>
> Along the lines of a comment to an earlier patch, please add a blank ahead of
> the opeing paren.
>
Got it.
> > + get_cdp_code(feat, 0) = cat_default_val(feat->cbm_len);
> > + get_cdp_data(feat, 0) = cat_default_val(feat->cbm_len);
>
> Wouldn't you better do this prior to enabling CDP?
>
Sure.
> > @@ -1294,11 +1344,21 @@ static void psr_cpu_init(void)
> > {
> > cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> >
> > - feat = feat_l3_cat;
> > - feat_l3_cat = NULL;
> > - feat_props[PSR_SOCKET_L3_CAT] = &l3_cat_props;
> > -
> > - cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CAT);
> > + if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
> > + !info->features[PSR_SOCKET_L3_CDP] )
>
> Doesn't this last check mean you'd set up CAT in case would come here for
> the 2nd CPU on a socket? In the end the check is simplky pointless afaict,
> due to psr_cpu_init() calling cat_init_feature() only if ->feat_init is still
> false. But please remove it as being potentially confusing (and inconsistent
> with the else branch).
>
Ok, will remove the last check here.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-06-30 6:45 ` Jan Beulich
@ 2017-06-30 7:08 ` Yi Sun
0 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-30 7:08 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-30 00:45:33, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 7:46 AM >>>
> >On 17-06-29 12:00:12, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:25 AM >>>
> >> > +struct cos_write_info
> >> > +{
> >> > + unsigned int cos;
> >> > + struct feat_node *feature;
> >> > + uint32_t *val;
> >>
> >> const?
> >>
> >The member of feature, 'cos_reg_val', will be written in 'do_write_psr_msrs'.
> >So, I cannot use const here.
>
> cos_reg_val is a member of struct feat_node, not struct cos_write_info. Note also
> the difference in field names.
>
Oh, sorry. I mis-understood. Should add 'const' for 'val'.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-30 6:42 ` Jan Beulich
@ 2017-06-30 7:22 ` Yi Sun
2017-06-30 8:54 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-30 7:22 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau
On 17-06-30 00:42:22, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> > +static void l3_cdp_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
> > +{
> > + wrmsrl(((type == PSR_CBM_TYPE_L3_DATA) ?
> > + MSR_IA32_PSR_L3_MASK_DATA(cos) :
> > + MSR_IA32_PSR_L3_MASK_CODE(cos)),
> > + val);
> > +}
> > +
> > static const struct feat_props l3_cdp_props = {
> > .cos_num = 2,
> > .type[0] = PSR_CBM_TYPE_L3_DATA,
> > .type[1] = PSR_CBM_TYPE_L3_CODE,
> > .get_feat_info = l3_cdp_get_feat_info,
> > + .write_msr = l3_cdp_write_msr,
>
> Adding this hook only now means the earlier CDP patches must not be applied on
> their own. You should state this prominently (in the patch introducing
> l3_cdp_props) for whoever is going to eventually commit (parts of) this series.
>
Ok, I will highlight it in CDP init patch. Also apply it to L2 CAT patches.
> > @@ -805,17 +816,24 @@ static int insert_val_into_array(uint32_t val[],
> > if ( !psr_check_cbm(feat->cbm_len, new_val) )
> > return -EINVAL;
> >
> > - /* Value setting position is same as feature array. */
> > + /*
> > + * Value setting position is same as feature array.
> > + * For CDP, user may set both DATA and CODE to same value. For such case,
> > + * user input 'PSR_CBM_TYPE_L3' as type. The strict feature type of
> > + * 'PSR_CBM_TYPE_L3' is L3 CAT. So, we should set new_val to both of DATA
> > + * and CODE under such case.
> > + */
> > for ( i = 0; i < props->cos_num; i++ )
> > {
> > - if ( type == props->type[i] )
> > + if ( type == props->type[i] ||
> > + feat_type != psr_cbm_type_to_feat_type(type, true) )
>
> While I think it is correct (at least up to the L2 CAT additions), it still
> seems fragile to me to use != here (effectively allowing any other type to
> come back). Couldn't props gain a field indicating the permitted alternative
> type?
>
Thanks for the good idea. Will add 'enum psr_feat_type alt_type;' in props
to handle such case.
> > {
> > val[i] = new_val;
> > - return 0;
> > + ret = 0;
> > }
>
> Wouldn't it be better to return -EINVAL in a to be added else branch here
> and ...
>
> > }
> >
> > - return -EINVAL;
> > + return ret;
> > }
>
> ... to return zero here?
>
Sure.
> Jan
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 16/23] x86: L2 CAT: implement CPU init flow.
2017-06-30 6:58 ` Jan Beulich
@ 2017-06-30 7:27 ` Yi Sun
2017-06-30 7:36 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-30 7:27 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-30 00:58:47, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> > @@ -279,10 +281,14 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
> > switch ( type )
> > {
> > case PSR_SOCKET_L3_CAT:
> > + case PSR_SOCKET_L2_CAT:
> > /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> > feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
> >
> > - wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
> > + if ( type == PSR_SOCKET_L3_CAT )
> > + wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
> > + else
> > + wrmsrl(MSR_IA32_PSR_L2_MASK(0), cat_default_val(feat->cbm_len));
>
> Once again I think a conditional expression would yield in easier to read code,
> as that would make even more obvious that the second argument is the same for
> both cases.
>
Ok, will use conditional expression here to make codes clearer.
> > @@ -317,7 +323,8 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
> > return;
> >
> > printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
> > - ((type == PSR_SOCKET_L3_CDP) ? "CDP" : "L3 CAT"),
> > + ((type == PSR_SOCKET_L3_CDP) ? "CDP" :
> > + ((type == PSR_SOCKET_L3_CAT) ? "L3 CAT": "L2 CAT")),
>
> At this point it would probably be better to have a static const lookup array
> for the types, or for this descriptive string to be passed into the function.
>
Got it.
> > @@ -375,6 +382,12 @@ static const struct feat_props l3_cdp_props = {
> > .write_msr = l3_cdp_write_msr,
> > };
> >
> > +/* L2 CAT props */
> > +static const struct feat_props l2_cat_props = {
> > + .cos_num = 1,
> > + .type[0] = PSR_CBM_TYPE_L2,
> > +};
>
> Same remark as for CDP regarding the NULL function pointers left around here
> until the later patches populate them.
>
Will highlight it in comments.
> > @@ -1407,6 +1424,19 @@ static void psr_cpu_init(void)
> > info->feat_init = true;
> > }
> >
> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> > + if ( regs.b & PSR_RESOURCE_TYPE_L2 )
> > + {
> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, ®s);
> > +
> > + feat = feat_l2_cat;
> > + feat_l2_cat = NULL;
> > + feat_props[PSR_SOCKET_L2_CAT] = &l2_cat_props;
> > + cat_init_feature(®s, feat, info, PSR_SOCKET_L2_CAT);
> > +
> > + info->feat_init = true;
>
> This recurring setting of feat_init starts looking suspicious here. Why can't
> this be done once at the end of the function, outside of any if()-s?
>
I am afraid there is no any feature found through CPUID so I set feat_init in
every statement that a feature is found.
> > --- a/xen/include/asm-x86/psr.h
> > +++ b/xen/include/asm-x86/psr.h
> > @@ -23,6 +23,7 @@
> >
> > /* Resource Type Enumeration */
> > #define PSR_RESOURCE_TYPE_L3 0x2
> > +#define PSR_RESOURCE_TYPE_L2 0x4
>
> These are used in psr.c only afaics, so shouldn't be put in a header.
>
PSR_RESOURCE_TYPE_L3 is used in sysctl.c too. For L2, I just want to keep it
same place as L3.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-30 6:59 ` Yi Sun
@ 2017-06-30 7:33 ` Jan Beulich
2017-06-30 8:04 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 7:33 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 9:01 AM >>>
>On 17-06-30 00:40:35, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
>> > @@ -253,6 +271,26 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
>> >
>> > break;
>> >
>> > + case PSR_SOCKET_L3_CDP:
>> > + {
>> > + uint64_t val;
>> > +
>> > + /* Cut half of cos_max when CDP is enabled. */
>> > + feat->cos_max >>= 1;
>>
>> I'm afraid this is off by one in the unusual but possible case of cos_max
>> being an even number.
>>
>This accords to spec:
>"For CDP operations, COS_MAX_CDP is equal to (CPUID.(EAX=10H, ECX=1):EDX.COS_MAX_CAT >>1)."
>
>HW should make sure it is even number.
And how about someone using the command line option to shrink the to be used set?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 16/23] x86: L2 CAT: implement CPU init flow.
2017-06-30 7:27 ` Yi Sun
@ 2017-06-30 7:36 ` Jan Beulich
2017-06-30 8:05 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 7:36 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 9:28 AM >>>
>On 17-06-30 00:58:47, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
>> > @@ -1407,6 +1424,19 @@ static void psr_cpu_init(void)
>> > info->feat_init = true;
>> > }
>> >
>> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
>> > + if ( regs.b & PSR_RESOURCE_TYPE_L2 )
>> > + {
>> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, ®s);
>> > +
>> > + feat = feat_l2_cat;
>> > + feat_l2_cat = NULL;
>> > + feat_props[PSR_SOCKET_L2_CAT] = &l2_cat_props;
>> > + cat_init_feature(®s, feat, info, PSR_SOCKET_L2_CAT);
>> > +
>> > + info->feat_init = true;
>>
>> This recurring setting of feat_init starts looking suspicious here. Why can't
>> this be done once at the end of the function, outside of any if()-s?
>>
>I am afraid there is no any feature found through CPUID so I set feat_init in
>every statement that a feature is found.
Well, even if no feature is available, you're done with initializing by the time
you finish this function.
>> > --- a/xen/include/asm-x86/psr.h
>> > +++ b/xen/include/asm-x86/psr.h
>> > @@ -23,6 +23,7 @@
>> >
>> > /* Resource Type Enumeration */
>> > #define PSR_RESOURCE_TYPE_L3 0x2
>> > +#define PSR_RESOURCE_TYPE_L2 0x4
>>
>> These are used in psr.c only afaics, so shouldn't be put in a header.
>>
>PSR_RESOURCE_TYPE_L3 is used in sysctl.c too. For L2, I just want to keep it
>same place as L3.
I must have overlooked that one - of course they should stay together.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-30 7:33 ` Jan Beulich
@ 2017-06-30 8:04 ` Yi Sun
2017-06-30 9:18 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-30 8:04 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-30 01:33:02, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 9:01 AM >>>
> >On 17-06-30 00:40:35, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> >> > @@ -253,6 +271,26 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
> >> >
> >> > break;
> >> >
> >> > + case PSR_SOCKET_L3_CDP:
> >> > + {
> >> > + uint64_t val;
> >> > +
> >> > + /* Cut half of cos_max when CDP is enabled. */
> >> > + feat->cos_max >>= 1;
> >>
> >> I'm afraid this is off by one in the unusual but possible case of cos_max
> >> being an even number.
> >>
> >This accords to spec:
> >"For CDP operations, COS_MAX_CDP is equal to (CPUID.(EAX=10H, ECX=1):EDX.COS_MAX_CAT >>1)."
> >
> >HW should make sure it is even number.
>
> And how about someone using the command line option to shrink the to be used set?
>
Good question. The command line option saved in 'opt_cos_max', even it is
not even number and less than 'EDX.COS_MAX_CAT' so that the 'cos_max' here is
same as it, the right shift operation does not cause any issue I think.
The description in docs/misc/xen-command-line.markdown is clear that "the cos_max
in use will automatically reduce to half when CDP is enabled". E.g. 'opt_cos_max'
is 5, then we get 2 for CDP. I think user should be aware of this by reading the
markdown file.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 16/23] x86: L2 CAT: implement CPU init flow.
2017-06-30 7:36 ` Jan Beulich
@ 2017-06-30 8:05 ` Yi Sun
0 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-06-30 8:05 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-30 01:36:57, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 9:28 AM >>>
> >On 17-06-30 00:58:47, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/14/17 3:26 AM >>>
> >> > @@ -1407,6 +1424,19 @@ static void psr_cpu_init(void)
> >> > info->feat_init = true;
> >> > }
> >> >
> >> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> >> > + if ( regs.b & PSR_RESOURCE_TYPE_L2 )
> >> > + {
> >> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, ®s);
> >> > +
> >> > + feat = feat_l2_cat;
> >> > + feat_l2_cat = NULL;
> >> > + feat_props[PSR_SOCKET_L2_CAT] = &l2_cat_props;
> >> > + cat_init_feature(®s, feat, info, PSR_SOCKET_L2_CAT);
> >> > +
> >> > + info->feat_init = true;
> >>
> >> This recurring setting of feat_init starts looking suspicious here. Why can't
> >> this be done once at the end of the function, outside of any if()-s?
> >>
> >I am afraid there is no any feature found through CPUID so I set feat_init in
> >every statement that a feature is found.
>
> Well, even if no feature is available, you're done with initializing by the time
> you finish this function.
>
Ok, I get your point. Thanks!
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-30 7:22 ` Yi Sun
@ 2017-06-30 8:54 ` Yi Sun
2017-06-30 9:33 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-30 8:54 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-30 15:22:56, Yi Sun wrote:
> On 17-06-30 00:42:22, Jan Beulich wrote:
>
> > > @@ -805,17 +816,24 @@ static int insert_val_into_array(uint32_t val[],
> > > if ( !psr_check_cbm(feat->cbm_len, new_val) )
> > > return -EINVAL;
> > >
> > > - /* Value setting position is same as feature array. */
> > > + /*
> > > + * Value setting position is same as feature array.
> > > + * For CDP, user may set both DATA and CODE to same value. For such case,
> > > + * user input 'PSR_CBM_TYPE_L3' as type. The strict feature type of
> > > + * 'PSR_CBM_TYPE_L3' is L3 CAT. So, we should set new_val to both of DATA
> > > + * and CODE under such case.
> > > + */
> > > for ( i = 0; i < props->cos_num; i++ )
> > > {
> > > - if ( type == props->type[i] )
> > > + if ( type == props->type[i] ||
> > > + feat_type != psr_cbm_type_to_feat_type(type, true) )
> >
> > While I think it is correct (at least up to the L2 CAT additions), it still
> > seems fragile to me to use != here (effectively allowing any other type to
> > come back). Couldn't props gain a field indicating the permitted alternative
> > type?
> >
> Thanks for the good idea. Will add 'enum psr_feat_type alt_type;' in props
> to handle such case.
>
> > > {
> > > val[i] = new_val;
> > > - return 0;
> > > + ret = 0;
> > > }
> >
> > Wouldn't it be better to return -EINVAL in a to be added else branch here
> > and ...
> >
After reading codes again, I think we cannot return -EINVAL in else branch here.
E.g. for CDP, user wants to set CODE. Then, the 'type' is CODE. At the first
iteration, the props->type[0] is DATA which does not match 'type'. But we cannot
return error here. We should iterate next 'type[]'.
After iterating all type[], if we still do not find matching one, return the
error back. So, I use 'ret' here.
> > > }
> > >
> > > - return -EINVAL;
> > > + return ret;
> > > }
> >
> > ... to return zero here?
> >
> Sure.
>
> > Jan
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > https://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-30 8:04 ` Yi Sun
@ 2017-06-30 9:18 ` Jan Beulich
2017-07-04 1:40 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 9:18 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 10:05 AM >>>
>On 17-06-30 01:33:02, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 9:01 AM >>>
>> >This accords to spec:
>> >"For CDP operations, COS_MAX_CDP is equal to (CPUID.(EAX=10H, ECX=1):EDX.COS_MAX_CAT >>1)."
>> >
>> >HW should make sure it is even number.
>>
>> And how about someone using the command line option to shrink the to be used set?
>>
>Good question. The command line option saved in 'opt_cos_max', even it is
>not even number and less than 'EDX.COS_MAX_CAT' so that the 'cos_max' here is
>same as it, the right shift operation does not cause any issue I think.
>
>The description in docs/misc/xen-command-line.markdown is clear that "the cos_max
>in use will automatically reduce to half when CDP is enabled". E.g. 'opt_cos_max'
>is 5, then we get 2 for CDP. I think user should be aware of this by reading the
>markdown file.
I don't really follow all this argumentation - what's the problem of simply subtracting
1 before doing the shift? Talking of which - isn't the general cos_max > 1 also
insufficient for the CDP case?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-30 8:54 ` Yi Sun
@ 2017-06-30 9:33 ` Jan Beulich
2017-06-30 11:29 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 9:33 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 10:56 AM >>>
>On 17-06-30 15:22:56, Yi Sun wrote:
>> On 17-06-30 00:42:22, Jan Beulich wrote:
>> > > {
>> > > val[i] = new_val;
>> > > - return 0;
>> > > + ret = 0;
>> > > }
>> >
>> > Wouldn't it be better to return -EINVAL in a to be added else branch here
>> > and ...
>> >
>After reading codes again, I think we cannot return -EINVAL in else branch here.
>E.g. for CDP, user wants to set CODE. Then, the 'type' is CODE. At the first
>iteration, the props->type[0] is DATA which does not match 'type'. But we cannot
>return error here. We should iterate next 'type[]'.
But that's why you're adding the second check in the if(). If neither of the two
conditions are true, this is an error, isn't it? On the converse, why would you
return 0 if the first loop iteration is fine but the second isn't.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-30 9:33 ` Jan Beulich
@ 2017-06-30 11:29 ` Yi Sun
2017-06-30 12:02 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-06-30 11:29 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-30 03:33:17, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 10:56 AM >>>
> >On 17-06-30 15:22:56, Yi Sun wrote:
> >> On 17-06-30 00:42:22, Jan Beulich wrote:
> >> > > {
> >> > > val[i] = new_val;
> >> > > - return 0;
> >> > > + ret = 0;
> >> > > }
> >> >
> >> > Wouldn't it be better to return -EINVAL in a to be added else branch here
> >> > and ...
> >> >
> >After reading codes again, I think we cannot return -EINVAL in else branch here.
> >E.g. for CDP, user wants to set CODE. Then, the 'type' is CODE. At the first
> >iteration, the props->type[0] is DATA which does not match 'type'. But we cannot
> >return error here. We should iterate next 'type[]'.
>
> But that's why you're adding the second check in the if(). If neither of the two
> conditions are true, this is an error, isn't it? On the converse, why would you
> return 0 if the first loop iteration is fine but the second isn't.
>
The second check in the if() is for the case to set both DATA and CODE at same
time. The above example I wrote is for the case that user just wants to set CODE
but not DATA.
Under such case, the input 'feat_type' is CDP so that the second check is always
false.
The input 'type' is CODE. The props->type[0] is DATA and props->type[1] is CODE.
In the first iteration, the props->type[0] is DATA so that it does not match
'type' and the second check is false too. If we use else branch here, it will
enter the branch and return -EVINVAL. But this is not we want, right? We hope
the second iteration should be executed to set CODE.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-30 11:29 ` Yi Sun
@ 2017-06-30 12:02 ` Jan Beulich
2017-07-03 6:33 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-06-30 12:02 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 1:30 PM >>>
>The input 'type' is CODE. The props->type[0] is DATA and props->type[1] is CODE.
>In the first iteration, the props->type[0] is DATA so that it does not match
>'type' and the second check is false too. If we use else branch here, it will
>enter the branch and return -EVINVAL. But this is not we want, right? We hope
>the second iteration should be executed to set CODE.
I see. That'll then call for yet another solution; I don't think the code should
stay as is.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-30 12:02 ` Jan Beulich
@ 2017-07-03 6:33 ` Yi Sun
2017-07-03 7:01 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-07-03 6:33 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau
On 17-06-30 06:02:32, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 1:30 PM >>>
> >The input 'type' is CODE. The props->type[0] is DATA and props->type[1] is CODE.
> >In the first iteration, the props->type[0] is DATA so that it does not match
> >'type' and the second check is false too. If we use else branch here, it will
> >enter the branch and return -EVINVAL. But this is not we want, right? We hope
> >the second iteration should be executed to set CODE.
>
> I see. That'll then call for yet another solution; I don't think the code should
> stay as is.
>
Then, how about ASSERT() at the beginning to check if input 'type' is correct?
enum cbm_type {
PSR_CBM_TYPE_L3,
PSR_CBM_TYPE_L3_DATA,
PSR_CBM_TYPE_L3_CODE,
PSR_CBM_TYPE_L2,
};
ASSERT((type >= props->type[0] && type <= props->type[props->cos_num - 1]) ||
type == props->alt_type);
We don't need 'ret' anymore with above check.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-07-03 6:33 ` Yi Sun
@ 2017-07-03 7:01 ` Jan Beulich
2017-07-03 8:40 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-07-03 7:01 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.07.17 at 08:33, <yi.y.sun@linux.intel.com> wrote:
> On 17-06-30 06:02:32, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 1:30 PM >>>
>> >The input 'type' is CODE. The props->type[0] is DATA and props->type[1] is CODE.
>> >In the first iteration, the props->type[0] is DATA so that it does not match
>> >'type' and the second check is false too. If we use else branch here, it will
>> >enter the branch and return -EVINVAL. But this is not we want, right? We hope
>> >the second iteration should be executed to set CODE.
>>
>> I see. That'll then call for yet another solution; I don't think the code should
>> stay as is.
>>
> Then, how about ASSERT() at the beginning to check if input 'type' is
> correct?
> enum cbm_type {
> PSR_CBM_TYPE_L3,
> PSR_CBM_TYPE_L3_DATA,
> PSR_CBM_TYPE_L3_CODE,
> PSR_CBM_TYPE_L2,
> };
>
> ASSERT((type >= props->type[0] && type <= props->type[props->cos_num - 1]) ||
> type == props->alt_type);
Baking in ordering assumptions? No, please don't.
> We don't need 'ret' anymore with above check.
So in a release build you'd then do what in case of a bad type finding
its way in?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-07-03 7:01 ` Jan Beulich
@ 2017-07-03 8:40 ` Yi Sun
2017-07-03 9:18 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-07-03 8:40 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-07-03 01:01:09, Jan Beulich wrote:
> >>> On 03.07.17 at 08:33, <yi.y.sun@linux.intel.com> wrote:
> > On 17-06-30 06:02:32, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 1:30 PM >>>
> >> >The input 'type' is CODE. The props->type[0] is DATA and props->type[1] is CODE.
> >> >In the first iteration, the props->type[0] is DATA so that it does not match
> >> >'type' and the second check is false too. If we use else branch here, it will
> >> >enter the branch and return -EVINVAL. But this is not we want, right? We hope
> >> >the second iteration should be executed to set CODE.
> >>
> >> I see. That'll then call for yet another solution; I don't think the code should
> >> stay as is.
> >>
> > Then, how about ASSERT() at the beginning to check if input 'type' is
> > correct?
> > enum cbm_type {
> > PSR_CBM_TYPE_L3,
> > PSR_CBM_TYPE_L3_DATA,
> > PSR_CBM_TYPE_L3_CODE,
> > PSR_CBM_TYPE_L2,
> > };
> >
> > ASSERT((type >= props->type[0] && type <= props->type[props->cos_num - 1]) ||
> > type == props->alt_type);
>
> Baking in ordering assumptions? No, please don't.
>
> > We don't need 'ret' anymore with above check.
>
> So in a release build you'd then do what in case of a bad type finding
> its way in?
>
> Jan
To decide the return value, we have to know if input 'type' is correct or not.
There are two ways:
1. Check if input 'type' without iteration, like the above codes. Becaue you
don't agree the ordering assumptions, this way is not good.
2. Use iteration, like the original codes. Record if the statement is hit.
If yes, return 0. Otherwise, return -EINVAL. The original codes are below:
for ( i = 0; i < props->cos_num; i++ )
{
if ( type == props->type[i] || type == props->alt_type )
{
val[i] = new_val;
ret = 0;
}
}
I think the main issue you don't like in the original codes is that
the 'ret = 0' may happen for several times. How about below change?
for ( i = 0; i < props->cos_num; i++ )
{
if ( type == props->type[i] || type == props->alt_type )
{
val[i] = new_val;
if ( ret )
ret = 0;
}
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-07-03 8:40 ` Yi Sun
@ 2017-07-03 9:18 ` Jan Beulich
2017-07-03 12:52 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-07-03 9:18 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.07.17 at 10:40, <yi.y.sun@linux.intel.com> wrote:
> On 17-07-03 01:01:09, Jan Beulich wrote:
>> >>> On 03.07.17 at 08:33, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-06-30 06:02:32, Jan Beulich wrote:
>> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 1:30 PM >>>
>> >> >The input 'type' is CODE. The props->type[0] is DATA and props->type[1] is
> CODE.
>> >> >In the first iteration, the props->type[0] is DATA so that it does not match
>> >> >'type' and the second check is false too. If we use else branch here, it
> will
>> >> >enter the branch and return -EVINVAL. But this is not we want, right? We
> hope
>> >> >the second iteration should be executed to set CODE.
>> >>
>> >> I see. That'll then call for yet another solution; I don't think the code
> should
>> >> stay as is.
>> >>
>> > Then, how about ASSERT() at the beginning to check if input 'type' is
>> > correct?
>> > enum cbm_type {
>> > PSR_CBM_TYPE_L3,
>> > PSR_CBM_TYPE_L3_DATA,
>> > PSR_CBM_TYPE_L3_CODE,
>> > PSR_CBM_TYPE_L2,
>> > };
>> >
>> > ASSERT((type >= props->type[0] && type <= props->type[props->cos_num - 1]) ||
>> > type == props->alt_type);
>>
>> Baking in ordering assumptions? No, please don't.
>>
>> > We don't need 'ret' anymore with above check.
>>
>> So in a release build you'd then do what in case of a bad type finding
>> its way in?
>>
>> Jan
>
> To decide the return value, we have to know if input 'type' is correct or
> not.
> There are two ways:
> 1. Check if input 'type' without iteration, like the above codes. Becaue you
> don't agree the ordering assumptions, this way is not good.
> 2. Use iteration, like the original codes. Record if the statement is hit.
> If yes, return 0. Otherwise, return -EINVAL. The original codes are below:
> for ( i = 0; i < props->cos_num; i++ )
> {
> if ( type == props->type[i] || type == props->alt_type )
> {
> val[i] = new_val;
> ret = 0;
> }
> }
>
> I think the main issue you don't like in the original codes is that
> the 'ret = 0' may happen for several times. How about below change?
> for ( i = 0; i < props->cos_num; i++ )
> {
> if ( type == props->type[i] || type == props->alt_type )
> {
> val[i] = new_val;
> if ( ret )
> ret = 0;
> }
> }
No, the multiple assignments would be no issue at all. As said
before, what I dislike is the wrongness of the return value if
the first iteration sets ret to zero, but a subsequent one
wouldn't. In that case, an error should be signaled.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-07-03 9:18 ` Jan Beulich
@ 2017-07-03 12:52 ` Yi Sun
2017-07-03 13:02 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-07-03 12:52 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-07-03 03:18:05, Jan Beulich wrote:
> >>> On 03.07.17 at 10:40, <yi.y.sun@linux.intel.com> wrote:
> > On 17-07-03 01:01:09, Jan Beulich wrote:
> >> >>> On 03.07.17 at 08:33, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-06-30 06:02:32, Jan Beulich wrote:
> >> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 1:30 PM >>>
> >
> > To decide the return value, we have to know if input 'type' is correct or
> > not.
> > There are two ways:
> > 1. Check if input 'type' without iteration, like the above codes. Becaue you
> > don't agree the ordering assumptions, this way is not good.
> > 2. Use iteration, like the original codes. Record if the statement is hit.
> > If yes, return 0. Otherwise, return -EINVAL. The original codes are below:
> > for ( i = 0; i < props->cos_num; i++ )
> > {
> > if ( type == props->type[i] || type == props->alt_type )
> > {
> > val[i] = new_val;
> > ret = 0;
> > }
> > }
> >
> > I think the main issue you don't like in the original codes is that
> > the 'ret = 0' may happen for several times. How about below change?
> > for ( i = 0; i < props->cos_num; i++ )
> > {
> > if ( type == props->type[i] || type == props->alt_type )
> > {
> > val[i] = new_val;
> > if ( ret )
> > ret = 0;
> > }
> > }
>
> No, the multiple assignments would be no issue at all. As said
> before, what I dislike is the wrongness of the return value if
> the first iteration sets ret to zero, but a subsequent one
> wouldn't. In that case, an error should be signaled.
>
Ok. Then, how about below change? Thanks!
int ret = 0;
for ( i = 0; i < props->cos_num; i++ )
{
if ( type == props->type[i] )
{
val[i] = new_val;
ret = 0;
break;
}
else if ( type == props->alt_type )
val[i] = new_val;
else
ret = -EINVAL;
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-07-03 12:52 ` Yi Sun
@ 2017-07-03 13:02 ` Jan Beulich
0 siblings, 0 replies; 67+ messages in thread
From: Jan Beulich @ 2017-07-03 13:02 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.07.17 at 14:52, <yi.y.sun@linux.intel.com> wrote:
> Ok. Then, how about below change? Thanks!
> int ret = 0;
> for ( i = 0; i < props->cos_num; i++ )
> {
> if ( type == props->type[i] )
> {
> val[i] = new_val;
> ret = 0;
> break;
> }
> else if ( type == props->alt_type )
> val[i] = new_val;
> else
> ret = -EINVAL;
> }
This looks okay to me.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-30 9:18 ` Jan Beulich
@ 2017-07-04 1:40 ` Yi Sun
2017-07-04 7:28 ` Jan Beulich
0 siblings, 1 reply; 67+ messages in thread
From: Yi Sun @ 2017-07-04 1:40 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-30 03:18:53, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 10:05 AM >>>
> >On 17-06-30 01:33:02, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 9:01 AM >>>
> >> >This accords to spec:
> >> >"For CDP operations, COS_MAX_CDP is equal to (CPUID.(EAX=10H, ECX=1):EDX.COS_MAX_CAT >>1)."
> >> >
> >> >HW should make sure it is even number.
> >>
> >> And how about someone using the command line option to shrink the to be used set?
> >>
> >Good question. The command line option saved in 'opt_cos_max', even it is
> >not even number and less than 'EDX.COS_MAX_CAT' so that the 'cos_max' here is
> >same as it, the right shift operation does not cause any issue I think.
> >
> >The description in docs/misc/xen-command-line.markdown is clear that "the cos_max
> >in use will automatically reduce to half when CDP is enabled". E.g. 'opt_cos_max'
> >is 5, then we get 2 for CDP. I think user should be aware of this by reading the
> >markdown file.
>
> I don't really follow all this argumentation - what's the problem of simply subtracting
> 1 before doing the shift? Talking of which - isn't the general cos_max > 1 also
There will be problem if we substract 1 before doing the shift for the even
number. E.g. the original cos_max is 2, (2-1)>>1 will be 0. That is not we
want. For the odd number, direct right shift is good too. E.g. the original
cos_max is 3, 3 >> 1 is 1 which is what we want.
> insufficient for the CDP case?
We do not need check if cos_max > 1 because the 'cos_max = 0' works for us.
That means only COS ID 0 can be used. You can see, we use "cos <= cos_max" to
find COS ID. Of course, the ID 0 saves the default value which cannot be
overwritten.
>
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-07-04 1:40 ` Yi Sun
@ 2017-07-04 7:28 ` Jan Beulich
2017-07-05 1:45 ` Yi Sun
0 siblings, 1 reply; 67+ messages in thread
From: Jan Beulich @ 2017-07-04 7:28 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 04.07.17 at 03:40, <yi.y.sun@linux.intel.com> wrote:
> On 17-06-30 03:18:53, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 10:05 AM >>>
>> >On 17-06-30 01:33:02, Jan Beulich wrote:
>> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 9:01 AM >>>
>> >> >This accords to spec:
>> >> >"For CDP operations, COS_MAX_CDP is equal to (CPUID.(EAX=10H, ECX=1):EDX.COS_MAX_CAT >>1)."
>> >> >
>> >> >HW should make sure it is even number.
>> >>
>> >> And how about someone using the command line option to shrink the to be used set?
>> >>
>> >Good question. The command line option saved in 'opt_cos_max', even it is
>> >not even number and less than 'EDX.COS_MAX_CAT' so that the 'cos_max' here is
>> >same as it, the right shift operation does not cause any issue I think.
>> >
>> >The description in docs/misc/xen-command-line.markdown is clear that "the cos_max
>> >in use will automatically reduce to half when CDP is enabled". E.g. 'opt_cos_max'
>> >is 5, then we get 2 for CDP. I think user should be aware of this by reading the
>> >markdown file.
>>
>> I don't really follow all this argumentation - what's the problem of simply subtracting
>> 1 before doing the shift? Talking of which - isn't the general cos_max > 1 also
>
> There will be problem if we substract 1 before doing the shift for the even
> number. E.g. the original cos_max is 2, (2-1)>>1 will be 0. That is not we
> want. For the odd number, direct right shift is good too. E.g. the original
> cos_max is 3, 3 >> 1 is 1 which is what we want.
What we want is
raw CDP
0 unusable
1 0
2 0
3 1
4 1
5 2
6 2
Other than what you say, for original cos_max = 2 we indeed need
to convert it to 0, or else CDP would use indexes 2 ( = 1 * 2 + 0)
and 3 ( = 1 * 2 + 1), exceeding the mandated upper bound.
>> insufficient for the CDP case?
>
> We do not need check if cos_max > 1 because the 'cos_max = 0' works for us.
> That means only COS ID 0 can be used. You can see, we use "cos <= cos_max" to
> find COS ID. Of course, the ID 0 saves the default value which cannot be
> overwritten.
Please see the (current) check in init_psr_cat(), which your series
moves into init_psr(). Logically in the CDP case we should require
cos_max >= 3 to have at least one usable COS ID.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-07-04 7:28 ` Jan Beulich
@ 2017-07-05 1:45 ` Yi Sun
0 siblings, 0 replies; 67+ messages in thread
From: Yi Sun @ 2017-07-05 1:45 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-07-04 01:28:32, Jan Beulich wrote:
> >>> On 04.07.17 at 03:40, <yi.y.sun@linux.intel.com> wrote:
> > On 17-06-30 03:18:53, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 10:05 AM >>>
> >> >On 17-06-30 01:33:02, Jan Beulich wrote:
> >> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 06/30/17 9:01 AM >>>
> > There will be problem if we substract 1 before doing the shift for the even
> > number. E.g. the original cos_max is 2, (2-1)>>1 will be 0. That is not we
> > want. For the odd number, direct right shift is good too. E.g. the original
> > cos_max is 3, 3 >> 1 is 1 which is what we want.
>
> What we want is
>
> raw CDP
> 0 unusable
> 1 0
> 2 0
> 3 1
> 4 1
> 5 2
> 6 2
>
> Other than what you say, for original cos_max = 2 we indeed need
> to convert it to 0, or else CDP would use indexes 2 ( = 1 * 2 + 0)
> and 3 ( = 1 * 2 + 1), exceeding the mandated upper bound.
>
> >> insufficient for the CDP case?
> >
> > We do not need check if cos_max > 1 because the 'cos_max = 0' works for us.
> > That means only COS ID 0 can be used. You can see, we use "cos <= cos_max" to
> > find COS ID. Of course, the ID 0 saves the default value which cannot be
> > overwritten.
>
> Please see the (current) check in init_psr_cat(), which your series
> moves into init_psr(). Logically in the CDP case we should require
> cos_max >= 3 to have at least one usable COS ID.
>
> Jan
Thank you! You are right. I will modify codes to do so.
BRs,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 67+ messages in thread
end of thread, other threads:[~2017-07-05 1:46 UTC | newest]
Thread overview: 67+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-14 1:12 [PATCH v12 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-06-14 1:12 ` [PATCH v12 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-06-14 1:12 ` [PATCH v12 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
2017-06-14 1:12 ` [PATCH v12 03/23] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
2017-06-14 1:12 ` [PATCH v12 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
2017-06-28 7:12 ` Jan Beulich
2017-06-28 9:07 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
2017-06-28 7:13 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 06/23] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
2017-06-14 1:12 ` [PATCH v12 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
2017-06-28 7:14 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
2017-06-28 7:14 ` Jan Beulich
2017-06-28 9:09 ` Yi Sun
2017-06-28 11:43 ` Jan Beulich
2017-06-29 5:12 ` Yi Sun
2017-06-29 6:24 ` Jan Beulich
2017-06-29 7:21 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
2017-06-29 17:56 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
2017-06-29 17:57 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
2017-06-29 17:59 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
2017-06-29 18:00 ` Jan Beulich
2017-06-30 5:45 ` Yi Sun
2017-06-30 6:45 ` Jan Beulich
2017-06-30 7:08 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
2017-06-30 6:40 ` Jan Beulich
2017-06-30 6:59 ` Yi Sun
2017-06-30 7:33 ` Jan Beulich
2017-06-30 8:04 ` Yi Sun
2017-06-30 9:18 ` Jan Beulich
2017-07-04 1:40 ` Yi Sun
2017-07-04 7:28 ` Jan Beulich
2017-07-05 1:45 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
2017-06-30 6:41 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
2017-06-30 6:42 ` Jan Beulich
2017-06-30 7:22 ` Yi Sun
2017-06-30 8:54 ` Yi Sun
2017-06-30 9:33 ` Jan Beulich
2017-06-30 11:29 ` Yi Sun
2017-06-30 12:02 ` Jan Beulich
2017-07-03 6:33 ` Yi Sun
2017-07-03 7:01 ` Jan Beulich
2017-07-03 8:40 ` Yi Sun
2017-07-03 9:18 ` Jan Beulich
2017-07-03 12:52 ` Yi Sun
2017-07-03 13:02 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
2017-06-30 6:58 ` Jan Beulich
2017-06-30 7:27 ` Yi Sun
2017-06-30 7:36 ` Jan Beulich
2017-06-30 8:05 ` Yi Sun
2017-06-14 1:12 ` [PATCH v12 17/23] x86: L2 CAT: implement get hw info flow Yi Sun
2017-06-30 6:59 ` Jan Beulich
2017-06-14 1:12 ` [PATCH v12 18/23] x86: L2 CAT: implement get value flow Yi Sun
2017-06-14 1:12 ` [PATCH v12 19/23] x86: L2 CAT: implement set " Yi Sun
2017-06-14 1:12 ` [PATCH v12 20/23] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
2017-06-14 1:12 ` [PATCH v12 21/23] tools: L2 CAT: support show cbm " Yi Sun
2017-06-14 1:12 ` [PATCH v12 22/23] tools: L2 CAT: support set " Yi Sun
2017-06-14 1:12 ` [PATCH v12 23/23] docs: add L2 CAT description in docs Yi Sun
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.