* [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c
@ 2017-05-03 8:44 Yi Sun
2017-05-03 8:44 ` [PATCH v11 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
` (22 more replies)
0 siblings, 23 replies; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Hi all,
We plan to bring a new PSR (Platform Shared Resource) feature called
Intel L2 Cache Allocation Technology (L2 CAT) to Xen. It has been enabled
in Linux Kernel.
Besides the L2 CAT implementaion, we refactor the psr.c to make it more
flexible and easily to extend to add new features. We abstract the general
operations of all features and encapsulate them into a structure. Then,
the development of new feature is simple to mainly implement these callback
functions.
The patch set can be found at:
https://github.com/yisun-git/xen.git l2_cat_v11-3
---
Acked and Reviewed list before V11:
a - Acked-by
r - Reviewed-by
r patch 1 - docs: create Cache Allocation Technology (CAT) and Code and
Data Prioritization (CDP) feature document
a patch 2 - x86: move cpuid_count_leaf from cpuid.c to processor.h.
ar patch 3 - x86: refactor psr: remove L3 CAT/CDP codes.
a patch 20 - tools: L2 CAT: support get HW info for L2 CAT.
(Acked for Hypervisor side)
a patch 23 - docs: add L2 CAT description in docs.
---
V11 change list:
Patch 1:
- modify data structure descriptions according to code changes.
- modify revision.
Patch 4:
- handle 'feat_init'.
- merge main data structures implementation into CPU init/free patch,
including commit messages and change history.
(suggested by Jan Beulich)
- remove MSR restore action which is unnecessary.
(suggested by Jan Beulich)
- move 'type[]' declaration into this patch.
(suggested by Jan Beulich)
- modify comment.
(suggested by Jan Beulich)
Patch 5:
- replace 'feat_init_done()' to 'feat_init' flag.
(suggested by Jan Beulich)
- adjust parameters positions when calling 'psr_assoc_cos'.
(suggested by Jan Beulich)
- add comment to explain why to check 'psr_cos_ids'.
Patch 6:
- changes about 'cos_max' and 'cbm_len'.
(suggested by Jan Beulich)
- change about 'feat_props'.
(suggested by Jan Beulich)
- declare 'PSR_SOCKET_FEAT_UNKNOWN' to handle error case.
(suggested by Jan Beulich)
- use 'ASSERT' to check data.
(suggested by Jan Beulich)
- fix comment of 'get_feat_info'.
(suggested by Jan Beulich)
- fix a bug when checking 'feat_type'.
(suggested by Jan Beulich)
Patch 7:
- declare a 'switch()' wide variable 'val32' in domctl.
(suggested by Jan Beulich)
- remove 'get_val' callback function which is replaced by generic codes.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- do not 'ASSERT' domain pointer.
(suggested by Jan Beulich)
- modify commit message.
Patch 8:
- define 'dom_ids[]' and implement related flows.
- restore domain cos id to 0 in 'psr_get_val'.
- rename 'write_psr_msr' to 'write_psr_msrs' and change its parameters to
handle value array the feature's all MSRs.
- fix coding style issue.
(suggested by Jan Beulich)
- do not need check 'cos' in ASSERT.
(suggested by Jan Beulich)
- rename 'insert_val_to_array' to 'insert_val_into_array'.
(suggested by Jan Beulich)
- remove 'ref_lock' from parameter list in 'find_cos' and 'pick_avail_cos'.
(suggested by Jan Beulich)
- remove ASSERT check to 'ref_lock' in 'find_cos' and 'pick_avail_cos'.
(suggested by Jan Beulich)
- fix a bug for checking 'feat_type'.
(suggested by Jan Beulich)
- move 'free_array' label.
(suggested by Jan Beulich)
- modify comments and commit message.
Patch 9:
- changes about 'feat_props'.
(suggested by Jan Beulich)
- changes about 'get_val'. It is replaced by generic codes.
(suggested by Jan Beulich)
Patch 10:
- move 'compare_val' implementation from CDP patch to this.
(suggested by Jan Beulich)
- remove 'rc' in 'compare_val' to make codes clearer.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- fix bug.
(suggested by Jan Beulich)
- remove unnecessary variable initialization.
(suggested by Jan Beulich)
- remove unnecessary comment.
(suggested by Jan Beulich)
- adjust array check place prior to using the array.
(suggested by Jan Beulich)
- add comment.
Patch 11:
- remove unnecessary variable initialization.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- replace 'get_val' calling to generic codes.
(suggested by Jan Beulich)
Patch 12:
- rename 'write_psr_msr' to 'write_psr_msrs'.
- rename 'do_write_psr_msr' to 'do_write_psr_msrs'.
- change parameters and codes of 'write_psr_msrs' to handle value array.
- add 'feat_type' in 'struct cos_write_info' to handle props array.
- in 'do_write_psr_msrs', write value array into msrs according to
'props->type[i]'.
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
- move check from 'do_write_psr_msrs' to 'write_psr_msrs'.
(suggested by Jan Beulich)
- change about 'cos_max'.
(suggested by Jan Beulich)
- change about 'feat_props'.
(suggested by Jan Beulich)
Patch 13:
- changes about 'feat_props'.
(suggested by Jan Beulich)
- remove MSR restore action which is unnecessary.
(suggested by Jan Beulich)
- modify commit message.
Patch 14:
- modify 'psr_get_info' flow to make it simple to cover CDP case.
Patch 15:
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
- changes about setting both CDP DATA and CODE at same time.
- move 'type[]' declaration into previous patch which introduced 'cos_num'.
(suggested by Jan Beulich)
- changes about 'type[]'.
(suggested by Jan Beulich)
- move 'compare_val' to previous patch.
(suggested by Jan Beulich)
- changes about 'get_val' which has been replace by generic codes.
(suggested by Jan Beulich)
- remove 'restore_default_val' which is unnecessary now.
(suggested by Jan Beulich)
Patch 16:
- move l2 cat 'type[]' assignement into 'psr_cpu_init'.
- remove COS MSR restore action in 'cpu_init_feature'.
- set 'feat_init' to true after CPU init.
- modify commit message.
Patch 18:
- remove "get_val' assignment because it has been replaced by generic
codes.
(suggested by Jan Beulich)
Patch 19:
- remove 'domctl->u.psr_cat_op.data' check because it has been moved into
'psr_set_val'.
(suggested by Jan Beulich)
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
Patch 20:
- acked by Jan for Hypervisor side changes.
Patch 22:
- revert change in v9 to handle the case to set both CODE and DATA for CDP
at same time. This must be handled in hypervisor.
Yi Sun (23):
docs: create Cache Allocation Technology (CAT) and Code and Data
Prioritization (CDP) feature document
x86: move cpuid_count_leaf from cpuid.c to processor.h.
x86: refactor psr: remove L3 CAT/CDP codes.
x86: refactor psr: L3 CAT: implement main data structures, CPU init
and free flows.
x86: refactor psr: L3 CAT: implement Domain init/free and schedule
flows.
x86: refactor psr: L3 CAT: implement get hw info flow.
x86: refactor psr: L3 CAT: implement get value flow.
x86: refactor psr: L3 CAT: set value: implement framework.
x86: refactor psr: L3 CAT: set value: assemble features value array.
x86: refactor psr: L3 CAT: set value: implement cos finding flow.
x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
x86: refactor psr: L3 CAT: set value: implement write msr flow.
x86: refactor psr: CDP: implement CPU init flow.
x86: refactor psr: CDP: implement get hw info flow.
x86: refactor psr: CDP: implement set value callback function.
x86: L2 CAT: implement CPU init flow.
x86: L2 CAT: implement get hw info flow.
x86: L2 CAT: implement get value flow.
x86: L2 CAT: implement set value flow.
tools: L2 CAT: support get HW info for L2 CAT.
tools: L2 CAT: support show cbm for L2 CAT.
tools: L2 CAT: support set cbm for L2 CAT.
docs: add L2 CAT description in docs.
docs/features/intel_psr_cat_cdp.pandoc | 461 +++++++++++
docs/man/xl.pod.1.in | 25 +-
docs/misc/xl-psr.markdown | 18 +-
tools/libxc/include/xenctrl.h | 7 +-
tools/libxc/xc_psr.c | 45 +-
tools/libxl/libxl.h | 9 +
tools/libxl/libxl_psr.c | 28 +-
tools/libxl/libxl_types.idl | 1 +
tools/xl/xl_cmdtable.c | 6 +-
tools/xl/xl_psr.c | 168 ++--
xen/arch/x86/cpuid.c | 6 -
xen/arch/x86/domctl.c | 51 +-
xen/arch/x86/psr.c | 1343 ++++++++++++++++++++++++--------
xen/arch/x86/sysctl.c | 38 +-
xen/include/asm-x86/msr-index.h | 1 +
xen/include/asm-x86/processor.h | 7 +
xen/include/asm-x86/psr.h | 26 +-
xen/include/public/domctl.h | 2 +
xen/include/public/sysctl.h | 3 +-
19 files changed, 1807 insertions(+), 438 deletions(-)
create mode 100644 docs/features/intel_psr_cat_cdp.pandoc
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* [PATCH v11 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
` (21 subsequent siblings)
22 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch creates CAT and CDP feature document in doc/features/. It describes
key points to implement L3 CAT/CDP and L2 CAT which is described in details in
Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES".
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v11:
- modify data structure descriptions according to code changes.
- modify revision.
v9:
- add 'CMT' to the list of acronyms.
(suggested by Wei Liu)
- change feature list to feature array.
- modify data structure descriptions according to latest codes.
- modify revision.
v8:
- change revision info.
(suggested by Konrad Rzeszutek Wilk)
- add content int 'Areas for improvement'.
(suggested by Konrad Rzeszutek Wilk)
v7:
- correct typo.
(suggested by Konrad Rzeszutek Wilk)
- replace application/VM to domain.
(suggested by Konrad Rzeszutek Wilk)
- amend description of `feat_mask` to make it clearer.
(suggested by Konrad Rzeszutek Wilk)
- update revision.
(suggested by Konrad Rzeszutek Wilk)
- other minor fixes.
(suggested by Konrad Rzeszutek Wilk)
v6:
- write a new feature document to cover L3 CAT/CDP and L2 CAT.
(suggested by Kevin Tian)
- adjust 'Terminology' position in document.
(suggested by Dario Faggioli)
- fix wordings.
(suggested by Dario Faggioli, Kevin Tian and Konrad Rzeszutek Wilk)
- add SDM chapter title in commit message.
(suggested by Konrad Rzeszutek Wilk)
- add more explanations.
(suggested by Kevin Tian)
v4:
- change file name to be more descriptive, 'intel_psr_l2_cat.pandoc'.
(suggested by Dario Faggioli)
- remove 'Ext' and 'New' prefixes.
(suggested by Dario Faggioli)
- remove change log in Revison part.
(suggested by Dario Faggioli)
- adjust Xen release number to 4.9 to show this feature targets 4.9.
(suggested by Dario Faggioli)
- provide 'Terminology' and more sections.
(suggested by Dario Faggioli)
- fix wordings.
(suggested by Konrad Rzeszutek Wilk)
- remove chapter number.
(suggested by Konrad Rzeszutek Wilk)
v3:
- make design document be a patch.
(suggested by Konrad Rzeszutek Wilk)
v2:
- provide chapter for the L2 CAT.
(suggested by Meng Xu)
---
docs/features/intel_psr_cat_cdp.pandoc | 461 +++++++++++++++++++++++++++++++++
1 file changed, 461 insertions(+)
create mode 100644 docs/features/intel_psr_cat_cdp.pandoc
diff --git a/docs/features/intel_psr_cat_cdp.pandoc b/docs/features/intel_psr_cat_cdp.pandoc
new file mode 100644
index 0000000..132e708
--- /dev/null
+++ b/docs/features/intel_psr_cat_cdp.pandoc
@@ -0,0 +1,461 @@
+% Intel Cache Allocation Technology and Code and Data Prioritization Features
+% Revision 1.11
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+ Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+ Component(s): Hypervisor, toolstack
+
+ Hardware: L3 CAT: Haswell and beyond CPUs
+ CDP : Broadwell and beyond CPUs
+ L2 CAT: Atom codename Goldmont and beyond CPUs
+---------------- ----------------------------------------------------
+
+# Terminology
+
+* CAT Cache Allocation Technology
+* CBM Capacity BitMasks
+* CDP Code and Data Prioritization
+* CMT Cache Monitoring Technology
+* COS/CLOS Class of Service
+* MSRs Machine Specific Registers
+* PSR Intel Platform Shared Resource
+
+# Overview
+
+Intel provides a set of allocation capabilities including Cache Allocatation
+Technology (CAT) and Code and Data Prioritization (CDP).
+
+CAT allows an OS or hypervisor to control allocation of a CPU's shared cache
+based on application/domain priority or Class of Service (COS). Each COS is
+configured using capacity bitmasks (CBMs) which represent cache capacity and
+indicate the degree of overlap and isolation between classes. Once CAT is co-
+nfigured, the processor allows access to portions of cache according to the
+established COS. Intel Xeon processor E5 v4 family (and some others) introduce
+capabilities to configure and make use of the CAT mechanism on the L3 cache.
+Intel Goldmont processor provides support for control over the L2 cache.
+
+Code and Data Prioritization (CDP) Technology is an extension of CAT. CDP
+enables isolation and separate prioritization of code and data fetches to
+the L3 cache in a SW configurable manner, which can enable workload priorit-
+ization and tuning of cache capacity to the characteristics of the workload.
+CDP extends CAT by providing separate code and data masks per Class of Service
+(COS). When SW configures to enable CDP, L3 CAT is disabled.
+
+# User details
+
+* Feature Enabling:
+
+ Add "psr=cat" to boot line parameter to enable all supported level CAT featu-
+ res. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW.
+
+* xl interfaces:
+
+ 1. `psr-cat-show [OPTIONS] domain-id`:
+
+ Show L2 CAT or L3 CAT/CDP CBM of the domain designated by Xen domain-id.
+
+ Option `-l`:
+ `-l2`: Show cbm for L2 cache.
+ `-l3`: Show cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ 2. `psr-cat-set [OPTIONS] domain-id cbm`:
+
+ Set L2 CAT or L3 CAT/CDP CBM to the domain designated by Xen domain-id.
+
+ Option `-s`: Specify the socket to process, otherwise all sockets are
+ processed.
+
+ Option `-l`:
+ `-l2`: Specify cbm for L2 cache.
+ `-l3`: Specify cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ Option `-c` or `-d`:
+ `-c`: Set L3 CDP code cbm.
+ `-d`: Set L3 CDP data cbm.
+
+ 3. `psr-hwinfo [OPTIONS]`:
+
+ Show CMT & L2 CAT & L3 CAT/CDP HW information on every socket.
+
+ Option `-m, --cmt`: Show Cache Monitoring Technology (CMT) hardware info.
+
+ Option `-a, --cat`: Show CAT/CDP hardware info.
+
+# Technical details
+
+L3 CAT/CDP and L2 CAT are all members of Intel PSR features, they share the base
+PSR infrastructure in Xen.
+
+## Hardware perspective
+
+ CAT/CDP defines a range of MSRs to assign different cache access patterns
+ which are known as CBMs, each CBM is associated with a COS.
+
+ ```
+ E.g. L2 CAT:
+ +----------------------------+----------------+
+ IA32_PQR_ASSOC | MSR (per socket) | Address |
+ +----+---+-------+ +----------------------------+----------------+
+ | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
+ +----+---+-------+ +----------------------------+----------------+
+ └-------------> | ... | ... |
+ +----------------------------+----------------+
+ | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
+ +----------------------------+----------------+
+ ```
+
+ L3 CAT/CDP uses a range of MSRs from 0xC90 ~ 0xC90+n (n<128).
+
+ L2 CAT uses a range of MSRs from 0xD10 ~ 0xD10+n (n<64), following the L3
+ CAT/CDP MSRs, setting different L2 cache accessing patterns from L3 cache is
+ supported.
+
+ Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the
+ hardware indicating the cache space a domain should be limited to as well as
+ providing an indication of overlap and isolation in the CAT-capable cache from
+ other domains contending for the cache.
+
+ Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
+ note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
+ 0FF0H, 003CH, etc.).
+
+ ```
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Default Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Overlapped Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | | | | | Isolated Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | | |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+ ```
+
+ We can get the CBM length through CPUID. The default value of CBM is calcul-
+ ated by `(1ull << cbm_len) - 1`. That is a fully open bitmask, all ones bitm-
+ ask. The COS[0] always stores the default value without change.
+
+ There is a `IA32_PQR_ASSOC` register which stores the COS ID of the VCPU. HW
+ enforces cache allocation according to the corresponding CBM.
+
+## The relationship between L3 CAT/CDP and L2 CAT
+
+ HW may support all features. By default, CDP is disabled on the processor.
+ If the L3 CAT MSRs are used without enabling CDP, the processor operates in
+ a traditional CAT-only mode. When CDP is enabled:
+ * the CAT mask MSRs are re-mapped into interleaved pairs of mask MSRs for
+ data or code fetches.
+ * the range of COS for CAT is re-indexed, with the lower-half of the COS
+ range available for CDP.
+
+ L2 CAT is independent of L3 CAT/CDP, which means L2 CAT can be enabled while
+ L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are both enabled.
+
+ As a requirement, the bits of CBM of CAT/CDP must be continuous.
+
+ N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same associate
+ register `IA32_PQR_ASSOC`, which means one COS is associated with a pair of
+ L2 CAT CBM and L3 CAT/CDP CBM.
+
+ Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or other
+ PSR features in future). In some cases, a domain is permitted to have a COS
+ that is beyond one (or more) of PSR features but within the others. For
+ instance, let's assume the max COS of L2 CAT is 8 but the max COS of L3
+ CAT is 16, when a domain is assigned 9 as COS, the L3 CAT CBM associated to
+ COS 9 would be enforced, but for L2 CAT, the HW works as default value is
+ set since COS 9 is beyond the max COS (8) of L2 CAT.
+
+## Design Overview
+
+* Core COS/CBM association
+
+ When enforcing CAT/CDP, all cores of domains have the same default COS (COS0)
+ which is associated with the fully open CBM (all ones bitmask) to access all
+ cache. The default COS is used only in hypervisor and is transparent to tool
+ stack and user.
+
+ System administrator can change PSR allocation policy at runtime by tool stack.
+ Since L2 CAT shares COS with L3 CAT/CDP, a COS corresponds to a 2-tuple, like
+ [L2 CBM, L3 CBM] with only-CAT enabled, when CDP is enabled, one COS correspo-
+ nds to a 3-tuple, like [L2 CBM, L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT
+ nor L3 CDP is enabled, things would be easier, one COS corresponds to one L2
+ CBM.
+
+* VCPU schedule
+
+ When context switch happens, the COS of VCPU is written to per-thread MSR
+ `IA32_PQR_ASSOC`, and then hardware enforces cache allocation according to
+ the corresponding CBM.
+
+* Multi-sockets
+
+ Different sockets may have different CAT/CDP capability (e.g. max COS) alth-
+ ough it is consistent on the same socket. So the capability of per-socket CAT/
+ CDP is specified.
+
+ 'psr-cat-set' can set CBM for one domain per socket. On each socket, we main-
+ tain a COS array for all domains. One domain uses one COS at one time. One COS
+ stores the CBM of the domain to work. So, when a VCPU of the domain is migrat-
+ ed from socket 1 to socket 2, it follows configuration on socket 2.
+
+ E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets do-
+ main 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
+ is migrated from socket 1 to 2, the COS ID used is 7, that means 0x3f is the
+ CBM to work for this domain 1 now.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+ 1. Boot line parameter "psr=cat" enables L2 CAT and L3 CAT if hardware suppo-
+ rted. "psr=cdp" enables CDP if hardware supported.
+
+ 2. SYSCTL:
+ - XEN_SYSCTL_PSR_CAT_get_l3_info: Get L3 CAT/CDP information.
+ - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
+
+ 3. DOMCTL:
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: Get L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: Set L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE: Get CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: Set CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA: Get CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: Set CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
+
+* xl interfaces:
+
+ 1. psr-cat-show -lX domain-id
+ Show LX cbm for a domain.
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
+
+ 2. psr-cat-set -lX domain-id cbm
+ Set LX cbm for a domain.
+ => XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
+
+ 3. psr-hwinfo
+ Show PSR HW information, including L3 CAT/CDP/L2 CAT
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info
+
+* Key data structure:
+
+ 1. Feature properties
+
+ ```
+ static const struct feat_props {
+ unsigned int cos_num;
+ enum cbm_type type[PSR_MAX_COS_NUM];
+ bool (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len);
+ void (*write_msr)(unsigned int cos, uint32_t val, enum cbm_type type);
+ } *feat_props[PSR_SOCKET_FEAT_NUM];
+ ```
+
+ Every feature has its own properties, e.g. some data and actions. A
+ feature property pointer array is declared to save every feature's
+ properties.
+
+ - Member `cos_num`
+
+ `cos_num` is the number of COS registers the feature uses, e.g. L3/L2
+ CAT uses 1 register but CDP uses 2 registers.
+
+ - Member `type`
+
+ `type` is an array to save all 'enum cbm_type' values of the feature.
+ It is used with cos_num together to get/write a feature's COS registers
+ values one by one.
+
+ - Member `get_feat_info`
+
+ `get_feat_info` is used to return feature HW info through sysctl.
+
+ - Member `write_msr`
+
+ `write_msr` is used to write out feature MSR register.
+
+ 2. Feature node
+
+ ```
+ struct feat_node {
+ unsigned int cos_max;
+ unsigned int cbm_len;
+ uint32_t cos_reg_val[MAX_COS_REG_CNT];
+ };
+ ```
+
+ When a PSR enforcement feature is enabled, it will be added into a
+ feature array.
+
+ - Member `cos_max`
+
+ `cos_max` is one of the hardware info of CAT. It means the max number
+ of COS registers. As L3 CAT/CDP/L2 CAT all have it, it is declared in
+ `feat_node`.
+
+ - Member `cbm_len`
+
+ `cbm_len` is one of the hardware info of CAT. It means the max number
+ of bits to set.
+
+ - Member `cos_reg_val`
+
+ `cos_reg_val` is an array to maintain the value set in all COS registers
+ of the feature. The array is indexed by COS ID.
+
+ 3. Per-socket PSR features information structure
+
+ ```
+ struct psr_socket_info {
+ struct feat_node *features[PSR_SOCKET_FEAT_NUM];
+ bool feat_init;
+ unsigned int cos_ref[MAX_COS_REG_CNT];
+ spinlock_t ref_lock;
+ DECLARE_BITMAP(dom_ids, DOMID_IDLE + 1);
+ };
+ ```
+
+ We collect all PSR allocation features information of a socket in this
+ `struct psr_socket_info`.
+
+ - Member `features`
+
+ `features` is a pointer array to save all enabled features poniters
+ according to feature position defined in `enum psr_feat_type`.
+
+ - Member `feat_init`
+
+ `feat_init` is a flag, to indicate whether the CPU init on a socket
+ has been done.
+
+ - Member `cos_ref`
+
+ `cos_ref` is an array which maintains the reference of one COS. It maps
+ to cos_reg_val[MAX_COS_REG_NUM] in `struct feat_node`. If one COS is
+ used by one domain, the corresponding reference will increase by one. If
+ a domain releases the COS, the reference will decrease by one. The array
+ is indexed by COS ID.
+
+ - Member `ref_lock`
+
+ `ref_lock` is a spin lock to protect `cos_ref`.
+
+ - Member `dom_ids`
+
+ `dom_ids` is a bitmap, every bit corresponds to a domain. Index is
+ domain_id. It is used to help restore domain_id to 0 when a socket is
+ offline and then online again.
+
+# Limitations
+
+CAT/CDP can only work on HW which enables it(check by CPUID). So far, there is
+no HW which enables both L2 CAT and L3 CAT/CDP. But SW implementation has cons-
+idered such scenario to enable both L2 CAT and L3 CAT/CDP.
+
+# Testing
+
+We can execute above xl commands to verify L2 CAT and L3 CAT/CDP on different
+HWs support them.
+
+For example:
+ root@:~$ xl psr-hwinfo --cat
+ Cache Allocation Technology (CAT): L2
+ Socket ID : 0
+ Maximum COS : 3
+ CBM length : 8
+ Default CBM : 0xff
+
+ root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
+
+ root@:~$ xl psr-cat-show -l2 1
+ Socket ID : 0
+ Default CBM : 0xff
+ ID NAME CBM
+ 1 ubuntu14 0x7f
+
+# Areas for improvement
+
+A hexadecimal number is used to set/show CBM for a domain now. Although this
+is convenient to cover overlap/isolated bitmask requirement, it is not user-
+friendly.
+
+To improve this, the libxl interfaces can be wrapped in libvirt to provide more
+usr-friendly interfaces to user, e.g. a percentage number of the cache to set
+and show.
+
+# Known issues
+
+N/A
+
+# References
+
+"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date Revision Version Notes
+---------- -------- -------- -------------------------------------------
+2016-08-12 1.0 Xen 4.9 Design document written
+2017-02-13 1.7 Xen 4.9 Changes:
+ 1. Modify the design document to cover L3
+ CAT/CDP and L2 CAT;
+ 2. Fix typos;
+ 3. Amend description of `feat_mask` to make
+ it clearer;
+ 4. Other minor changes.
+2017-02-15 1.8 Xen 4.9 Changes:
+ 1. Add content in 'Areas for improvement';
+ 2. Adjust revision number.
+2017-03-16 1.9 Xen 4.9 Changes:
+ 1. Add 'CMT' in 'Terminology';
+ 2. Change 'feature list' to 'feature array'.
+ 3. Modify data structure descriptions.
+ 4. Adjust revision number.
+2017-05-03 1.11 Xen 4.9 Changes:
+ 1. Modify data structure descriptions.
+ 2. Adjust revision number.
+---------- -------- -------- -------------------------------------------
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-05-03 8:44 ` [PATCH v11 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 03/23] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
` (20 subsequent siblings)
22 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch moves 'cpuid_count_leaf' from cpuid.c to processor.h to
make it available to external codes.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
v9:
- create this patch alone to move 'cpuid_count_leaf'.
(suggested by Wei Liu)
v6:
- use 'struct cpuid_leaf' in psr.c. So we have to access 'cpuid_count_leaf'
which has to be moved to processor.h.
(suggested by Andrew Cooper)
---
xen/arch/x86/cpuid.c | 6 ------
xen/include/asm-x86/processor.h | 7 +++++++
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/cpuid.c b/xen/arch/x86/cpuid.c
index 1c6a6c6..a4a0c4c 100644
--- a/xen/arch/x86/cpuid.c
+++ b/xen/arch/x86/cpuid.c
@@ -35,12 +35,6 @@ static void cpuid_leaf(uint32_t leaf, struct cpuid_leaf *data)
cpuid(leaf, &data->a, &data->b, &data->c, &data->d);
}
-static void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
- struct cpuid_leaf *data)
-{
- cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
-}
-
static void sanitise_featureset(uint32_t *fs)
{
/* for_each_set_bit() uses unsigned longs. Extend with zeroes. */
diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h
index 75632d9..9283fe6 100644
--- a/xen/include/asm-x86/processor.h
+++ b/xen/include/asm-x86/processor.h
@@ -13,6 +13,7 @@
#include <asm/types.h>
#include <asm/cpufeature.h>
#include <asm/desc.h>
+#include <asm/x86_emulate.h>
#endif
#include <asm/x86-defns.h>
@@ -264,6 +265,12 @@ static always_inline unsigned int cpuid_count_ebx(
return ebx;
}
+static always_inline void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
+ struct cpuid_leaf *data)
+{
+ cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
+}
+
static inline unsigned long read_cr0(void)
{
unsigned long cr0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 03/23] x86: refactor psr: remove L3 CAT/CDP codes.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-05-03 8:44 ` [PATCH v11 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-05-03 8:44 ` [PATCH v11 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
` (19 subsequent siblings)
22 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
The current cache allocation codes in psr.c do not consider
future features addition and are not friendly to extend.
To make psr.c be more flexible to add new features and fulfill
the program principle, open for extension but closed for
modification, we have to refactor the psr.c:
1. Analyze cache allocation features and abstract general data
structures.
2. Analyze the init and all other functions flow, abstract all
steps that different features may have different implementations.
Make these steps be callback functions and register feature
specific fuctions. Then, the main processes will not be changed
when introducing a new feature.
Because the quantity of refactor codes is big and the logics are
changed a lot, it will cause reviewers confused if just change
old codes. Reviewers have to understand both old codes and new
implementations. After review iterations from V1 to V3, Jan has
proposed to remove all old cache allocation codes firstly, then
implement new codes step by step. This will help to make codes
be more easily reviewable.
There is no construction without destruction. So, this patch
removes all current L3 CAT/CDP codes in psr.c. The following
patches will introduce the new mechanism.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v4:
- create this patch to make codes easily understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 470 +----------------------------------------------------
1 file changed, 5 insertions(+), 465 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0b5073c..96a8589 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -23,24 +23,6 @@
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
-struct psr_cat_cbm {
- union {
- uint64_t cbm;
- struct {
- uint64_t code;
- uint64_t data;
- };
- };
- unsigned int ref;
-};
-
-struct psr_cat_socket_info {
- unsigned int cbm_len;
- unsigned int cos_max;
- struct psr_cat_cbm *cos_to_cbm;
- spinlock_t cbm_lock;
-};
-
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -48,26 +30,11 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
-static unsigned long *__read_mostly cat_socket_enable;
-static struct psr_cat_socket_info *__read_mostly cat_socket_info;
-static unsigned long *__read_mostly cdp_socket_enable;
-
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
-static unsigned int __read_mostly opt_cos_max = 255;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
-static struct psr_cat_cbm *temp_cos_to_cbm;
-
-static unsigned int get_socket_cpu(unsigned int socket)
-{
- if ( likely(socket < nr_sockets) )
- return cpumask_any(socket_cpumask[socket]);
-
- return nr_cpu_ids;
-}
-
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -107,9 +74,6 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
- if ( val_str && !strcmp(s, "cos_max") )
- opt_cos_max = simple_strtoul(val_str, NULL, 0);
-
s = ss + 1;
} while ( ss );
}
@@ -213,16 +177,7 @@ static inline void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( cat_socket_info )
- {
- unsigned int socket = cpu_to_socket(smp_processor_id());
-
- if ( test_bit(socket, cat_socket_enable) )
- psra->cos_mask = ((1ull << get_count_order(
- cat_socket_info[socket].cos_max)) - 1) << 32;
- }
-
- if ( psr_cmt_enabled() || psra->cos_mask )
+ if ( psr_cmt_enabled() )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -231,12 +186,6 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
-static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
- uint64_t cos_mask)
-{
- *reg = (*reg & ~cos_mask) | (((uint64_t)cos << 32) & cos_mask);
-}
-
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -245,459 +194,54 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
- if ( psra->cos_mask )
- psr_assoc_cos(®, d->arch.psr_cos_ids ?
- d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
- 0, psra->cos_mask);
-
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
psra->val = reg;
}
}
-static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
-{
- if ( !cat_socket_info )
- return ERR_PTR(-ENODEV);
-
- if ( socket >= nr_sockets )
- return ERR_PTR(-ENOTSOCK);
-
- if ( !test_bit(socket, cat_socket_enable) )
- return ERR_PTR(-ENOENT);
-
- return cat_socket_info + socket;
-}
-
-static inline bool_t cdp_is_enabled(unsigned int socket)
-{
- return cdp_socket_enable && test_bit(socket, cdp_socket_enable);
-}
int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
uint32_t *cos_max, uint32_t *flags)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- *cbm_len = info->cbm_len;
- *cos_max = info->cos_max;
-
- *flags = 0;
- if ( cdp_is_enabled(socket) )
- *flags |= XEN_SYSCTL_PSR_CAT_L3_CDP;
-
return 0;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
- bool_t cdp_enabled = cdp_is_enabled(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- if ( cdp_enabled )
- return -EXDEV;
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].code;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].data;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- }
-
- return 0;
-}
-
-static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
-{
- unsigned int first_bit, zero_bit;
-
- /* Set bits should only in the range of [0, cbm_len). */
- if ( cbm & (~0ull << cbm_len) )
- return 0;
-
- /* At least one bit need to be set. */
- if ( cbm == 0 )
- return 0;
-
- first_bit = find_first_bit(&cbm, cbm_len);
- zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
-
- /* Set bits should be contiguous. */
- if ( zero_bit < cbm_len &&
- find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
- return 0;
-
- return 1;
-}
-
-struct cos_cbm_info
-{
- unsigned int cos;
- bool_t cdp;
- uint64_t cbm_code;
- uint64_t cbm_data;
-};
-
-static void do_write_l3_cbm(void *data)
-{
- struct cos_cbm_info *info = data;
-
- if ( info->cdp )
- {
- wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(info->cos), info->cbm_code);
- wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(info->cos), info->cbm_data);
- }
- else
- wrmsrl(MSR_IA32_PSR_L3_MASK(info->cos), info->cbm_code);
-}
-
-static int write_l3_cbm(unsigned int socket, unsigned int cos,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp)
-{
- struct cos_cbm_info info =
- {
- .cos = cos,
- .cbm_code = cbm_code,
- .cbm_data = cbm_data,
- .cdp = cdp,
- };
-
- if ( socket == cpu_to_socket(smp_processor_id()) )
- do_write_l3_cbm(&info);
- else
- {
- unsigned int cpu = get_socket_cpu(socket);
-
- if ( cpu >= nr_cpu_ids )
- return -ENOTSOCK;
- on_selected_cpus(cpumask_of(cpu), do_write_l3_cbm, &info, 1);
- }
-
return 0;
}
-static int find_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
-{
- unsigned int cos;
-
- for ( cos = 0; cos <= cos_max; cos++ )
- {
- if ( (map[cos].ref || cos == 0) &&
- ((!cdp_enabled && map[cos].cbm == cbm_code) ||
- (cdp_enabled && map[cos].code == cbm_code &&
- map[cos].data == cbm_data)) )
- return cos;
- }
-
- return -ENOENT;
-}
-
-static int pick_avail_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- unsigned int old_cos)
-{
- unsigned int cos;
-
- /* If old cos is referred only by the domain, then use it. */
- if ( map[old_cos].ref == 1 && old_cos != 0 )
- return old_cos;
-
- /* Find an unused one other than cos0. */
- for ( cos = 1; cos <= cos_max; cos++ )
- if ( map[cos].ref == 0 )
- return cos;
-
- return -ENOENT;
-}
-
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type)
{
- unsigned int old_cos, cos_max;
- int cos, ret;
- uint64_t cbm_data, cbm_code;
- bool_t cdp_enabled = cdp_is_enabled(socket);
- struct psr_cat_cbm *map;
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- if ( !psr_check_cbm(info->cbm_len, cbm) )
- return -EINVAL;
-
- if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
- type == PSR_CBM_TYPE_L3_DATA) )
- return -ENXIO;
-
- cos_max = info->cos_max;
- old_cos = d->arch.psr_cos_ids[socket];
- map = info->cos_to_cbm;
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- cbm_code = cbm;
- cbm_data = cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- cbm_code = cbm;
- cbm_data = map[old_cos].data;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- cbm_code = map[old_cos].code;
- cbm_data = cbm;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- return -EINVAL;
- }
-
- spin_lock(&info->cbm_lock);
- cos = find_cos(map, cos_max, cbm_code, cbm_data, cdp_enabled);
- if ( cos >= 0 )
- {
- if ( cos == old_cos )
- {
- spin_unlock(&info->cbm_lock);
- return 0;
- }
- }
- else
- {
- cos = pick_avail_cos(map, cos_max, old_cos);
- if ( cos < 0 )
- {
- spin_unlock(&info->cbm_lock);
- return cos;
- }
-
- /* We try to avoid writing MSR. */
- if ( (cdp_enabled &&
- (map[cos].code != cbm_code || map[cos].data != cbm_data)) ||
- (!cdp_enabled && map[cos].cbm != cbm_code) )
- {
- ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
- if ( ret )
- {
- spin_unlock(&info->cbm_lock);
- return ret;
- }
- map[cos].code = cbm_code;
- map[cos].data = cbm_data;
- }
- }
-
- map[cos].ref++;
- map[old_cos].ref--;
- spin_unlock(&info->cbm_lock);
-
- d->arch.psr_cos_ids[socket] = cos;
-
return 0;
}
-/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
-static void psr_free_cos(struct domain *d)
-{
- unsigned int socket;
- unsigned int cos;
- struct psr_cat_socket_info *info;
-
- if( !d->arch.psr_cos_ids )
- return;
-
- for_each_set_bit(socket, cat_socket_enable, nr_sockets)
- {
- if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
- continue;
-
- info = cat_socket_info + socket;
- spin_lock(&info->cbm_lock);
- info->cos_to_cbm[cos].ref--;
- spin_unlock(&info->cbm_lock);
- }
-
- xfree(d->arch.psr_cos_ids);
- d->arch.psr_cos_ids = NULL;
-}
-
int psr_domain_init(struct domain *d)
{
- if ( cat_socket_info )
- {
- d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
- if ( !d->arch.psr_cos_ids )
- return -ENOMEM;
- }
-
return 0;
}
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
- psr_free_cos(d);
-}
-
-static int cat_cpu_prepare(unsigned int cpu)
-{
- if ( !cat_socket_info )
- return 0;
-
- if ( temp_cos_to_cbm == NULL &&
- (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
- opt_cos_max + 1UL)) == NULL )
- return -ENOMEM;
-
- return 0;
-}
-
-static void cat_cpu_init(void)
-{
- unsigned int eax, ebx, ecx, edx;
- struct psr_cat_socket_info *info;
- unsigned int socket;
- unsigned int cpu = smp_processor_id();
- uint64_t val;
- const struct cpuinfo_x86 *c = cpu_data + cpu;
-
- if ( !cpu_has(c, X86_FEATURE_PQE) || c->cpuid_level < PSR_CPUID_LEVEL_CAT )
- return;
-
- socket = cpu_to_socket(cpu);
- if ( test_bit(socket, cat_socket_enable) )
- return;
-
- cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
- if ( ebx & PSR_RESOURCE_TYPE_L3 )
- {
- cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
- info = cat_socket_info + socket;
- info->cbm_len = (eax & 0x1f) + 1;
- info->cos_max = min(opt_cos_max, edx & 0xffff);
-
- info->cos_to_cbm = temp_cos_to_cbm;
- temp_cos_to_cbm = NULL;
- /* cos=0 is reserved as default cbm(all ones). */
- info->cos_to_cbm[0].cbm = (1ull << info->cbm_len) - 1;
-
- spin_lock_init(&info->cbm_lock);
-
- set_bit(socket, cat_socket_enable);
-
- if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
- cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
- {
- info->cos_to_cbm[0].code = (1ull << info->cbm_len) - 1;
- info->cos_to_cbm[0].data = (1ull << info->cbm_len) - 1;
-
- /* We only write mask1 since mask0 is always all ones by default. */
- wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->cbm_len) - 1);
-
- rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
- wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
-
- /* Cut half of cos_max when CDP is enabled. */
- info->cos_max >>= 1;
-
- set_bit(socket, cdp_socket_enable);
- }
- printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
- socket, info->cos_max, info->cbm_len,
- cdp_is_enabled(socket) ? "on" : "off");
- }
-}
-
-static void cat_cpu_fini(unsigned int cpu)
-{
- unsigned int socket = cpu_to_socket(cpu);
-
- if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
- {
- struct psr_cat_socket_info *info = cat_socket_info + socket;
-
- if ( info->cos_to_cbm )
- {
- xfree(info->cos_to_cbm);
- info->cos_to_cbm = NULL;
- }
-
- if ( cdp_is_enabled(socket) )
- clear_bit(socket, cdp_socket_enable);
-
- clear_bit(socket, cat_socket_enable);
- }
-}
-
-static void __init psr_cat_free(void)
-{
- xfree(cat_socket_enable);
- cat_socket_enable = NULL;
- xfree(cat_socket_info);
- cat_socket_info = NULL;
-}
-
-static void __init init_psr_cat(void)
-{
- if ( opt_cos_max < 1 )
- {
- printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
- return;
- }
-
- cat_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
- cat_socket_info = xzalloc_array(struct psr_cat_socket_info, nr_sockets);
- cdp_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
-
- if ( !cat_socket_enable || !cat_socket_info )
- psr_cat_free();
}
static int psr_cpu_prepare(unsigned int cpu)
{
- return cat_cpu_prepare(cpu);
+ return 0;
}
static void psr_cpu_init(void)
{
- if ( cat_socket_info )
- cat_cpu_init();
-
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
- if ( cat_socket_info )
- cat_cpu_fini(cpu);
+ return;
}
static int cpu_callback(
@@ -738,14 +282,10 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- if ( opt_psr & PSR_CAT )
- init_psr_cat();
-
- if ( psr_cpu_prepare(0) )
- psr_cat_free();
+ psr_cpu_prepare(0);
psr_cpu_init();
- if ( psr_cmt_enabled() || cat_socket_info )
+ if ( psr_cmt_enabled() )
register_cpu_notifier(&cpu_nfb);
return 0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (2 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 03/23] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-30 13:05 ` Jan Beulich
2017-05-31 9:36 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
` (18 subsequent siblings)
22 siblings, 2 replies; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
To construct an extendible framework, we need analyze PSR features
and abstract the common things and feature specific things. Then,
encapsulate them into different data structures.
By analyzing PSR features, we can get below map.
+------+------+------+
--------->| Dom0 | Dom1 | ... |
| +------+------+------+
| |
|Dom ID | cos_id of domain
| V
| +-----------------------------------------------------------------------------+
User --------->| PSR |
Socket ID | +--------------+---------------+---------------+ |
| | Socket0 Info | Socket 1 Info | ... | |
| +--------------+---------------+---------------+ |
| | cos_id=0 cos_id=1 ... |
| | +-----------------------+-----------------------+-----------+ |
| |->Ref : | ref 0 | ref 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L3 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L2 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------+-----------+-----------+-----------+-----------+ |
| |->CDP : | cos0 code | cos0 data | cos1 code | cos1 data | ... | |
| +-----------+-----------+-----------+-----------+-----------+ |
+-----------------------------------------------------------------------------+
So, we need define a socket info data structure, 'struct
psr_socket_info' to manage information per socket. It contains a
reference count array according to COS ID and a feature array to
manage all features enabled. Every entry of the reference count
array is used to record how many domains are using the COS registers
according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
Dom1 uses COS_ID=1 registers of both features to save CBM values, like
below.
+-------+-------+-------+-----+
| COS 0 | COS 1 | COS 2 | ... |
+-------+-------+-------+-----+
L3 CAT | 0x7ff | 0x1ff | ... | ... |
+-------+-------+-------+-----+
L2 CAT | 0xff | 0xff | ... | ... |
+-------+-------+-------+-----+
If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
That means, both Dom1 and Dom2 use same COS registers(ID=1) to keep same
L3/L2 values. So, the value of ref[1] is 2 which means 2 domains are using
COS_ID 1.
To manage a feature, we need define a feature node data structure,
'struct feat_node', to manage feature's specific HW info, and an array of all
COS registers values of this feature.
To manage feature properties, we need define a feature property data structure,
'struct feat_props', to manage common properties (callback functions - all
feature's specific behaviors are encapsulated into these callback functions,
and generic values - e.g. the cos_max), the feature independent values.
CDP is a special feature which uses two entries of the array
for one COS ID. So, the number of CDP COS registers is the half of L3
CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
it is enabled. CDP uses the COS registers array as below.
+-----------+-----------+-----------+-----------+-----------+
CDP cos_reg_val[] index: | 0 | 1 | 2 | 3 | ... |
+-----------+-----------+-----------+-----------+-----------+
value: | cos0 code | cos0 data | cos1 code | cos1 data | ... |
+-----------+-----------+-----------+-----------+-----------+
For more details, please refer SDM and patches to implement 'get value' and
'set value'.
This patch also implements the CPU init and free flow including L3 CAT
initialization and some resources free. It includes below flows:
1. presmp init:
- parse command line parameter.
- allocate socket info for every socket.
- allocate feature resource.
- initialize socket info, get feature info and add feature into feature
array per cpuid result.
- free resources allocated if error happens.
- register cpu notifier to handle cpu events.
2. cpu notifier:
- handle cpu online events, if initialization work has been done before,
do nothing.
- handle cpu offline events, if it is the last cpu offline, free some
socket resources.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- handle 'feat_init'.
- merge main data structures implementation into CPU init/free patch,
including commit messages and change history.
(suggested by Jan Beulich)
- remove MSR restore action which is unnecessary.
(suggested by Jan Beulich)
- move 'type[]' declaration into this patch.
(suggested by Jan Beulich)
- modify comment.
(suggested by Jan Beulich)
v10:
- remove initialization for 'PSR_SOCKET_L3_CAT'.
(suggested by Jan Beulich)
- rename 'feat_ops' to 'feat_props'.
(suggested by Jan Beulich)
- move 'cbm_len' to 'feat_props' because it is feature independent so far.
(suggested by Jan Beulich)
- move 'cos_max' to 'feat_props' because it is feature independent.
(suggested by Jan Beulich)
- move 'cos_num' to 'feat_props' because it is feature independent.
(suggested by Jan Beulich)
- remove union 'info' and struct 'psr_cat_hw_info'.
- remove 'get_cos_max' from 'feat_props'.
(suggested by Jan Beulich)
- remove 'feat_mask' from 'psr_socket_info' because we can use 'features[]'
to check if any feature is initialized.
(suggested by Jan Beulich)
- move 'ref_lock' above 'cos_ref'.
(suggested by Jan Beulich)
- adjust comments and commit message according to above changes.
- remove 'asm/x86_emulate.h' inclusion as it has been indirectly included.
(suggested by Jan Beulich)
- remove 'CAT_COS_NUM' as it is only used once.
(suggested by Jan Beulich)
- remove 'feat_mask'.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- remove 'get_cos_max' hook declaration.
(suggested by Jan Beulich)
- modify 'cat_default_val' implementation.
(suggested by Jan Beulich)
- modify 'psr_alloc_feat_enabled' implementation to make it simple.
(suggested by Jan Beulich)
- rename 'free_feature' to 'free_socket_resources' because it is executed
when socket is offline. It needs free resources related to the socket.
(suggested by Jan Beulich)
- define 'feat_init_done' to iterate feature array to check if any feature
has been initialized.
(suggested by Jan Beulich)
- input 'struct cpuid_leaf' pointer into 'cat_init_feature' to avoid memory
copy.
(suggested by Jan Beulich)
- modify 'cat_init_feature' to use switch and things related to above
changes.
(suggested by Jan Beulich)
- add an indentation for label.
(suggested by Jan Beulich)
v9:
- replace feature list to a feature pointer array.
(suggested by Roger Pau)
- add 'PSR_SOCKET_MAX_FEAT' in 'enum psr_feat_type' to know features
account.
(suggested by Roger Pau)
- move 'feat_ops' declaration into 'feat_node' structure.
(suggested by Roger Pau)
- directly use uninon for feature HW info and move its declaration into
'feat_node' structure.
(suggested by Roger Pau)
- remove 'enum psr_feat_type feature' declared in 'feat_ops' because it is
not useful after using feature pointer array.
(suggested by Roger Pau)
- rename 'l3_cat_info' to 'cat_info' to be used by all CAT/CDP features.
- remove 'nr_feat' which is only for a record.
(suggested by Jan Beulich)
- add 'cos_num' to record how many COS registers are used by a feature in
one time access.
(suggested by Jan Beulich)
- replace 'uint64_t' to 'uint32_t' for cbm value because SDM specifies the
max 32 bits for it.
(suggested by Jan Beulich)
- add commit message to explain the flows.
- handle cpu offline and online again case to read MSRs registers values
back and save them into cos array to make user can get real data.
- create a new patch about moving 'cpuid_count_leaf'.
(suggested by Wei Liu)
- modify comment to explain why not free some resource in 'free_feature'.
(suggested by Wei Liu)
- implement 'psr_alloc_feat_enabled' to check if allocation feature is
enabled in cmdline and some initialization work done.
(suggested by Wei Liu)
- implement 'cat_default_val' to set default value for CAT features.
(suggested by Wei Liu)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- implement a common 'cat_init_feature' to replace L3 CAT/L2 CAT specific
init functions.
(suggested by Roger Pau)
- modify comments for global feature node.
(suggested by Jan Beulich)
- remove unnecessary comments.
(suggested by Jan Beulich)
- remove unnecessary 'else'.
(suggested by Jan Beulich)
- remove 'nr_feat'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- check global flag with boot cpu operations.
(suggested by Jan Beulich)
- remove 'cpu_init_work' and move codes into 'psr_cpu_init'.
(suggested by Jan Beulich)
- remove 'cpu_fini_work' and move codes into 'psr_cpu_fini'.
(suggested by Jan Beulich)
- assign value for 'cos_num'.
(suggested by Jan Beulich)
- change about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- fix format issue.
(suggested by Konrad Rzeszutek Wilk)
- add comments to explain why we care about cpumask_empty when the last
cpu on socket is offline.
(suggested by Konrad Rzeszutek Wilk)
v7:
- sort inclusion files position.
(suggested by Wei Liu)
- initialize structure objects for avoiding surprise.
(suggested by Konrad Rzeszutek Wilk)
- fix typo.
(suggested by Konrad Rzeszutek Wilk)
- fix a logical mistake when handling the last cpu offline event.
(suggested by Konrad Rzeszutek Wilk)
v6:
- make commit message be clearer.
(suggested by Konrad Rzeszutek Wilk)
- fix wordings.
(suggested by Konrad Rzeszutek Wilk)
- add comments to explain relationship between 'feat_mask' and
'enum psr_feat_type'.
(suggested by Konrad Rzeszutek Wilk)
- use 'struct cpuid_leaf' introduced in Andrew's patch.
(suggested by Konrad Rzeszutek Wilk)
- add comments about cpu_add_remove_lock.
(suggested by Konrad Rzeszutek Wilk)
- change 'clear_bit' to '__clear_bit'.
(suggested by Konrad Rzeszutek Wilk)
- add 'ASSERT' check when setting 'feat_mask'.
(suggested by Konrad Rzeszutek Wilk)
- adjust 'printk' position to avoid odd spacing.
(suggested by Konrad Rzeszutek Wilk)
- add comment to explain usage of 'feat_l3_cat'.
(suggested by Konrad Rzeszutek Wilk)
- fix wording.
(suggested by Konrad Rzeszutek Wilk)
- move 'cpuid_count_leaf' helper function to 'asm-x86/processor.h'.
It cannot be moved to 'cpuid.h' which causes compilation error because
of header file loop reference.
(suggested by Andrew Cooper)
v5:
- remove section number.
(suggested by Jan Beulich)
- remove double blank.
(suggested by Jan Beulich)
- add comment to explain the reason to define 'feat_l3_cat'.
(suggested by Jan Beulich)
- use 'list_for_each_entry_safe'.
(suggested by Jan Beulich)
- remove codes to free 'feat_l3_cat' in 'free_feature' to avoid the need
for an allocation the next time a CPU comes online.
(suggested by Jan Beulich)
- define 'struct cpuid_leaf_regs' to encapsulate eax~edx.
(suggested by Jan Beulich)
- print feature info on a socket only when 'opt_cpu_info' is true.
(suggested by Jan Beulich)
- declare global variable 'l3_cat_ops' to 'static const'.
(suggested by Jan Beulich)
- use 'current_cpu_data'.
(suggested by Jan Beulich)
- rename 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- clear PQE feature bit when the maximum CPUID level is too low.
(suggested by Jan Beulich)
- directly call 'l3_cat_init_feature'. No need to make it a callback
function.
(suggested by Jan Beulich)
- remove local variable 'info'.
(suggested by Jan Beulich)
- move 'INIT_LIST_HEAD' into 'cpu_init_work' to be together with
spin_lock_init().
(suggested by Jan Beulich)
- remove 'cpu_prepare_work' and move its content into 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v4:
- create this patch because of removing all old CAT/CDP codes to make
implementation be more easily understood.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 277 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 271 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 96a8589..b73856e 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -13,16 +13,111 @@
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
* more details.
*/
-#include <xen/init.h>
#include <xen/cpu.h>
#include <xen/err.h>
+#include <xen/init.h>
#include <xen/sched.h>
#include <asm/psr.h>
+/*
+ * Terminology:
+ * - CAT Cache Allocation Technology
+ * - CBM Capacity BitMasks
+ * - CDP Code and Data Prioritization
+ * - CMT Cache Monitoring Technology
+ * - COS/CLOS Class of Service. Also mean COS registers.
+ * - COS_MAX Max number of COS for the feature (minus 1)
+ * - MSRs Machine Specific Registers
+ * - PSR Intel Platform Shared Resource
+ */
+
#define PSR_CMT (1<<0)
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
+#define CAT_CBM_LEN_MASK 0x1f
+#define CAT_COS_MAX_MASK 0xffff
+
+/*
+ * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
+ * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
+ * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
+ *
+ * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for
+ * up to 64 L2 CAT COS. The COS_ID=[0,63].
+ *
+ * So, the maximum COS register count of one feature is 128.
+ */
+#define MAX_COS_REG_CNT 128
+
+/*
+ * Every PSR feature uses some COS registers for each COS ID, e.g. CDP uses 2
+ * COS registers (DATA and CODE) for one COS ID, but CAT uses 1 COS register.
+ * We use below macro as the max number of COS registers used by all features.
+ * So far, it is 2 which means CDP's COS registers number.
+ */
+#define PSR_MAX_COS_NUM 2
+
+enum psr_feat_type {
+ PSR_SOCKET_L3_CAT,
+ PSR_SOCKET_FEAT_NUM,
+};
+
+/*
+ * This structure represents one feature.
+ * cos_max - The max COS registers number got through CPUID.
+ * cbm_len - The length of CBM got through CPUID.
+ * cos_reg_val - Array to store the values of COS registers. One entry stores
+ * the value of one COS register.
+ * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
+ * For CDP, two entries correspond to one COS_ID. E.g.
+ * COS_ID=0 corresponds to cos_reg_val[0] (Data) and
+ * cos_reg_val[1] (Code).
+ */
+struct feat_node {
+ /* cos_max and cbm_len are common values for all features so far. */
+ unsigned int cos_max;
+ unsigned int cbm_len;
+ uint32_t cos_reg_val[MAX_COS_REG_CNT];
+};
+
+/*
+ * This structure defines feature specific values, e.g. cos_num.
+ *
+ * Array 'feat_props' is defined to save every feature's properties. We use
+ * 'enum psr_feat_type' as index.
+ */
+static const struct feat_props {
+ /*
+ * cos_num - COS registers number that feature uses for one COS ID.
+ * It is defined in SDM.
+ */
+ unsigned int cos_num;
+
+ /*
+ * An array to save all 'enum cbm_type' values of the feature. It is
+ * used with cos_num together to get/write a feature's COS registers
+ * values one by one.
+ */
+ enum cbm_type type[PSR_MAX_COS_NUM];
+} *feat_props[PSR_SOCKET_FEAT_NUM];
+
+/*
+ * PSR features are managed per socket. Below structure defines the members
+ * used to manage these features.
+ * ref_lock - A lock to protect cos_ref.
+ * features - A feature node array used to manage all features enabled.
+ * cos_ref - A reference count array to record how many domains are using the
+ * COS ID. Every entry of cos_ref corresponds to one COS ID.
+ */
+struct psr_socket_info {
+ bool feat_init;
+ spinlock_t ref_lock;
+ /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
+ struct feat_node *features[PSR_SOCKET_FEAT_NUM];
+ unsigned int cos_ref[MAX_COS_REG_CNT];
+};
+
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -30,11 +125,95 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
+static struct psr_socket_info *__read_mostly socket_info;
+
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
+static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
+/*
+ * Declare global feature node for every feature to facilitate the feature
+ * array creation. It is used to transiently store a spare node.
+ */
+static struct feat_node *feat_l3_cat;
+
+/* Common functions */
+#define cat_default_val(len) (0xffffffff >> (32 - (len)))
+
+/*
+ * Use this function to check if any allocation feature has been enabled
+ * in cmdline.
+ */
+static bool psr_alloc_feat_enabled(void)
+{
+ return !!socket_info;
+}
+
+static void free_socket_resources(unsigned int socket)
+{
+ unsigned int i;
+ struct psr_socket_info *info = socket_info + socket;
+
+ if ( !info )
+ return;
+
+ /*
+ * Free resources of features. The global feature object, e.g. feat_l3_cat,
+ * may not be freed here if it is not added into array. It is simply being
+ * kept until the next CPU online attempt.
+ */
+ for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
+ {
+ xfree(info->features[i]);
+ info->features[i] = NULL;
+ }
+
+ info->feat_init = false;
+}
+
+/* CAT common functions implementation. */
+static void cat_init_feature(const struct cpuid_leaf *regs,
+ struct feat_node *feat,
+ struct psr_socket_info *info,
+ enum psr_feat_type type)
+{
+ /* No valid value so do not enable feature. */
+ if ( !regs->a || !regs->d )
+ return;
+
+ feat->cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
+ feat->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
+
+ switch ( type )
+ {
+ case PSR_SOCKET_L3_CAT:
+ /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+ feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
+
+ break;
+
+ default:
+ return;
+ }
+
+ /* Add this feature into array. */
+ info->features[type] = feat;
+
+ if ( !opt_cpu_info )
+ return;
+
+ printk(XENLOG_INFO "%s CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ ((type == PSR_SOCKET_L3_CAT) ? "L3" : "L2"),
+ cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
+}
+
+/* L3 CAT props */
+static struct feat_props l3_cat_props = {
+ .cos_num = 1,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -74,6 +253,9 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
+ if ( val_str && !strcmp(s, "cos_max") )
+ opt_cos_max = simple_strtoul(val_str, NULL, 0);
+
s = ss + 1;
} while ( ss );
}
@@ -229,19 +411,98 @@ void psr_domain_free(struct domain *d)
psr_free_rmid(d);
}
-static int psr_cpu_prepare(unsigned int cpu)
+static void __init init_psr(void)
+{
+ if ( opt_cos_max < 1 )
+ {
+ printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
+ return;
+ }
+
+ socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
+
+ if ( !socket_info )
+ {
+ printk(XENLOG_INFO "Failed to alloc socket_info!\n");
+ return;
+ }
+}
+
+static void __init psr_free(void)
{
+ xfree(socket_info);
+ socket_info = NULL;
+}
+
+static int psr_cpu_prepare(void)
+{
+ if ( !psr_alloc_feat_enabled() )
+ return 0;
+
+ /* Malloc memory for the global feature node here. */
+ if ( feat_l3_cat == NULL &&
+ (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
static void psr_cpu_init(void)
{
+ struct psr_socket_info *info;
+ unsigned int socket;
+ unsigned int cpu = smp_processor_id();
+ struct feat_node *feat;
+ struct cpuid_leaf regs;
+
+ if ( !psr_alloc_feat_enabled() || !boot_cpu_has(X86_FEATURE_PQE) )
+ goto assoc_init;
+
+ if ( boot_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
+ {
+ setup_clear_cpu_cap(X86_FEATURE_PQE);
+ goto assoc_init;
+ }
+
+ socket = cpu_to_socket(cpu);
+ info = socket_info + socket;
+ if ( info->feat_init )
+ goto assoc_init;
+
+ spin_lock_init(&info->ref_lock);
+
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L3 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
+
+ feat = feat_l3_cat;
+ feat_l3_cat = NULL;
+ l3_cat_props.type[0] = PSR_CBM_TYPE_L3;
+ feat_props[PSR_SOCKET_L3_CAT] = &l3_cat_props;
+
+ cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CAT);
+
+ info->feat_init = true;
+ }
+
+ assoc_init:
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
- return;
+ unsigned int socket = cpu_to_socket(cpu);
+
+ if ( !psr_alloc_feat_enabled() )
+ return;
+
+ /*
+ * We only free when we are the last CPU in the socket. The socket_cpumask
+ * is cleared prior to this notification code by remove_siblinginfo().
+ */
+ if ( socket_cpumask[socket] && cpumask_empty(socket_cpumask[socket]) )
+ free_socket_resources(socket);
}
static int cpu_callback(
@@ -253,7 +514,7 @@ static int cpu_callback(
switch ( action )
{
case CPU_UP_PREPARE:
- rc = psr_cpu_prepare(cpu);
+ rc = psr_cpu_prepare();
break;
case CPU_STARTING:
psr_cpu_init();
@@ -282,10 +543,14 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- psr_cpu_prepare(0);
+ if ( opt_psr & PSR_CAT )
+ init_psr();
+
+ if ( psr_cpu_prepare() )
+ psr_free();
psr_cpu_init();
- if ( psr_cmt_enabled() )
+ if ( psr_cmt_enabled() || psr_alloc_feat_enabled() )
register_cpu_notifier(&cpu_nfb);
return 0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (3 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-30 13:26 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 06/23] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
` (17 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the Domain init/free and schedule flows.
- When domain init, its psr resource should be allocated.
- When domain free, its psr resource should be freed too.
- When domain is scheduled, its COS ID on the socket should be
set into ASSOC register to make corresponding COS MSR value
work.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- replace 'feat_init_done()' to 'feat_init' flag.
(suggested by Jan Beulich)
- adjust parameters positions when calling 'psr_assoc_cos'.
(suggested by Jan Beulich)
- add comment to explain why to check 'psr_cos_ids'.
v10:
- remove 'cat_get_cos_max' as 'cos_max' is a feature property now which
can be directly used.
(suggested by Jan Beulich)
- replace 'info->feat_mask' check to 'feat_init_done'.
(suggested by Jan Beulich)
v9:
- rename 'l3_cat_get_cos_max' to 'cat_get_cos_max' to cover all CAT/CDP
features.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- implement 'psr_alloc_cos' to match 'psr_free_cos'.
(suggested by Wei Liu)
- use 'psr_alloc_feat_enabled'.
(suggested by Wei Liu)
- fix coding style issue.
(suggested by Wei Liu)
- remove 'inline'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- remove 'psr_cos_ids' check in 'psr_free_cos'.
(suggested by Jan Beulich)
v6:
- change 'PSR_ASSOC_REG_POS' to 'PSR_ASSOC_REG_SHIFT'.
(suggested by Konrad Rzeszutek Wilk)
v5:
- rename 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- define 'PSR_ASSOC_REG_POS'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 70 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index b73856e..bda325d 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -50,6 +50,8 @@
*/
#define MAX_COS_REG_CNT 128
+#define PSR_ASSOC_REG_SHIFT 32
+
/*
* Every PSR feature uses some COS registers for each COS ID, e.g. CDP uses 2
* COS registers (DATA and CODE) for one COS ID, but CAT uses 1 COS register.
@@ -355,11 +357,38 @@ void psr_free_rmid(struct domain *d)
d->arch.psr_rmid = 0;
}
-static inline void psr_assoc_init(void)
+static unsigned int get_max_cos_max(const struct psr_socket_info *info)
+{
+ unsigned int cos_max = 0, i;
+
+ for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
+ {
+ const struct feat_node *feat = info->features[i];
+ if ( !feat )
+ continue;
+
+ cos_max = max(feat->cos_max, cos_max);
+ }
+
+ return cos_max;
+}
+
+static void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( psr_cmt_enabled() )
+ if ( psr_alloc_feat_enabled() )
+ {
+ unsigned int socket = cpu_to_socket(smp_processor_id());
+ const struct psr_socket_info *info = socket_info + socket;
+ unsigned int cos_max = get_max_cos_max(info);
+
+ if ( info->feat_init )
+ psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
+ PSR_ASSOC_REG_SHIFT;
+ }
+
+ if ( psr_cmt_enabled() || psra->cos_mask )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -368,6 +397,13 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
+static void psr_assoc_cos(uint64_t *reg, unsigned int cos,
+ uint64_t cos_mask)
+{
+ *reg = (*reg & ~cos_mask) |
+ (((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
+}
+
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -376,6 +412,14 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
+ /* IDLE domain's 'psr_cos_ids' is NULL so we set default value for it. */
+ if ( psra->cos_mask )
+ psr_assoc_cos(®,
+ d->arch.psr_cos_ids ?
+ d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
+ 0,
+ psra->cos_mask);
+
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
@@ -401,14 +445,37 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
return 0;
}
-int psr_domain_init(struct domain *d)
+/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
+static void psr_free_cos(struct domain *d)
+{
+ xfree(d->arch.psr_cos_ids);
+ d->arch.psr_cos_ids = NULL;
+}
+
+static int psr_alloc_cos(struct domain *d)
{
+ d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
+ if ( !d->arch.psr_cos_ids )
+ return -ENOMEM;
+
return 0;
}
+int psr_domain_init(struct domain *d)
+{
+ /* Init to success value */
+ int ret = 0;
+
+ if ( psr_alloc_feat_enabled() )
+ ret = psr_alloc_cos(d);
+
+ return ret;
+}
+
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
+ psr_free_cos(d);
}
static void __init init_psr(void)
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 06/23] x86: refactor psr: L3 CAT: implement get hw info flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (4 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-30 13:51 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
` (16 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements get HW info flow including L3 CAT callback
function.
It also changes sysctl interface to make it more general.
With this patch, 'psr-hwinfo' can work for L3 CAT.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- changes about 'cos_max' and 'cbm_len'.
(suggested by Jan Beulich)
- change about 'feat_props'.
(suggested by Jan Beulich)
- declare 'PSR_SOCKET_FEAT_UNKNOWN' to handle error case.
(suggested by Jan Beulich)
- use 'ASSERT' to check data.
(suggested by Jan Beulich)
- fix comment of 'get_feat_info'.
(suggested by Jan Beulich)
- fix a bug when checking 'feat_type'.
(suggested by Jan Beulich)
v10:
- remove 'PSR_SOCKET_UNKNOWN' and use 'ASSERT_UNREACHABLE()' to handle
this case.
(suggested by Jan Beulich)
- check 'feat_type'.
(suggested by Jan Beulich)
- adjust macros names and values to make them more appropriate.
(suggested by Jan Beulich)
- use 'feat_init_done'.
(suggested by Jan Beulich)
- changes about 'cbm_len'.
(suggested by Jan Beulich)
v9:
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- define 'PSR_INFO_SIZE'.
(suggested by Roger Pau)
- fix coding style issue.
(suggested by Roger Pau and Jan Beulich)
- use 'ARRAY_SIZE'.
(suggested by Roger Pau)
- rename 'l3_cat_get_feat_info' to 'cat_get_feat_info' to make it a common
function for both L3/L2 CAT.
(suggested by Roger Pau)
- move constant to the right of comparison.
(suggested by Wei Liu)
- remove wrong comment.
(suggested by Jan Beulich)
- rename macros used by psr_get_info to make them meaningful.
(suggested by Jan Beulich)
- remove assignment for 'PSR_SOCKET_UNKNOWN'.
(suggested by Jan Beulich)
- retain blank line after 'case XEN_SYSCTL_PSR_CAT_get_l3_info'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- move common data check into common function.
(suggested by Jan Beulich)
v6:
- fix coding style issue.
(suggested by Konrad Rzeszutek Wilk)
- define 'PSR_SOCKET_UNKNOWN' in 'psr_feat_type'.
(suggested by Konrad Rzeszutek Wilk)
- change '-ENOTSOCK' to 'ERANGE'.
(suggested by Konrad Rzeszutek Wilk)
- modify position of macros to remove odd spacing in psr.h.
(suggested by Konrad Rzeszutek Wilk)
v5:
- change 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
- modify parameter type to avoid fixed width type when there is no such
intention.
(suggested by Jan Beulich)
- use 'const' when it is possible.
(suggested by Jan Beulich)
- check feature type outside callback function.
(suggested by Jan Beulich)
- modify macros names to add prefix 'PSR_' and change 'CDP_FLAG' to
'PSR_FLAG'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 75 +++++++++++++++++++++++++++++++++++++++++++++--
xen/arch/x86/sysctl.c | 19 +++++++++---
xen/include/asm-x86/psr.h | 16 ++++++----
3 files changed, 98 insertions(+), 12 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index bda325d..2e6595d 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -63,6 +63,7 @@
enum psr_feat_type {
PSR_SOCKET_L3_CAT,
PSR_SOCKET_FEAT_NUM,
+ PSR_SOCKET_FEAT_UNKNOWN,
};
/*
@@ -102,6 +103,10 @@ static const struct feat_props {
* values one by one.
*/
enum cbm_type type[PSR_MAX_COS_NUM];
+
+ /* get_feat_info is used to return feature HW info through sysctl. */
+ bool (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len);
} *feat_props[PSR_SOCKET_FEAT_NUM];
/*
@@ -175,6 +180,22 @@ static void free_socket_resources(unsigned int socket)
info->feat_init = false;
}
+static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
+{
+ enum psr_feat_type feat_type = PSR_SOCKET_FEAT_UNKNOWN;
+
+ switch ( type )
+ {
+ case PSR_CBM_TYPE_L3:
+ feat_type = PSR_SOCKET_L3_CAT;
+ break;
+ default:
+ ASSERT_UNREACHABLE();
+ }
+
+ return feat_type;
+}
+
/* CAT common functions implementation. */
static void cat_init_feature(const struct cpuid_leaf *regs,
struct feat_node *feat,
@@ -211,9 +232,23 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
}
+static bool cat_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len)
+{
+ if ( array_len != PSR_INFO_ARRAY_SIZE )
+ return false;
+
+ data[PSR_INFO_IDX_COS_MAX] = feat->cos_max;
+ data[PSR_INFO_IDX_CAT_CBM_LEN] = feat->cbm_len;
+ data[PSR_INFO_IDX_CAT_FLAG] = 0;
+
+ return true;
+}
+
/* L3 CAT props */
static struct feat_props l3_cat_props = {
.cos_num = 1,
+ .get_feat_info = cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -427,10 +462,44 @@ void psr_ctxt_switch_to(struct domain *d)
}
}
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags)
+static struct psr_socket_info *get_socket_info(unsigned int socket)
{
- return 0;
+ if ( !socket_info )
+ return ERR_PTR(-ENODEV);
+
+ if ( socket >= nr_sockets )
+ return ERR_PTR(-ERANGE);
+
+ if ( !socket_info[socket].feat_init )
+ return ERR_PTR(-ENOENT);
+
+ return socket_info + socket;
+}
+
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len)
+{
+ const struct psr_socket_info *info = get_socket_info(socket);
+ const struct feat_node *feat;
+ enum psr_feat_type feat_type;
+
+ ASSERT(data);
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( feat_type >= ARRAY_SIZE(info->features) )
+ return -ENOENT;
+
+ feat = info->features[feat_type];
+ if ( !feat || !feat_props[feat_type] )
+ return -ENOENT;
+
+ if ( feat_props[feat_type]->get_feat_info(feat, data, array_len) )
+ return 0;
+
+ return -EINVAL;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 2f7056e..c23270d 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -175,14 +175,25 @@ long arch_do_sysctl(
switch ( sysctl->u.psr_cat_op.cmd )
{
case XEN_SYSCTL_PSR_CAT_get_l3_info:
- ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
- &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
- &sysctl->u.psr_cat_op.u.l3_info.cos_max,
- &sysctl->u.psr_cat_op.u.l3_info.flags);
+ {
+ uint32_t data[PSR_INFO_ARRAY_SIZE];
+
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
+ if ( ret )
+ break;
+
+ sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ data[PSR_INFO_IDX_COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ data[PSR_INFO_IDX_CAT_CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.flags =
+ data[PSR_INFO_IDX_CAT_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
ret = -EFAULT;
break;
+ }
default:
ret = -EOPNOTSUPP;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 57f47e9..af3a465 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -19,20 +19,26 @@
#include <xen/types.h>
/* CAT cpuid level */
-#define PSR_CPUID_LEVEL_CAT 0x10
+#define PSR_CPUID_LEVEL_CAT 0x10
/* Resource Type Enumeration */
#define PSR_RESOURCE_TYPE_L3 0x2
/* L3 Monitoring Features */
-#define PSR_CMT_L3_OCCUPANCY 0x1
+#define PSR_CMT_L3_OCCUPANCY 0x1
/* CDP Capability */
-#define PSR_CAT_CDP_CAPABILITY (1u << 2)
+#define PSR_CAT_CDP_CAPABILITY (1u << 2)
/* L3 CDP Enable bit*/
#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
+/* Used by psr_get_info() */
+#define PSR_INFO_IDX_COS_MAX 0
+#define PSR_INFO_IDX_CAT_CBM_LEN 1
+#define PSR_INFO_IDX_CAT_FLAG 2
+#define PSR_INFO_ARRAY_SIZE 3
+
struct psr_cmt_l3 {
unsigned int features;
unsigned int upscaling_factor;
@@ -63,8 +69,8 @@ int psr_alloc_rmid(struct domain *d);
void psr_free_rmid(struct domain *d);
void psr_ctxt_switch_to(struct domain *d);
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags);
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len);
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (5 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 06/23] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-30 14:05 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
` (15 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
There is an interface in user space to show feature value of
domains.
This patch implements get value flow in hypervisor.
It also changes domctl interface to make it more general.
With this patch, 'psr-cat-show' can work for L3 CAT but not for
L3 code/data which is implemented in CDP related patches.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- declare a 'switch()' wide variable 'val32' in domctl.
(suggested by Jan Beulich)
- remove 'get_val' callback function which is replaced by generic codes.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- do not 'ASSERT' domain pointer.
(suggested by Jan Beulich)
- modify commit message.
v10:
- use an intermediate variable to get value and avoid cast in domctl.
(suggested by Jan Beulich)
- remove 'type' in 'get_val' parameters and will add it back when
implementing CDP.
(suggested by Jan Beulich)
- remove unnecessary variable and return error about 'info' in
'psr_get_feat'.
(suggested by Jan Beulich)
- use 'ASSERT' to check input parameter in 'psr_get_val'.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
v9:
- add commit message to explain there is an user space interface.
- rename 'l3_cat_get_val' to 'cat_get_val' to cover all L3/L2 CAT features.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- change parameter of 'psr_get'. Use 'psr_cos_ids' directly to replace
domain. Also declare it to 'const'.
(suggested by Jan Beulich)
- change code flow to remove 'psr_get' but add 'psr_get_feat' to make codes
more reasonable.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- move cos check into common function because this check is required by all
features.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v7:
- rename '__psr_get' to 'psr_get'.
(suggested by Wei Liu)
v6:
- modify commit message to make it clearer.
(suggested by Konrad Rzeszutek Wilk)
- remove one extra space in code.
(suggested by Konrad Rzeszutek Wilk)
- remove unnecessary comment.
(suggested by Konrad Rzeszutek Wilk)
- write a helper function to move get info and get val functions into
it. Because most codes of 'get_info' and 'get_val' are same.
(suggested by Konrad Rzeszutek Wilk)
v5:
- rename 'dat[]' to 'data[]'
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- check if feature type match in caller of feature callback function.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/domctl.c | 20 +++++++++-------
xen/arch/x86/psr.c | 60 +++++++++++++++++++++++++++++++++++++++--------
xen/include/asm-x86/psr.h | 4 ++--
3 files changed, 63 insertions(+), 21 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 1220224..5b62c5c 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1408,6 +1408,8 @@ long arch_do_domctl(
case XEN_DOMCTL_psr_cat_op:
switch ( domctl->u.psr_cat_op.cmd )
{
+ uint32_t val32;
+
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
domctl->u.psr_cat_op.data,
@@ -1427,23 +1429,23 @@ long arch_do_domctl(
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L3);
+ domctl->u.psr_cat_op.data = val32;
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L3_CODE);
+ domctl->u.psr_cat_op.data = val32;
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L3_DATA);
+ domctl->u.psr_cat_op.data = val32;
copyback = 1;
break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 2e6595d..1b781e1 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -476,23 +476,34 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
return socket_info + socket;
}
+static struct feat_node *psr_get_feat_and_type(unsigned int socket,
+ enum cbm_type type,
+ enum psr_feat_type *feat_type)
+{
+ const struct psr_socket_info *info = get_socket_info(socket);
+
+ if ( IS_ERR(info) )
+ return ERR_PTR(PTR_ERR(info));
+
+ *feat_type = psr_cbm_type_to_feat_type(type);
+ if ( *feat_type >= ARRAY_SIZE(info->features) )
+ return NULL;
+
+ return info->features[*feat_type];
+}
+
int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len)
{
- const struct psr_socket_info *info = get_socket_info(socket);
const struct feat_node *feat;
enum psr_feat_type feat_type;
ASSERT(data);
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- feat_type = psr_cbm_type_to_feat_type(type);
- if ( feat_type >= ARRAY_SIZE(info->features) )
- return -ENOENT;
+ feat = psr_get_feat_and_type(socket, type, &feat_type);
+ if ( IS_ERR(feat) )
+ return PTR_ERR(feat);
- feat = info->features[feat_type];
if ( !feat || !feat_props[feat_type] )
return -ENOENT;
@@ -502,9 +513,38 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
return -EINVAL;
}
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type)
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint32_t *val, enum cbm_type type)
{
+ const struct feat_node *feat;
+ enum psr_feat_type feat_type;
+ unsigned int cos, i;
+
+ ASSERT(val);
+
+ feat = psr_get_feat_and_type(socket, type, &feat_type);
+ if ( IS_ERR(feat) )
+ return PTR_ERR(feat);
+
+ if ( !feat || !feat_props[feat_type] )
+ return -ENOENT;
+
+ cos = d->arch.psr_cos_ids[socket];
+ /*
+ * If input cos exceeds current feature's cos_max, we should return its
+ * default value which is stored in cos 0. This case only happens
+ * when more than two features enabled concurrently and at least one
+ * features's cos_max is bigger than others. When a domain's working cos
+ * id is bigger than some features' cos_max, HW automatically works as
+ * default value for those features which cos_max is smaller.
+ */
+ if ( cos > feat->cos_max )
+ cos = 0;
+
+ for ( i = 0; i < feat_props[feat_type]->cos_num; i++ )
+ if ( type == feat_props[feat_type]->type[i] )
+ *val = feat->cos_reg_val[cos * feat_props[feat_type]->cos_num + i];
+
return 0;
}
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index af3a465..7c6d38a 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -71,8 +71,8 @@ void psr_ctxt_switch_to(struct domain *d);
int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type);
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint32_t *val, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (6 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-30 14:32 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
` (14 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
As set value flow is the most complicated one in psr, it will be
divided to some patches to make things clearer. This patch
implements the set value framework to show a whole picture firstly.
It also changes domctl interface to make it more general.
To make the set value flow be general and can support multiple features
at same time, it includes below steps:
1. Get COS ID that current domain is using.
2. Gather a value array to store all features current value
into it and replace the current value of the feature which is
being set to the new input value.
3. Find if there is already a COS ID on which all features'
values are same as the array. Then, we can reuse this COS
ID.
4. If fail to find, we need pick an available COS ID. Only COS ID which ref
is 0 or 1 can be picked.
5. Write the feature's MSRs according to the COS ID.
6. Update ref according to COS ID.
7. Save the COS ID into current domain's psr_cos_ids[socket] so that we
can know which COS the domain is using on the socket.
8. Set dom_ids bit corresponding to the domain so that we can know the domain
has been set and the COS ID of the domain is valid.
So, some functions are abstracted and the callback functions will be
implemented in next patches.
Here is an example to understand the process. The CPU supports
two featuers, e.g. L3 CAT and L2 CAT. User wants to set L3 CAT
of Dom1 to 0x1ff.
1. At the initial time, the old_cos of Dom1 is 0. The COS registers values
are below at this time.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | 0x7ff | 0x7ff | ... |
-------------------------------
L2 CAT | 0xff | 0xff | 0xff | ... |
-------------------------------
2. Gather the value array and insert new value into it:
val[0]: 0x1ff
val[1]: 0xff
3. It cannot find a matching COS.
4. Pick COS 1 to store the value set.
5. Write the L3 CAT COS 1 registers. The COS registers values are
changed to below now.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | 0x1ff | ... | ... |
-------------------------------
L2 CAT | 0xff | 0xff | ... | ... |
-------------------------------
6. The ref[1] is increased to 1 because Dom1 is using it now.
7. Save 1 to Dom1's psr_cos_ids[socket].
8. Set the bit in 'dom_ids[]'.
Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
of Dom2 is 0 too. Repeat above flow.
The val array assembled is:
val[0]: 0x1ff
val[1]: 0xff
So, it can find a matching COS, COS 1. Then, it can reuse COS 1
for Dom2.
The ref[1] is increased to 2 now because both Dom1 and Dom2 are
using this COS ID. Set 1 to Dom2's psr_cos_ids[socket].
There is one thing need to emphasize that we need restore domain's COS ID to
0 when socket is offline. Otherwise, a wrong COS ID will be used when the
socket is online again. That may cause user see the wrong CBM shown. But it
takes much time to iterate all domains to restore COS ID to 0. So, we define
a 'dom_ids[]' to represents all domains, one bit corresponds to one domain.
If the bit is 0 when entering 'psr_ctxt_switch_to', that means this is the
first time the domain is switched to this socket or domain's COS ID has not
been set since the socket is online. So, the COS ID set to ASSOC register on
this socket should be default value, 0. If not, that means the domain's COS
ID has been set when the socket was online. So, this COS ID is valid and we
can directly use it. We restore the domain's COS ID to 0 if the bit
corresponding to the domain is 0 but the domain's COS ID is not 0 when
'psr_get_val' is called. This can avoid CPU serialization if restoring action
is exectued in 'psr_ctxt_switch_to'.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- define 'dom_ids[]' and implement related flows.
- restore domain cos id to 0 in 'psr_get_val'.
- rename 'write_psr_msr' to 'write_psr_msrs' and change its parameters to
handle value array the feature's all MSRs.
- fix coding style issue.
(suggested by Jan Beulich)
- do not need check 'cos' in ASSERT.
(suggested by Jan Beulich)
- rename 'insert_val_to_array' to 'insert_val_into_array'.
(suggested by Jan Beulich)
- remove 'ref_lock' from parameter list in 'find_cos' and 'pick_avail_cos'.
(suggested by Jan Beulich)
- remove ASSERT check to 'ref_lock' in 'find_cos' and 'pick_avail_cos'.
(suggested by Jan Beulich)
- fix a bug for checking 'feat_type'.
(suggested by Jan Beulich)
- move 'free_array' label.
(suggested by Jan Beulich)
- modify comments and commit message.
v10:
- restore domain cos id to 0 when socket is offline.
(suggested by Jan Beulich)
- check 'psr_cat_op.data' to make sure only lower 32 bits are valid.
(suggested by Jan Beulich)
- remove unnecessary fixed width type of parameters and variables.
(suggested by Jan Beulich)
- rename 'insert_new_val_to_array' to 'insert_val_to_array'.
(suggested by Jan Beulich)
- input 'ref_lock' pointer into functions to check if it has been locked.
(suggested by Jan Beulich)
- add comment to declare the set process is protected by 'domctl_lock'.
(suggested by Jan Beulich)
- check 'feat_type'.
(suggested by Jan Beulich)
- remove 'feat_mask'.
(suggested by Jan Beulich)
- remove unnecessary criteria of ASSERT.
(suggested by Jan Beulich)
- adjust flow of 'psr_set_val' to avoid 'goto' for successful cases.
(suggested by Jan Beulich)
- use ASSERT to check 'socket_info' in 'psr_free_cos'.
(suggested by Jan Beulich)
- remove unnecessary comment in 'psr_free_cos'.
(suggested by Jan Beulich)
v9:
- use goto style error handling in 'psr_set_val'.
(suggested by Wei Liu)
- use ASSERT for checking old_cos.
(suggested by Wei Liu and Jan Beulich)
- fix coding style issue.
(suggested by Wei Liu)
- rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
(suggested by Wei Liu)
- use 'spin_is_locked' to check ref_lock.
(suggested by Roger Pau)
- add an input parameter 'array_len' for 'write_psr_msr'.
- check 'socket_info' and 'psr_cos_ids' in this patch.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- fix commit message words.
(suggested by Jan Beulich)
- change 'assemble_val_array' to 'gather_val_array'.
(suggested by Jan Beulich)
- change 'set_new_val_to_array' to 'insert_new_val_to_array'.
(suggested by Jan Beulich)
- change parameter 'm' of 'insert_new_val_to_array' to 'new_val'.
(suggested by Jan Beulich)
- change 'write_psr_msr' to 'write_psr_msrs'.
(suggested by Jan Beulich)
- correct comments.
(suggested by Jan Beulich)
- remove unnecessary comments.
(suggested by Jan Beulich)
- adjust conditions after 'find_cos' to save a level of indentation.
(suggested by Jan Beulich)
- add 'ASSERT(!old_cos || ref[old_cos])'.
(suggested by Jan Beulich)
- move ASSERT() check into locked region.
(suggested by Jan Beulich)
- replace parameter '*val' to 'val[]' in some functions.
(suggested by Jan Beulich)
- change 'write_psr_msr' parameters to prepare to only set one new value
for one feature.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
- add explanation about context switch.
(suggested by Jan Beulich)
v5:
- modify commit message.
(suggested by Jan Beulich)
- return an error for all helper functions in set flow.
(suggested by Jan Beulich)
- remove unnecessary cast.
(suggested by Jan Beulich)
- divide 'get_old_set_new' to two functions, 'assemble_val_array' and
'set_new_val_to_array'.
(suggested by Jan Beulich)
- modify comments.
(suggested by Jan Beulich)
- adjust code format.
(suggested by Jan Beulich)
- change 'alloc_new_cos' to 'pick_avail_cos' to make name accurate.
(suggested by Jan Beulich)
- check feature type when entering 'psr_set_val'.
(suggested by Jan Beulich)
- use ASSERT to check ref.
(suggested by Jan Beulich)
- rename 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/domctl.c | 18 ++--
xen/arch/x86/psr.c | 239 ++++++++++++++++++++++++++++++++++++++++++++--
xen/include/asm-x86/psr.h | 4 +-
3 files changed, 241 insertions(+), 20 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 5b62c5c..124a1c6 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1411,21 +1411,21 @@ long arch_do_domctl(
uint32_t val32;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_CODE);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_DATA);
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 1b781e1..7658786 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -118,11 +118,13 @@ static const struct feat_props {
* COS ID. Every entry of cos_ref corresponds to one COS ID.
*/
struct psr_socket_info {
- bool feat_init;
- spinlock_t ref_lock;
/* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
struct feat_node *features[PSR_SOCKET_FEAT_NUM];
+ bool feat_init;
unsigned int cos_ref[MAX_COS_REG_CNT];
+ spinlock_t ref_lock;
+ /* Every bit corresponds to a domain. Index is domain_id. */
+ DECLARE_BITMAP(dom_ids, DOMID_IDLE + 1);
};
struct psr_assoc {
@@ -178,6 +180,10 @@ static void free_socket_resources(unsigned int socket)
}
info->feat_init = false;
+
+ memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
+
+ memset(info->dom_ids, 0, ((DOMID_IDLE + 1) + 7) / 8);
}
static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
@@ -449,11 +455,19 @@ void psr_ctxt_switch_to(struct domain *d)
/* IDLE domain's 'psr_cos_ids' is NULL so we set default value for it. */
if ( psra->cos_mask )
- psr_assoc_cos(®,
- d->arch.psr_cos_ids ?
- d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
- 0,
- psra->cos_mask);
+ {
+ unsigned int socket = cpu_to_socket(smp_processor_id());
+ struct psr_socket_info *info = socket_info + socket;
+ unsigned int cos = 0;
+
+ if ( d->arch.psr_cos_ids )
+ cos = d->arch.psr_cos_ids[socket];
+
+ if ( unlikely(!test_bit(d->domain_id, info->dom_ids)) )
+ cos = 0;
+
+ psr_assoc_cos(®, cos, psra->cos_mask);
+ }
if ( reg != psra->val )
{
@@ -529,6 +543,10 @@ int psr_get_val(struct domain *d, unsigned int socket,
if ( !feat || !feat_props[feat_type] )
return -ENOENT;
+ if ( !test_bit(d->domain_id, socket_info[socket].dom_ids) &&
+ d->arch.psr_cos_ids[socket] )
+ d->arch.psr_cos_ids[socket] = 0;
+
cos = d->arch.psr_cos_ids[socket];
/*
* If input cos exceeds current feature's cos_max, we should return its
@@ -548,15 +566,218 @@ int psr_get_val(struct domain *d, unsigned int socket,
return 0;
}
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type)
+/* Set value functions */
+static unsigned int get_cos_num(const struct psr_socket_info *info)
{
return 0;
}
+static int gather_val_array(uint32_t val[],
+ unsigned int array_len,
+ const struct psr_socket_info *info,
+ unsigned int old_cos)
+{
+ return -EINVAL;
+}
+
+static int insert_val_into_array(uint32_t val[],
+ unsigned int array_len,
+ const struct psr_socket_info *info,
+ enum psr_feat_type feat_type,
+ enum cbm_type type,
+ uint32_t new_val)
+{
+ return -EINVAL;
+}
+
+static int find_cos(const uint32_t val[], unsigned int array_len,
+ enum psr_feat_type feat_type,
+ const struct psr_socket_info *info)
+{
+ return -ENOENT;
+}
+
+static int pick_avail_cos(const struct psr_socket_info *info,
+ const uint32_t val[], unsigned int array_len,
+ unsigned int old_cos,
+ enum psr_feat_type feat_type)
+{
+ return -ENOENT;
+}
+
+static int write_psr_msrs(unsigned int socket, unsigned int cos,
+ uint32_t val[], unsigned int array_len,
+ enum psr_feat_type feat_type)
+{
+ return -ENOENT;
+}
+
+/* The whole set process is protected by domctl_lock. */
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t new_val, enum cbm_type type)
+{
+ unsigned int old_cos;
+ int cos, ret;
+ unsigned int *ref;
+ uint32_t *val_array, val;
+ struct psr_socket_info *info = get_socket_info(socket);
+ unsigned int array_len;
+ enum psr_feat_type feat_type;
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ if ( new_val != (uint32_t)new_val )
+ return -EINVAL;
+
+ val = new_val;
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( feat_type >= ARRAY_SIZE(info->features) ||
+ !info->features[feat_type] )
+ return -ENOENT;
+
+ /*
+ * Step 0:
+ * old_cos means the COS ID current domain is using. By default, it is 0.
+ *
+ * For every COS ID, there is a reference count to record how many domains
+ * are using the COS register corresponding to this COS ID.
+ * - If ref[old_cos] is 0, that means this COS is not used by any domain.
+ * - If ref[old_cos] is 1, that means this COS is only used by current
+ * domain.
+ * - If ref[old_cos] is more than 1, that mean multiple domains are using
+ * this COS.
+ */
+ old_cos = d->arch.psr_cos_ids[socket];
+ ASSERT(old_cos < MAX_COS_REG_CNT);
+
+ ref = info->cos_ref;
+
+ /*
+ * Step 1:
+ * Gather a value array to store all features cos_reg_val[old_cos].
+ * And, set the input new val into array according to the feature's
+ * position in array.
+ */
+ array_len = get_cos_num(info);
+ val_array = xzalloc_array(uint32_t, array_len);
+ if ( !val_array )
+ return -ENOMEM;
+
+ if ( (ret = gather_val_array(val_array, array_len, info, old_cos)) != 0 )
+ goto free_array;
+
+ if ( (ret = insert_val_into_array(val_array, array_len, info,
+ feat_type, type, val)) != 0 )
+ goto free_array;
+
+ spin_lock(&info->ref_lock);
+
+ /*
+ * Step 2:
+ * Try to find if there is already a COS ID on which all features' values
+ * are same as the array. Then, we can reuse this COS ID.
+ */
+ cos = find_cos(val_array, array_len, feat_type, info);
+ if ( cos == old_cos )
+ {
+ ret = 0;
+ goto unlock_free_array;
+ }
+
+ /*
+ * Step 3:
+ * If fail to find, we need pick an available COS ID.
+ * In fact, only COS ID which ref is 1 or 0 can be picked for current
+ * domain. If old_cos is not 0 and its ref==1, that means only current
+ * domain is using this old_cos ID. So, this old_cos ID certainly can
+ * be reused by current domain. Ref==0 means there is no any domain
+ * using this COS ID. So it can be used for current domain too.
+ */
+ if ( cos < 0 )
+ {
+ cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
+ if ( cos < 0 )
+ {
+ ret = cos;
+ goto unlock_free_array;
+ }
+
+ /*
+ * Step 4:
+ * Write the feature's MSRs according to the COS ID.
+ */
+ ret = write_psr_msrs(socket, cos, val_array, array_len, feat_type);
+ if ( ret )
+ goto unlock_free_array;
+ }
+
+ /*
+ * Step 5:
+ * Find the COS ID (find_cos result is '>= 0' or an available COS ID is
+ * picked, then update ref according to COS ID.
+ */
+ ref[cos]++;
+ ASSERT(!cos || ref[cos]);
+ ASSERT(!old_cos || ref[old_cos]);
+ ref[old_cos]--;
+ spin_unlock(&info->ref_lock);
+
+ /*
+ * Step 6:
+ * Save the COS ID into current domain's psr_cos_ids[] so that we can know
+ * which COS the domain is using on the socket. One domain can only use
+ * one COS ID at same time on each socket.
+ */
+ d->arch.psr_cos_ids[socket] = cos;
+
+ /*
+ * Step 7:
+ * Then, set the dom_ids bit which corresponds to domain_id to mark this
+ * domain has been set and the COS ID of the domain is valid.
+ */
+ set_bit(d->domain_id, info->dom_ids);
+
+ free_array:
+ xfree(val_array);
+ return ret;
+
+ unlock_free_array:
+ spin_unlock(&info->ref_lock);
+ xfree(val_array);
+ return ret;
+}
+
/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
static void psr_free_cos(struct domain *d)
{
+ unsigned int socket, cos;
+
+ ASSERT(socket_info);
+
+ if ( !d->arch.psr_cos_ids )
+ return;
+
+ /* Domain is destroied so its cos_ref should be decreased. */
+ for ( socket = 0; socket < nr_sockets; socket++ )
+ {
+ struct psr_socket_info *info;
+
+ /* cos 0 is default one which does not need be handled. */
+ cos = d->arch.psr_cos_ids[socket];
+ if ( cos == 0 )
+ continue;
+
+ info = socket_info + socket;
+ spin_lock(&info->ref_lock);
+ ASSERT(info->cos_ref[cos]);
+ info->cos_ref[cos]--;
+ spin_unlock(&info->ref_lock);
+
+ clear_bit(d->domain_id, info->dom_ids);
+ }
+
xfree(d->arch.psr_cos_ids);
d->arch.psr_cos_ids = NULL;
}
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 7c6d38a..ae52d85 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -73,8 +73,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
int psr_get_val(struct domain *d, unsigned int socket,
uint32_t *val, enum cbm_type type);
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type);
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t val, enum cbm_type type);
int psr_domain_init(struct domain *d);
void psr_domain_free(struct domain *d);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (7 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-30 15:17 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
` (13 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Only can one COS ID be used by one domain at one time. That means all enabled
features' COS registers at this COS ID are valid for this domain at that time.
When user updates a feature's value, we need make sure all other features'
values are not affected. So, we firstly need gather an array which contains
all features current values and replace the setting feature's value in array
to new value.
Then, we can try to find if there is a COS ID on which all features' COS
registers values are same as the array. If we can find, we just use this COS
ID. If fail to find, we need pick a new COS ID.
This patch implements value array assembling flow.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- changes about 'feat_props'.
(suggested by Jan Beulich)
- changes about 'get_val'. It is replaced by generic codes.
(suggested by Jan Beulich)
v10:
- remove 'get_old_val' to directly call 'get_val' to get needed val.
(suggested by Jan Beulich)
- move 'psr_check_cbm' into 'insert_val_to_array'.
(suggested by Jan Beulich)
- change type of 'cbm' in 'psr_check_cbm' to 'unsigned long'.
(suggested by Jan Beulich)
- remove 'set_new_val' as it can be handled in generic process.
- changes related to 'feat_props'.
(suggested by Jan Beulich)
- adjust flow in 'gather_val_array' to avoid array cross.
(suggested by Jan Beulich)
- adjust flow in 'insert_val_to_array' to avoid array cross.
(suggested by Jan Beulich)
v9:
- add comments about boundary checking.
(suggested by Wei Liu)
- rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
(suggested by Wei Liu)
- rename 'l3_cat_get_cos_num' to 'cat_get_cos_num' to cover all L3/L2 CAT
features.
(suggested by Roger Pau)
- rename 'l3_cat_get_old_val' to 'cat_get_old_val' to cover all L3/L2 CAT
features and reuse cat_get_val in it.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- replace 'm' to 'new_val'.
(suggested by Jan Beulich)
- move cos check outside callback function.
(suggested by Jan Beulich)
- remove 'get_cos_num' callback function.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v6:
- change 'assemble_val_array' to 'combine_val_array'.
(suggested by Konrad Rzeszutek Wilk)
- check return value of 'get_old_val'.
(suggested by Konrad Rzeszutek Wilk)
- replace some 'EINVAL' to 'ENOSPC'.
(suggested by Konrad Rzeszutek Wilk)
v5:
- modify comments according to changes of codes.
(suggested by Jan Beulich)
- change 'bool_t' to 'bool'.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 110 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 107 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 7658786..b15924e 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -202,6 +202,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
return feat_type;
}
+static bool psr_check_cbm(unsigned int cbm_len, unsigned long cbm)
+{
+ unsigned int first_bit, zero_bit;
+
+ /* Set bits should only in the range of [0, cbm_len]. */
+ if ( cbm & (~0ul << cbm_len) )
+ return false;
+
+ /* At least one bit need to be set. */
+ if ( cbm == 0 )
+ return false;
+
+ first_bit = find_first_bit(&cbm, cbm_len);
+ zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
+
+ /* Set bits should be contiguous. */
+ if ( zero_bit < cbm_len &&
+ find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
+ return false;
+
+ return true;
+}
+
/* CAT common functions implementation. */
static void cat_init_feature(const struct cpuid_leaf *regs,
struct feat_node *feat,
@@ -569,7 +592,14 @@ int psr_get_val(struct domain *d, unsigned int socket,
/* Set value functions */
static unsigned int get_cos_num(const struct psr_socket_info *info)
{
- return 0;
+ unsigned int num = 0, i;
+
+ /* Get all features total amount. */
+ for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
+ if ( feat_props[i] )
+ num += feat_props[i]->cos_num;
+
+ return num;
}
static int gather_val_array(uint32_t val[],
@@ -577,7 +607,41 @@ static int gather_val_array(uint32_t val[],
const struct psr_socket_info *info,
unsigned int old_cos)
{
- return -EINVAL;
+ unsigned int i;
+
+ if ( !val )
+ return -EINVAL;
+
+ /* Get all features current values according to old_cos. */
+ for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
+ {
+ unsigned int cos = old_cos, j;
+ const struct feat_node *feat = info->features[i];
+ const struct feat_props *props = feat_props[i];
+
+ if ( !feat || !props )
+ continue;
+
+ if ( array_len < props->cos_num )
+ return -ENOSPC;
+
+ /*
+ * If old_cos exceeds current feature's cos_max, we should get
+ * default value. So assign cos to 0 which stores default value.
+ */
+ if ( cos > feat->cos_max )
+ cos = 0;
+
+ /* Value getting order is same as feature array. */
+ for ( j = 0; j < props->cos_num; j++ )
+ val[j] = feat->cos_reg_val[cos * props->cos_num + j];
+
+ array_len -= props->cos_num;
+
+ val += props->cos_num;
+ }
+
+ return 0;
}
static int insert_val_into_array(uint32_t val[],
@@ -587,7 +651,47 @@ static int insert_val_into_array(uint32_t val[],
enum cbm_type type,
uint32_t new_val)
{
- return -EINVAL;
+ const struct feat_node *feat;
+ const struct feat_props *props;
+ unsigned int i;
+
+ ASSERT(feat_type < PSR_SOCKET_FEAT_NUM);
+
+ /* Insert new value into array according to feature's position in array. */
+ for ( i = 0; i < feat_type; i++ )
+ {
+ props = feat_props[i];
+ if ( !info->features[i] || !props )
+ continue;
+
+ if ( array_len <= props->cos_num )
+ return -ENOSPC;
+
+ array_len -= props->cos_num;
+
+ val += props->cos_num;
+ }
+
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ props = feat_props[feat_type];
+ if ( !props )
+ return -ENOENT;
+
+ if ( array_len < props->cos_num )
+ return -ENOSPC;
+
+ if ( !psr_check_cbm(feat->cbm_len, new_val) )
+ return -EINVAL;
+
+ /* Value setting position is same as feature array. */
+ for ( i = 0; i < props->cos_num; i++ )
+ if ( type == props->type[i] )
+ val[i] = new_val;
+
+ return 0;
}
static int find_cos(const uint32_t val[], unsigned int array_len,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (8 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-30 15:23 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
` (12 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Continue from patch:
'x86: refactor psr: L3 CAT: set value: assemble features value array'
We can try to find if there is a COS ID on which all features' COS registers
values are same as the array assembled before.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- move 'compare_val' implementation from CDP patch to this.
(suggested by Jan Beulich)
- remove 'rc' in 'compare_val' to make codes clearer.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- fix bug.
(suggested by Jan Beulich)
- remove unnecessary variable initialization.
(suggested by Jan Beulich)
- remove unnecessary comment.
(suggested by Jan Beulich)
- adjust array check place prior to using the array.
(suggested by Jan Beulich)
- add comment.
v10:
- remove 'compare_val' hook and its CAT implementation. Make its
functionality be generic in 'find_cos' flow.
(suggested by Jan Beulich)
- changes related to 'props'.
(suggested by Jan Beulich)
- rename 'val_array' to 'val_ptr'.
(suggested by Jan Beulich)
- rename 'find' to 'found'.
(suggested by Jan Beulich)
- move some variables declaration and initialization into loop.
(suggested by Jan Beulich)
- adjust codes positions.
(suggested by Jan Beulich)
v9:
- modify comments of 'compare_val' to be same as current implementation.
(suggested by Wei Liu)
- fix indentation issue.
(suggested by Wei Liu)
- rename 'l3_cat_compare_val' to 'cat_compare_val' to cover all L3/L2 CAT
features.
(suggested by Roger Pau)
- remove parameter 'found' from 'cat_compare_val' and modify the return
values to let caller know if the id is found or not.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- replace 'get_cos_num' to 'feat->cos_num'.
(suggested by Jan Beulich)
- directly use 'cos_reg_val[0]' as default value.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v5:
- modify commit message to provide exact patch name to continue from.
(suggested by Jan Beulich)
- remove 'get_cos_max_from_type' because it can be replaced by
'get_cos_max'.
- move type check out from callback functions to caller.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- modify comments according to changes of codes.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 96 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index b15924e..c81ccc2 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -694,10 +694,106 @@ static int insert_val_into_array(uint32_t val[],
return 0;
}
+static int compare_val(const uint32_t val[],
+ const struct feat_node *feat,
+ const struct feat_props *props,
+ unsigned int cos)
+{
+ unsigned int i;
+
+ for ( i = 0; i < props->cos_num; i++ )
+ {
+ uint32_t feat_val;
+
+ /* If cos is bigger than cos_max, we need compare default value. */
+ if ( cos > feat->cos_max )
+ {
+ /*
+ * COS ID 0 always stores the default value.
+ * For CDP:
+ * - DATA default value stored in cos_reg_val[0];
+ * - CODE default value stored in cos_reg_val[1].
+ */
+ feat_val = feat->cos_reg_val[i];
+
+ /*
+ * If cos is bigger than feature's cos_max, the val should be
+ * default value. Otherwise, it fails to find a COS ID. So we
+ * have to exit find flow.
+ */
+ if ( val[i] != feat_val )
+ return -EINVAL;
+ }
+ else
+ {
+ feat_val = feat->cos_reg_val[cos * props->cos_num + i];
+ if ( val[i] != feat_val )
+ return 0;
+ }
+ }
+
+ return 1;
+}
+
static int find_cos(const uint32_t val[], unsigned int array_len,
enum psr_feat_type feat_type,
const struct psr_socket_info *info)
{
+ unsigned int cos, i;
+ const unsigned int *ref = info->cos_ref;
+ const struct feat_node *feat;
+ unsigned int cos_max;
+
+ /* cos_max is the one of the feature which is being set. */
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ cos_max = feat->cos_max;
+
+ for ( cos = 0; cos <= cos_max; cos++ )
+ {
+ const uint32_t *val_ptr = val;
+ unsigned int len = array_len;
+ int rc = 0;
+
+ if ( cos && !ref[cos] )
+ continue;
+
+ for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
+ {
+ const struct feat_props *props = feat_props[i];
+
+ feat = info->features[i];
+ if ( !feat || !props )
+ continue;
+
+ if ( len < props->cos_num )
+ return -ENOSPC;
+
+ /*
+ * Compare value according to feature array order.
+ * We must follow this order because value array is assembled
+ * as this order.
+ */
+ rc = compare_val(val_ptr, feat, props, cos);
+ if ( rc < 0 )
+ return rc;
+
+ /* If fail to match, go to next cos to compare. */
+ if ( !rc )
+ break;
+
+ len -= props->cos_num;
+
+ val_ptr += props->cos_num;
+ }
+
+ /* For this COS ID all entries in the values array do match. Use it. */
+ if ( rc )
+ return cos;
+ }
+
return -ENOENT;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (9 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
` (11 subsequent siblings)
22 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Continue from previous patch:
'x86: refactor psr: L3 CAT: set value: implement cos finding flow.'
If fail to find a COS ID, we need pick a new COS ID for domain. Only COS ID
that ref[COS_ID] is 1 or 0 can be picked to input a new set feature values.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- remove unnecessary variable initialization.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- replace 'get_val' calling to generic codes.
(suggested by Jan Beulich)
v10:
- remove 'fits_cos_max' hook and CAT implementation. Move the process into
generic flow.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- adjust codes positions.
(suggested by Jan Beulich)
v9:
- modify return value of 'pick_avail_cos' to make it more accurate.
- rename 'l3_cat_fits_cos_max' to 'cat_fits_cos_max' to cover L3/L2 CAT
features.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- fix comment.
(suggested by Wei Liu)
- directly use 'cos_reg_val[0]' as default value.
(suggested by Jan Beulich)
- replace 'get_cos_num' to 'feat->cos_num'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v5:
- modify commit message to provide exact patch name to continue from.
(suggested by Jan Beulich)
- change 'exceeds_cos_max' to 'fits_cos_max' to be accurate.
(suggested by Jan Beulich)
- modify comments according to changes of codes.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- move type check out from callback functions to caller.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- modify code format.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 81 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index c81ccc2..9693506 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -797,12 +797,92 @@ static int find_cos(const uint32_t val[], unsigned int array_len,
return -ENOENT;
}
+static bool fits_cos_max(const uint32_t val[],
+ uint32_t array_len,
+ const struct psr_socket_info *info,
+ unsigned int cos)
+{
+ unsigned int i;
+
+ for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
+ {
+ const struct feat_node *feat = info->features[i];
+ const struct feat_props *props = feat_props[i];
+
+ if ( !feat || !props )
+ continue;
+
+ if ( array_len < props->cos_num )
+ return false;
+
+ if ( cos > feat->cos_max )
+ {
+ unsigned int j;
+
+ for ( j = 0; j < props->cos_num; j++ )
+ {
+ /*
+ * Get default value which cos id is 0.
+ * For CDP:
+ * - DATA default value stored in cos_reg_val[0];
+ * - CODE default value stored in cos_reg_val[1].
+ */
+ uint32_t default_val = feat->cos_reg_val[j];
+
+ if ( val[j] != default_val )
+ return false;
+ }
+ }
+
+ array_len -= props->cos_num;
+
+ val += props->cos_num;
+ }
+
+ return true;
+}
+
static int pick_avail_cos(const struct psr_socket_info *info,
const uint32_t val[], unsigned int array_len,
unsigned int old_cos,
enum psr_feat_type feat_type)
{
- return -ENOENT;
+ unsigned int cos;
+ unsigned int cos_max = 0;
+ const struct feat_node *feat;
+ const unsigned int *ref = info->cos_ref;
+
+ /* cos_max is the one of the feature which is being set. */
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ cos_max = feat->cos_max;
+ if ( !cos_max )
+ return -ENOENT;
+
+ /* We cannot use id 0 because it stores the default values. */
+ if ( old_cos && ref[old_cos] == 1 &&
+ fits_cos_max(val, array_len, info, old_cos) )
+ return old_cos;
+
+ /* Find an unused one other than cos0. */
+ for ( cos = 1; cos <= cos_max; cos++ )
+ {
+ /*
+ * ref is 0 means this COS is not used by other domain and
+ * can be used for current setting.
+ */
+ if ( !ref[cos] )
+ {
+ if ( !fits_cos_max(val, array_len, info, cos) )
+ break;
+
+ return cos;
+ }
+ }
+
+ return -EOVERFLOW;
}
static int write_psr_msrs(unsigned int socket, unsigned int cos,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (10 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-30 15:35 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
` (10 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Continue from previous patch:
'x86: refactor psr: L3 CAT: set value: implement cos id picking flow.'
We have got the feature value and COS ID to set. Then, we write MSRs of the
designated feature.
Till now, set value process is completed.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- rename 'write_psr_msr' to 'write_psr_msrs'.
- rename 'do_write_psr_msr' to 'do_write_psr_msrs'.
- change parameters and codes of 'write_psr_msrs' to handle value array.
- add 'feat_type' in 'struct cos_write_info' to handle props array.
- in 'do_write_psr_msrs', write value array into msrs according to
'props->type[i]'.
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
- move check from 'do_write_psr_msrs' to 'write_psr_msrs'.
(suggested by Jan Beulich)
- change about 'cos_max'.
(suggested by Jan Beulich)
- change about 'feat_props'.
(suggested by Jan Beulich)
v10:
- remove 'type' from 'write_msr' parameter list. Will add it back when
implementing CDP.
(suggested by Jan Beulich)
- remove unnecessary casts.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
v9:
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- add 'array_len' in 'struct cos_write_info' and check if val array
exceeds it.
- modify 'write_psr_msr' flow only to set one value a time. No need to
set whole feature array values.
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- modify 'write_msr' callback function to 'void' because we have to set
all features' cbm. When input cos exceeds some features' cos_max, just
skip them but not break the iteration.
v5:
- modify commit message to provide exact patch name to continue from.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- move type check out from callback functions to caller.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- correct code format.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 86 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 9693506..9b8428d 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -107,6 +107,9 @@ static const struct feat_props {
/* get_feat_info is used to return feature HW info through sysctl. */
bool (*get_feat_info)(const struct feat_node *feat,
uint32_t data[], unsigned int array_len);
+
+ /* write_msr is used to write out feature MSR register. */
+ void (*write_msr)(unsigned int cos, uint32_t val, enum cbm_type type);
} *feat_props[PSR_SOCKET_FEAT_NUM];
/*
@@ -275,9 +278,15 @@ static bool cat_get_feat_info(const struct feat_node *feat,
}
/* L3 CAT props */
+static void l3_cat_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
+{
+ wrmsrl(MSR_IA32_PSR_L3_MASK(cos), val);
+}
+
static struct feat_props l3_cat_props = {
.cos_num = 1,
.get_feat_info = cat_get_feat_info,
+ .write_msr = l3_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -885,11 +894,87 @@ static int pick_avail_cos(const struct psr_socket_info *info,
return -EOVERFLOW;
}
+static unsigned int get_socket_cpu(unsigned int socket)
+{
+ if ( likely(socket < nr_sockets) )
+ return cpumask_any(socket_cpumask[socket]);
+
+ return nr_cpu_ids;
+}
+
+struct cos_write_info
+{
+ unsigned int cos;
+ struct feat_node *feature;
+ uint32_t *val;
+ enum psr_feat_type feat_type;
+};
+
+static void do_write_psr_msrs(void *data)
+{
+ struct cos_write_info *info = data;
+ unsigned int cos = info->cos;
+ struct feat_node *feat = info->feature;
+ const struct feat_props *props = feat_props[info->feat_type];
+ unsigned int i;
+
+ for ( i = 0; i < props->cos_num; i++ )
+ {
+ if ( feat->cos_reg_val[cos * props->cos_num + i] != info->val[i] )
+ {
+ feat->cos_reg_val[cos * props->cos_num + i] = info->val[i];
+ props->write_msr(cos, info->val[i], props->type[i]);
+ }
+ }
+}
+
static int write_psr_msrs(unsigned int socket, unsigned int cos,
uint32_t val[], unsigned int array_len,
enum psr_feat_type feat_type)
{
- return -ENOENT;
+ unsigned int i;
+ struct psr_socket_info *info = get_socket_info(socket);
+ struct cos_write_info data =
+ {
+ .cos = cos,
+ .feature = info->features[feat_type],
+ .feat_type = feat_type,
+ };
+
+ if ( cos > info->features[feat_type]->cos_max )
+ return -EINVAL;
+
+ /* Skip to the feature's value head. */
+ for ( i = 0; i < feat_type; i++ )
+ {
+ if ( !info->features[i] )
+ continue;
+
+ if ( array_len <= feat_props[feat_type]->cos_num )
+ return -ENOSPC;
+
+ array_len -= feat_props[feat_type]->cos_num;
+
+ val += feat_props[feat_type]->cos_num;
+ }
+
+ if ( array_len < feat_props[feat_type]->cos_num )
+ return -ENOSPC;
+
+ data.val = val;
+
+ if ( socket == cpu_to_socket(smp_processor_id()) )
+ do_write_psr_msrs(&data);
+ else
+ {
+ unsigned int cpu = get_socket_cpu(socket);
+
+ if ( cpu >= nr_cpu_ids )
+ return -ENOTSOCK;
+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msrs, &data, 1);
+ }
+
+ return 0;
}
/* The whole set process is protected by domctl_lock. */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (11 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-31 9:37 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
` (9 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the CPU init flow for CDP. The flow is almost
same as L3 CAT.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- changes about 'feat_props'.
(suggested by Jan Beulich)
- remove MSR restore action which is unnecessary.
(suggested by Jan Beulich)
- modify commit message.
v10:
- fix comment.
(suggested by Jan Beulich)
- use swith in 'cat_init_feature' to handle different feature types.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- restore MSRs to default value when cpu online.
(suggested by Jan Beulich)
- remove feat_mask.
(suggested by Jan Beulich)
v9:
- modify commit message to describe flow clearer.
- handle cpu offline and online again case to read MSRs registers values
back and save them into cos array to make user can get real data.
- modify error handling process in 'psr_cpu_prepare' to reduce redundant
codes.
- modify 'get_cdp_data' and 'get_cdp_code' to make them standard.
(suggested by Roger Pau and Jan Beulich)
- encapsulate CDP operations into 'cat_init_feature' to reduce redundant
codes.
(suggested by Roger Pau)
- reuse 'cat_get_cos_max' for CDP.
(suggested by Roger Pau)
- handle 'PSR_CDP' in psr_presmp_init to make init work can be done when
there is only 'psr=cdp' in cmdline.
- remove unnecessary comment.
(suggested by Jan Beulich)
- move CDP related codes in 'cpu_init_work' into 'psr_cpu_init'.
(suggested by Jan Beulich)
- add codes to handle CDP's 'cos_num'.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- do not free resources when allocation fails in 'psr_cpu_prepare'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v7:
- initialize 'l3_cdp'.
(suggested by Konrad Rzeszutek Wilk)
v6:
- use 'cpuid_leaf'.
(suggested by Konrad Rzeszutek Wilk and Jan Beulich)
v5:
- remove codes to free 'feat_l3_cdp' in 'free_feature'.
(suggested by Jan Beulich)
- encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
(suggested by Jan Beulich)
- print socket info when 'opt_cpu_info' is true.
(suggested by Jan Beulich)
- rename 'l3_cdp_get_max_cos_max' to 'l3_cdp_get_cos_max'.
(suggested by Jan Beulich)
- rename 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
- move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 68 insertions(+), 9 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 9b8428d..7000d95 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -62,6 +62,7 @@
enum psr_feat_type {
PSR_SOCKET_L3_CAT,
+ PSR_SOCKET_L3_CDP,
PSR_SOCKET_FEAT_NUM,
PSR_SOCKET_FEAT_UNKNOWN,
};
@@ -150,11 +151,28 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
* array creation. It is used to transiently store a spare node.
*/
static struct feat_node *feat_l3_cat;
+static struct feat_node *feat_l3_cdp;
/* Common functions */
#define cat_default_val(len) (0xffffffff >> (32 - (len)))
/*
+ * get_cdp_data - get DATA COS register value from input COS ID.
+ * @feat: the feature node.
+ * @cos: the COS ID.
+ */
+#define get_cdp_data(feat, cos) \
+ ( (feat)->cos_reg_val[(cos) * 2] )
+
+/*
+ * get_cdp_code - get CODE COS register value from input COS ID.
+ * @feat: the feature node.
+ * @cos: the COS ID.
+ */
+#define get_cdp_code(feat, cos) \
+ ( (feat)->cos_reg_val[(cos) * 2 + 1] )
+
+/*
* Use this function to check if any allocation feature has been enabled
* in cmdline.
*/
@@ -249,6 +267,25 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
break;
+ case PSR_SOCKET_L3_CDP:
+ {
+ unsigned long val;
+
+ /* Cut half of cos_max when CDP is enabled. */
+ feat->cos_max >>= 1;
+
+ /* We only write mask1 since mask0 is always all ones by default. */
+ wrmsrl(MSR_IA32_PSR_L3_MASK(1), cat_default_val(feat->cbm_len));
+ rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
+ wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
+
+ /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+ get_cdp_code(feat, 0) = cat_default_val(feat->cbm_len);
+ get_cdp_data(feat, 0) = cat_default_val(feat->cbm_len);
+
+ break;
+ }
+
default:
return;
}
@@ -259,8 +296,9 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
if ( !opt_cpu_info )
return;
- printk(XENLOG_INFO "%s CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
- ((type == PSR_SOCKET_L3_CAT) ? "L3" : "L2"),
+ printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ ((type == PSR_SOCKET_L3_CDP) ? "CDP" :
+ ((type == PSR_SOCKET_L3_CAT) ? "L3 CAT": "L2 CAT")),
cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
}
@@ -289,6 +327,11 @@ static struct feat_props l3_cat_props = {
.write_msr = l3_cat_write_msr,
};
+/* L3 CDP props */
+static struct feat_props l3_cdp_props = {
+ .cos_num = 2,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1206,6 +1249,10 @@ static int psr_cpu_prepare(void)
(feat_l3_cat = xzalloc(struct feat_node)) == NULL )
return -ENOMEM;
+ if ( feat_l3_cdp == NULL &&
+ (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
@@ -1238,12 +1285,24 @@ static void psr_cpu_init(void)
{
cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
- feat = feat_l3_cat;
- feat_l3_cat = NULL;
- l3_cat_props.type[0] = PSR_CBM_TYPE_L3;
- feat_props[PSR_SOCKET_L3_CAT] = &l3_cat_props;
-
- cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CAT);
+ if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
+ !info->features[PSR_SOCKET_L3_CDP] )
+ {
+ feat = feat_l3_cdp;
+ feat_l3_cdp = NULL;
+ l3_cdp_props.type[0] = PSR_CBM_TYPE_L3_DATA;
+ l3_cdp_props.type[1] = PSR_CBM_TYPE_L3_CODE;
+ feat_props[PSR_SOCKET_L3_CDP] = &l3_cdp_props;
+ cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CDP);
+ }
+ else
+ {
+ feat = feat_l3_cat;
+ feat_l3_cat = NULL;
+ l3_cat_props.type[0] = PSR_CBM_TYPE_L3;
+ feat_props[PSR_SOCKET_L3_CAT] = &l3_cat_props;
+ cat_init_feature(®s, feat, info, PSR_SOCKET_L3_CAT);
+ }
info->feat_init = true;
}
@@ -1305,7 +1364,7 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- if ( opt_psr & PSR_CAT )
+ if ( opt_psr & (PSR_CAT | PSR_CDP) )
init_psr();
if ( psr_cpu_prepare() )
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 14/23] x86: refactor psr: CDP: implement get hw info flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (12 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-31 9:40 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
` (8 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements get HW info flow for CDP including L3 CDP callback
function. The flow is almost same as L3 CAT.
With this patch, 'psr-hwinfo' can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- modify 'psr_get_info' flow to make it simple to cover CDP case.
v10:
- update renamed macros used by psr_get_info.
(suggested by Jan Beulich)
- change 'psr_get_info' flow to cover CDP case to make codes in sysctl
more simple.
(suggested by Jan Beulich)
- remove sysctl redundant codes after applying above changes.
(suggested by Jan Beulich)
v9:
- modify commit message to explain flow more clearly.
- reuse 'cat_get_feat_info' for CDP to reduce redundant codes.
(suggested by Roger Pau)
- fix coding style issues.
(suggested by Wei Liu and Roger Pau)
- rename macros used by psr_get_info to make them meaningful.
(suggested by Jan Beulich)
v5:
- rename 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
- remove type check in callback function.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 31 ++++++++++++++++++++++++++++---
1 file changed, 28 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 7000d95..55ac221 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -207,7 +207,9 @@ static void free_socket_resources(unsigned int socket)
memset(info->dom_ids, 0, ((DOMID_IDLE + 1) + 7) / 8);
}
-static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
+static enum psr_feat_type psr_cbm_type_to_feat_type(
+ const struct psr_socket_info *info,
+ enum cbm_type type)
{
enum psr_feat_type feat_type = PSR_SOCKET_FEAT_UNKNOWN;
@@ -215,7 +217,18 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
{
case PSR_CBM_TYPE_L3:
feat_type = PSR_SOCKET_L3_CAT;
+
+ /* If type is L3 CAT but we cannot find it in feature array, try CDP. */
+ if ( !info->features[feat_type] )
+ feat_type = PSR_SOCKET_L3_CDP;
+
+ break;
+
+ case PSR_CBM_TYPE_L3_DATA:
+ case PSR_CBM_TYPE_L3_CODE:
+ feat_type = PSR_SOCKET_L3_CDP;
break;
+
default:
ASSERT_UNREACHABLE();
}
@@ -328,8 +341,20 @@ static struct feat_props l3_cat_props = {
};
/* L3 CDP props */
+static bool l3_cdp_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], uint32_t array_len)
+{
+ if ( !cat_get_feat_info(feat, data, array_len) )
+ return false;
+
+ data[PSR_INFO_IDX_CAT_FLAG] |= XEN_SYSCTL_PSR_CAT_L3_CDP;
+
+ return true;
+}
+
static struct feat_props l3_cdp_props = {
.cos_num = 2,
+ .get_feat_info = l3_cdp_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -574,7 +599,7 @@ static struct feat_node *psr_get_feat_and_type(unsigned int socket,
if ( IS_ERR(info) )
return ERR_PTR(PTR_ERR(info));
- *feat_type = psr_cbm_type_to_feat_type(type);
+ *feat_type = psr_cbm_type_to_feat_type(info, type);
if ( *feat_type >= ARRAY_SIZE(info->features) )
return NULL;
@@ -1040,7 +1065,7 @@ int psr_set_val(struct domain *d, unsigned int socket,
val = new_val;
- feat_type = psr_cbm_type_to_feat_type(type);
+ feat_type = psr_cbm_type_to_feat_type(info, type);
if ( feat_type >= ARRAY_SIZE(info->features) ||
!info->features[feat_type] )
return -ENOENT;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (13 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-31 9:44 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
` (7 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L3 CDP set value related callback function.
With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
- changes about setting both CDP DATA and CODE at same time.
- move 'type[]' declaration into previous patch which introduced 'cos_num'.
(suggested by Jan Beulich)
- changes about 'type[]'.
(suggested by Jan Beulich)
- move 'compare_val' to previous patch.
(suggested by Jan Beulich)
- changes about 'get_val' which has been replace by generic codes.
(suggested by Jan Beulich)
- remove 'restore_default_val' which is unnecessary now.
(suggested by Jan Beulich)
v10:
- remove 'l3_cdp_get_old_val' and use 'l3_cdp_get_val' to replace it.
(suggested by Jan Beulich)
- remove 'l3_cdp_set_new_val'.
- modify 'insert_val_to_array' flow to handle multiple COSs case.
(suggested by Jan Beulich)
- remove 'l3_cdp_compare_val' and implement a generic function
'comapre_val'.
(suggested by Jan Beulich)
- remove 'l3_cdp_fits_cos_max'.
(suggested by Jan Beulich)
- introduce macro 'PSR_MAX_COS_NUM'.
- introduce a new member in 'feat_props', 'type[PSR_MAX_COS_NUM]' to record
all 'cbm_type' the feature has.
(suggested by Jan Beulich)
- modify 'gather_val_array' flow to handle multiple COSs case.
(suggested by Jan Beulich)
- modify 'find_cos' flow and implement 'compare_val' to handle multiple
COSs case.
(suggested by Jan Beulich)
- modify 'fits_cos_max' flow to handle multiple COSs case.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- remove cast in 'l3_cdp_write_msr'.
(suggested by Jan Beulich)
- implement 'compare_val' function to compare if feature values are what
we expect in finding flow.
- implement 'restore_default_val' function to restore feature's COS values
to default if the feature has multiple COSs. It is called when the COS
ID is reduced to 0.
v9:
- add comment to explain why CDP uses 2 COSs.
(suggested by Wei Liu)
- use 'cat_default_val'.
(suggested by Wei Liu)
- remove 'l3_cdp_get_cos_num' because we can directly get cos_num from
feat_node now.
(suggested by Jan Beulich)
- remove cos checking because it has been moved to common function.
(suggested by Jan Beulich)
- l3_cdp_set_new_val parameter 'm' is changed to 'new_val'.
(suggested by Jan Beulich)
- directly use get_cdp_data(feat, 0) and get_cdp_code(feat, 0) to get
default value.
(suggested by Jan Beulich)
- modify 'l3_cdp_write_msr' flow to write value into register according
to input type.
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- modify 'l3_cdp_write_msr' type to 'void'.
v5:
- remove type check in callback function.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- remove 'l3_cdp_get_cos_max_from_type'.
- rename 'l3_cdp_exceeds_cos_max' to 'l3_cdp_fits_cos_max'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 55ac221..bfdc20f 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -352,9 +352,21 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
return true;
}
+static void l3_cdp_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
+{
+ /* Data */
+ if ( type == PSR_CBM_TYPE_L3_DATA )
+ wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(cos), val);
+
+ /* Code */
+ if ( type == PSR_CBM_TYPE_L3_CODE )
+ wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(cos), val);
+}
+
static struct feat_props l3_cdp_props = {
.cos_num = 2,
.get_feat_info = l3_cdp_get_feat_info,
+ .write_msr = l3_cdp_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -765,7 +777,8 @@ static int insert_val_into_array(uint32_t val[],
/* Value setting position is same as feature array. */
for ( i = 0; i < props->cos_num; i++ )
- if ( type == props->type[i] )
+ if ( type == props->type[i] ||
+ (feat_type == PSR_SOCKET_L3_CDP && type == PSR_CBM_TYPE_L3) )
val[i] = new_val;
return 0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 16/23] x86: L2 CAT: implement CPU init flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (14 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 17/23] x86: L2 CAT: implement get hw info flow Yi Sun
` (6 subsequent siblings)
22 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the CPU init flow for L2 CAT.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- move l2 cat 'type[]' assignement into 'psr_cpu_init'.
- remove COS MSR restore action in 'cpu_init_feature'.
- set 'feat_init' to true after CPU init.
- modify commit message.
v10:
- implement L2 CAT case in 'cat_init_feature'.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- introduce 'PSR_CBM_TYPE_L2'.
v9:
- modify error handling process in 'psr_cpu_prepare' to reduce redundant
codes.
- reuse 'cat_init_feature' and 'cat_get_cos_max' for L2 CAT to reduce
redundant codes.
(suggested by Roger Pau)
- remove unnecessary comment.
(suggested by Jan Beulich)
- move L2 CAT related codes from 'cpu_init_work' into 'psr_cpu_init'.
(suggested by Jan Beulich)
- do not free resource when allocation fails in 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v7:
- initialize 'l2_cat'.
(suggested by Konrad Rzeszutek Wilk)
v6:
- use 'struct cpuid_leaf'.
(suggested by Konrad Rzeszutek Wilk and Jan Beulich)
v5:
- remove 'feat_l2_cat' free in 'free_feature'.
(suggested by Jan Beulich)
- encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
(suggested by Jan Beulich)
- print socket info when 'opt_cpu_info' is true.
(suggested by Jan Beulich)
- rename 'l2_cat_get_max_cos_max' to 'l2_cat_get_cos_max'.
(suggested by Jan Beulich)
- rename 'dat[]' to 'data[]'
(suggested by Jan Beulich)
- move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 26 ++++++++++++++++++++++++++
xen/include/asm-x86/msr-index.h | 1 +
xen/include/asm-x86/psr.h | 2 ++
3 files changed, 29 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index bfdc20f..75ebb39 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -63,6 +63,7 @@
enum psr_feat_type {
PSR_SOCKET_L3_CAT,
PSR_SOCKET_L3_CDP,
+ PSR_SOCKET_L2_CAT,
PSR_SOCKET_FEAT_NUM,
PSR_SOCKET_FEAT_UNKNOWN,
};
@@ -152,6 +153,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
*/
static struct feat_node *feat_l3_cat;
static struct feat_node *feat_l3_cdp;
+static struct feat_node *feat_l2_cat;
/* Common functions */
#define cat_default_val(len) (0xffffffff >> (32 - (len)))
@@ -275,6 +277,7 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
switch ( type )
{
case PSR_SOCKET_L3_CAT:
+ case PSR_SOCKET_L2_CAT:
/* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
@@ -369,6 +372,11 @@ static struct feat_props l3_cdp_props = {
.write_msr = l3_cdp_write_msr,
};
+/* L2 CAT props */
+static struct feat_props l2_cat_props = {
+ .cos_num = 1,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1291,6 +1299,10 @@ static int psr_cpu_prepare(void)
(feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
return -ENOMEM;
+ if ( feat_l2_cat == NULL &&
+ (feat_l2_cat = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
@@ -1345,6 +1357,20 @@ static void psr_cpu_init(void)
info->feat_init = true;
}
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L2 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, ®s);
+
+ feat = feat_l2_cat;
+ feat_l2_cat = NULL;
+ l2_cat_props.type[0] = PSR_CBM_TYPE_L2;
+ feat_props[PSR_SOCKET_L2_CAT] = &l2_cat_props;
+ cat_init_feature(®s, feat, info, PSR_SOCKET_L2_CAT);
+
+ info->feat_init = true;
+ }
+
assoc_init:
psr_assoc_init();
}
diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h
index 771e750..6c49c6d 100644
--- a/xen/include/asm-x86/msr-index.h
+++ b/xen/include/asm-x86/msr-index.h
@@ -345,6 +345,7 @@
#define MSR_IA32_PSR_L3_MASK(n) (0x00000c90 + (n))
#define MSR_IA32_PSR_L3_MASK_CODE(n) (0x00000c90 + (n) * 2 + 1)
#define MSR_IA32_PSR_L3_MASK_DATA(n) (0x00000c90 + (n) * 2)
+#define MSR_IA32_PSR_L2_MASK(n) (0x00000d10 + (n))
/* Intel Model 6 */
#define MSR_P6_PERFCTR(n) (0x000000c1 + (n))
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index ae52d85..58b9528 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -23,6 +23,7 @@
/* Resource Type Enumeration */
#define PSR_RESOURCE_TYPE_L3 0x2
+#define PSR_RESOURCE_TYPE_L2 0x4
/* L3 Monitoring Features */
#define PSR_CMT_L3_OCCUPANCY 0x1
@@ -56,6 +57,7 @@ enum cbm_type {
PSR_CBM_TYPE_L3,
PSR_CBM_TYPE_L3_CODE,
PSR_CBM_TYPE_L3_DATA,
+ PSR_CBM_TYPE_L2,
};
extern struct psr_cmt *psr_cmt;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 17/23] x86: L2 CAT: implement get hw info flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (15 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 18/23] x86: L2 CAT: implement get value flow Yi Sun
` (5 subsequent siblings)
22 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L2 CAT get HW info flow and interface in sysctl.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v10:
- modify macro name according to previous patch change.
(suggested by Jan Beulich)
- modify commit message.
v9:
- reuse 'cat_get_feat_info' for L2 CAT to reduce redundant codes.
(suggested by Roger Pau)
- modify sysctl implementation of L2 CAT to input data[3] to use
'cat_get_feat_info'.
(suggested by Roger Pau)
- modify macros names to newly defined ones.
(suggested by Jan Beulich)
- remove 'l2_info' to reuse 'l3_info'.
(suggested by Jan Beulich)
- modify macro name according to previous patch change.
(suggested by Jan Beulich)
v5:
- rename 'dat[]' to 'data[]'
(suggested by Jan Beulich)
- remove type check in callback function.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 5 +++++
xen/arch/x86/sysctl.c | 23 +++++++++++++++++++++--
xen/include/public/sysctl.h | 1 +
3 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 75ebb39..5e07d04 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -231,6 +231,10 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(
feat_type = PSR_SOCKET_L3_CDP;
break;
+ case PSR_CBM_TYPE_L2:
+ feat_type = PSR_SOCKET_L2_CAT;
+ break;
+
default:
ASSERT_UNREACHABLE();
}
@@ -375,6 +379,7 @@ static struct feat_props l3_cdp_props = {
/* L2 CAT props */
static struct feat_props l2_cat_props = {
.cos_num = 1,
+ .get_feat_info = cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index c23270d..5e91755 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -174,10 +174,10 @@ long arch_do_sysctl(
case XEN_SYSCTL_psr_cat_op:
switch ( sysctl->u.psr_cat_op.cmd )
{
- case XEN_SYSCTL_PSR_CAT_get_l3_info:
- {
uint32_t data[PSR_INFO_ARRAY_SIZE];
+ case XEN_SYSCTL_PSR_CAT_get_l3_info:
+ {
ret = psr_get_info(sysctl->u.psr_cat_op.target,
PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
if ( ret )
@@ -195,6 +195,25 @@ long arch_do_sysctl(
break;
}
+ case XEN_SYSCTL_PSR_CAT_get_l2_info:
+ {
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L2, data, ARRAY_SIZE(data));
+ if ( ret )
+ break;
+
+ sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ data[PSR_INFO_IDX_COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ data[PSR_INFO_IDX_CAT_CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.flags =
+ data[PSR_INFO_IDX_CAT_FLAG];
+
+ if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
+ ret = -EFAULT;
+ break;
+ }
+
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index ee76a66..4c76d3a 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
#define XEN_SYSCTL_PSR_CAT_get_l3_info 0
+#define XEN_SYSCTL_PSR_CAT_get_l2_info 1
struct xen_sysctl_psr_cat_op {
uint32_t cmd; /* IN: XEN_SYSCTL_PSR_CAT_* */
uint32_t target; /* IN */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 18/23] x86: L2 CAT: implement get value flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (16 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 17/23] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-31 9:51 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 19/23] x86: L2 CAT: implement set " Yi Sun
` (4 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L2 CAT get value interface in domctl.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- remove "get_val' assignment because it has been replaced by generic
codes.
(suggested by Jan Beulich)
v10:
- remove cast in domctl.
(suggested by Jan Beulich)
v9:
- reuse 'cat_get_val' for L2 CAT to reduce redundant codes
(suggested by Roger Pau)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v5:
- remove type check in callback function.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
xen/arch/x86/domctl.c | 7 +++++++
xen/include/public/domctl.h | 1 +
2 files changed, 8 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 124a1c6..2970118 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1449,6 +1449,13 @@ long arch_do_domctl(
copyback = 1;
break;
+ case XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM:
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L2);
+ domctl->u.psr_cat_op.data = val32;
+ copyback = 1;
+ break;
+
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 9e3ce21..16c08c0 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1132,6 +1132,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
uint64_t data; /* IN/OUT */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 19/23] x86: L2 CAT: implement set value flow.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (17 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 18/23] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-31 9:52 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 20/23] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
` (3 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L2 CAT set value related callback function
and domctl interface.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- remove 'domctl->u.psr_cat_op.data' check because it has been moved into
'psr_set_val'.
(suggested by Jan Beulich)
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
v10:
- check input data and remove cast in domctl.
(suggested by Jan Beulich)
- remove some hooks assignment due to previous patches changes.
(suggested by Jan Beulich)
- remove cast in 'l2_cat_write_msr'.
(suggested by Jan Beulich)
- remove 'return in 'l2_cat_write_msr'.
(suggested by Jan Beulich)
v9:
- reuse some CAT common functions for L2 CAT to reduce redundant codes.
(suggested by Roger Pau)
- remove parameter 'found' from 'cat_compare_val' and modify the return
values to let caller know if the id is found or not. These things are
done in patch "x86: refactor psr: set value: implement cos finding flow."
(suggested by Roger Pau and Dario Faggioli)
- remove 'get_cos_num' related codes.
(suggested by Jan Beulich)
- modify 'l2_cat_write_msr' according to previous patch change.
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- modify 'l2_cat_write_msr' to 'void'.
v5:
- remove type check in callback function.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- remove 'l2_cat_get_cos_max_from_type'.
(suggested by Jan Beulich)
- rename 'l2_cat_exceeds_cos_max' to 'l2_cat_fits_cos_max'.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
xen/arch/x86/domctl.c | 6 ++++++
xen/arch/x86/psr.c | 6 ++++++
xen/include/public/domctl.h | 1 +
3 files changed, 13 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 2970118..b5a892a 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1428,6 +1428,12 @@ long arch_do_domctl(
PSR_CBM_TYPE_L3_DATA);
break;
+ case XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM:
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L2);
+ break;
+
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
ret = psr_get_val(d, domctl->u.psr_cat_op.target,
&val32, PSR_CBM_TYPE_L3);
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5e07d04..a0afaf2 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -377,9 +377,15 @@ static struct feat_props l3_cdp_props = {
};
/* L2 CAT props */
+static void l2_cat_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
+{
+ wrmsrl(MSR_IA32_PSR_L2_MASK(cos), val);
+}
+
static struct feat_props l2_cat_props = {
.cos_num = 1,
.get_feat_info = cat_get_feat_info,
+ .write_msr = l2_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 16c08c0..a06c6c5 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1132,6 +1132,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM 6
#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 20/23] tools: L2 CAT: support get HW info for L2 CAT.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (18 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 19/23] x86: L2 CAT: implement set " Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-11 12:39 ` Wei Liu
2017-05-03 8:44 ` [PATCH v11 21/23] tools: L2 CAT: support show cbm " Yi Sun
` (2 subsequent siblings)
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements xl/xc changes to support get HW info
for L2 CAT.
'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
info.
Example(on machine which only supports L2 CAT):
Cache Monitoring Technology (CMT):
Enabled : 0
Cache Allocation Technology (CAT): L2
Socket ID : 0
Maximum COS : 3
CBM length : 8
Default CBM : 0xff
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
v11:
- acked by Jan for Hypervisor side changes.
v10:
- change macros names according to previous changes.
(suggested by Jan Beulich)
v9:
- add some cases to handle return error no.
- move xl_cmdimpl.c codes into xl/xl_psr.c.
- change 'l3_info' to 'cat_info' to cover both L3 and L2 CAT.
v6:
- adjust '{' position for 'switch'.
(suggested by Wei Liu)
- modify commit message to remove error log.
(suggested by Dario Faggioli)
v5:
- modify commit message to remove error log.
(suggested by Wei Liu and Jan Beulich)
- replace unnecessary 'return' to 'break'.
(suggested by Wei Liu)
- restore 'libxl_psr_cat_get_l3_info' to keep interface backward compatible
but change codes in it to call new function to get hw info.
(suggested by Wei Liu)
- add 'L2_CBM' into 'psr_cbm_type' because it is interface change which
should be in same patch with new 'LIBXL_HAVE_' macro.
(suggested by Wei Liu)
- addjust logs sentence to make unnecessary error logs not show.
(suggested by Wei Liu and Jan Beulich)
v4:
- create this patch to help reviewers better understand the codes.
---
tools/libxc/include/xenctrl.h | 6 ++---
tools/libxc/xc_psr.c | 39 +++++++++++++++++++++++---------
tools/libxl/libxl.h | 9 ++++++++
tools/libxl/libxl_psr.c | 28 ++++++++++++++++++-----
tools/libxl/libxl_types.idl | 1 +
tools/xl/xl_psr.c | 52 +++++++++++++++++++++++++++++++++----------
xen/arch/x86/sysctl.c | 12 +++++-----
xen/include/public/sysctl.h | 2 +-
8 files changed, 111 insertions(+), 38 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 01f8dfe..3fbcd70 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2468,9 +2468,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
xc_psr_cat_type type, uint32_t target,
uint64_t *data);
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled);
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len,
+ bool *cdp_enabled);
int xc_get_cpu_levelling_caps(xc_interface *xch, uint32_t *caps);
int xc_get_cpu_featureset(xc_interface *xch, uint32_t index,
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 43b3286..84a08c4 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -317,24 +317,41 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
return rc;
}
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled)
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len, bool *cdp_enabled)
{
- int rc;
+ int rc = -1;
DECLARE_SYSCTL;
sysctl.cmd = XEN_SYSCTL_psr_cat_op;
- sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
sysctl.u.psr_cat_op.target = socket;
- rc = xc_sysctl(xch, &sysctl);
- if ( !rc )
+ switch ( lvl )
{
- *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
- *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
- *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
- XEN_SYSCTL_PSR_CAT_L3_CDP;
+ case 2:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l2_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.cat_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.cat_info.cbm_len;
+ *cdp_enabled = false;
+ }
+ break;
+ case 3:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.cat_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.cat_info.cbm_len;
+ *cdp_enabled = sysctl.u.psr_cat_op.u.cat_info.flags &
+ XEN_SYSCTL_PSR_CAT_L3_CDP;
+ }
+ break;
+ default:
+ errno = EOPNOTSUPP;
+ break;
}
return rc;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index a402236..92da2ae 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -916,6 +916,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, const libxl_mac *src);
* If this is defined, the Code and Data Prioritization feature is supported.
*/
#define LIBXL_HAVE_PSR_CDP 1
+
+/*
+ * LIBXL_HAVE_PSR_L2_CAT
+ *
+ * If this is defined, the L2 Cache Allocation Technology feature is supported.
+ */
+#define LIBXL_HAVE_PSR_L2_CAT 1
#endif
/*
@@ -2194,6 +2201,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
* On success, the function returns an array of elements in 'info',
* and the length in 'nr'.
*/
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl);
int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
int *nr);
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr);
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index ec5c79d..f55ba1e 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -91,6 +91,15 @@ static void libxl__psr_cat_log_err_msg(libxl__gc *gc, int err)
case ENXIO:
msg = "Unable to set code or data CBM when CDP is disabled";
break;
+ case EINVAL:
+ msg = "Invalid input or some internal values are not expected";
+ break;
+ case ERANGE:
+ msg = "Socket number is wrong";
+ break;
+ case ENOSPC:
+ msg = "Value array exceeds the range";
+ break;
default:
libxl__psr_log_err_msg(gc, err);
@@ -352,8 +361,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
return rc;
}
-int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
- int *nr)
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl)
{
GC_INIT(ctx);
int rc;
@@ -380,9 +389,8 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
libxl_for_each_set_bit(socketid, socketmap) {
ptr[i].id = socketid;
- if (xc_psr_cat_get_l3_info(ctx->xch, socketid, &ptr[i].cos_max,
- &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
- libxl__psr_cat_log_err_msg(gc, errno);
+ if (xc_psr_cat_get_info(ctx->xch, socketid, lvl, &ptr[i].cos_max,
+ &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
rc = ERROR_FAIL;
free(ptr);
goto out;
@@ -398,6 +406,16 @@ out:
return rc;
}
+int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr)
+{
+ int rc;
+
+ rc = libxl_psr_cat_get_info(ctx, info, nr, 3);
+
+ return rc;
+}
+
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr)
{
int i;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index d970284..2776b6c 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -960,6 +960,7 @@ libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
(1, "L3_CBM"),
(2, "L3_CBM_CODE"),
(3, "L3_CBM_DATA"),
+ (4, "L2_CBM"),
])
libxl_psr_cat_info = Struct("psr_cat_info", [
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index c061b29..271b88f 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -294,21 +294,19 @@ int main_psr_cmt_show(int argc, char **argv)
}
#endif
-#ifdef LIBXL_HAVE_PSR_CAT
-static int psr_cat_hwinfo(void)
+#if defined(LIBXL_HAVE_PSR_CAT) || defined(LIBXL_HAVE_PSR_L2_CAT)
+static int psr_l3_cat_hwinfo(void)
{
- int rc;
- int i, nr;
+ int rc, nr;
+ unsigned int i;
uint32_t l3_cache_size;
libxl_psr_cat_info *info;
- printf("Cache Allocation Technology (CAT):\n");
-
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
- if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (rc)
return rc;
- }
+
+ printf("Cache Allocation Technology (CAT):\n");
for (i = 0; i < nr; i++) {
rc = libxl_psr_cmt_get_l3_cache_size(ctx, info[i].id, &l3_cache_size);
@@ -417,7 +415,7 @@ static int psr_cat_show(uint32_t domid)
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
if (rc) {
fprintf(stderr, "Failed to get cat info\n");
return rc;
@@ -434,6 +432,32 @@ out:
return rc;
}
+static int psr_l2_cat_hwinfo(void)
+{
+ int rc;
+ unsigned int i;
+ int nr;
+ libxl_psr_cat_info *info;
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 2);
+ if (rc)
+ return rc;
+
+ printf("Cache Allocation Technology (CAT): L2\n");
+
+ for (i = 0; i < nr; i++) {
+ /* There is no CMT on L2 cache so far. */
+ printf("%-16s: %u\n", "Socket ID", info[i].id);
+ printf("%-16s: %u\n", "Maximum COS", info[i].cos_max);
+ printf("%-16s: %u\n", "CBM length", info[i].cbm_len);
+ printf("%-16s: %#llx\n", "Default CBM",
+ (1ull << info[i].cbm_len) - 1);
+ }
+
+ libxl_psr_cat_info_list_free(info, nr);
+ return rc;
+}
+
int main_psr_cat_cbm_set(int argc, char **argv)
{
uint32_t domid;
@@ -551,7 +575,11 @@ int main_psr_hwinfo(int argc, char **argv)
ret = psr_cmt_hwinfo();
if (!ret && (all || cat))
- ret = psr_cat_hwinfo();
+ ret = psr_l3_cat_hwinfo();
+
+ /* L2 CAT is independent of CMT and L3 CAT */
+ if (all || cat)
+ ret = psr_l2_cat_hwinfo();
return ret;
}
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 5e91755..14c12b7 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -183,11 +183,11 @@ long arch_do_sysctl(
if ( ret )
break;
- sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ sysctl->u.psr_cat_op.u.cat_info.cos_max =
data[PSR_INFO_IDX_COS_MAX];
- sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ sysctl->u.psr_cat_op.u.cat_info.cbm_len =
data[PSR_INFO_IDX_CAT_CBM_LEN];
- sysctl->u.psr_cat_op.u.l3_info.flags =
+ sysctl->u.psr_cat_op.u.cat_info.flags =
data[PSR_INFO_IDX_CAT_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
@@ -202,11 +202,11 @@ long arch_do_sysctl(
if ( ret )
break;
- sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ sysctl->u.psr_cat_op.u.cat_info.cos_max =
data[PSR_INFO_IDX_COS_MAX];
- sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ sysctl->u.psr_cat_op.u.cat_info.cbm_len =
data[PSR_INFO_IDX_CAT_CBM_LEN];
- sysctl->u.psr_cat_op.u.l3_info.flags =
+ sysctl->u.psr_cat_op.u.cat_info.flags =
data[PSR_INFO_IDX_CAT_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 4c76d3a..9e51af6 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -754,7 +754,7 @@ struct xen_sysctl_psr_cat_op {
uint32_t cos_max; /* OUT: Maximum COS */
#define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
uint32_t flags; /* OUT: CAT flags */
- } l3_info;
+ } cat_info;
} u;
};
typedef struct xen_sysctl_psr_cat_op xen_sysctl_psr_cat_op_t;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 21/23] tools: L2 CAT: support show cbm for L2 CAT.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (19 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 20/23] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-11 12:40 ` Wei Liu
2017-05-03 8:44 ` [PATCH v11 22/23] tools: L2 CAT: support set " Yi Sun
2017-05-03 8:44 ` [PATCH v11 23/23] docs: add L2 CAT description in docs Yi Sun
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements changes in xl/xc changes to support
showing CBM of L2 CAT.
The new level option is introduced to original CAT showing
command in order to show CBM for specified level CAT.
- 'xl psr-cat-show' is updated to show CBM of a domain
according to input cache level.
Examples:
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
- move xl_cmdimpl.c changes into xl/xl_psr.c.
- move xl_cmdtable.c changes into xl/xl_cmdtable.c.
v6:
- check if input level is correct.
- adjust '{' postion for 'if'.
(suggested by Wei Liu)
v5:
- remove 'L2_CBM' in idl because it has been moved to patch 21:
"tools: L2 CAT: support get HW info for L2 CAT".
(suggested by Wei Liu)
v4:
- create this patch because of codes architecture change.
---
tools/libxc/include/xenctrl.h | 1 +
tools/libxc/xc_psr.c | 3 ++
tools/xl/xl_cmdtable.c | 3 +-
tools/xl/xl_psr.c | 85 +++++++++++++++++++++++++++++--------------
4 files changed, 63 insertions(+), 29 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 3fbcd70..0952f75 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2444,6 +2444,7 @@ enum xc_psr_cat_type {
XC_PSR_CAT_L3_CBM = 1,
XC_PSR_CAT_L3_CBM_CODE = 2,
XC_PSR_CAT_L3_CBM_DATA = 3,
+ XC_PSR_CAT_L2_CBM = 4,
};
typedef enum xc_psr_cat_type xc_psr_cat_type;
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 84a08c4..04f5927 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -299,6 +299,9 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 30eb93c..539cb64 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -557,7 +557,8 @@ struct cmd_spec cmd_table[] = {
{ "psr-cat-show",
&main_psr_cat_show, 0, 1,
"Show Cache Allocation Technology information",
- "<Domain>",
+ "[options] <Domain>",
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
},
#endif
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index 271b88f..575f4a0 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -342,7 +342,7 @@ static void psr_cat_print_one_domain_cbm_type(uint32_t domid, uint32_t socketid,
}
static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
char *domain_name;
@@ -350,27 +350,38 @@ static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
printf("%5d%25s", domid, domain_name);
free(domain_name);
- if (!cdp_enabled) {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM);
- } else {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ switch (lvl) {
+ case 3:
+ if (!cdp_enabled) {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM);
+ } else {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ }
+ break;
+ case 2:
psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ LIBXL_PSR_CBM_TYPE_L2_CBM);
+ break;
+ default:
+ printf("Input lvl %d is wrong!", lvl);
+ break;
}
printf("\n");
}
static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
int i, nr_domains;
libxl_dominfo *list;
if (domid != INVALID_DOMID) {
- psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled, lvl);
return 0;
}
@@ -380,49 +391,59 @@ static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
}
for (i = 0; i < nr_domains; i++)
- psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled, lvl);
libxl_dominfo_list_free(list, nr_domains);
return 0;
}
-static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
+static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info,
+ unsigned int lvl)
{
int rc;
uint32_t l3_cache_size;
- rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
- if (rc) {
- fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
- info->id);
- return -1;
+ printf("%-16s: %u\n", "Socket ID", info->id);
+
+ /* So far, CMT only supports L3 cache. */
+ if (lvl == 3) {
+ rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
+ if (rc) {
+ fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
+ info->id);
+ return -1;
+ }
+ printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
}
- printf("%-16s: %u\n", "Socket ID", info->id);
- printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
printf("%-16s: %#llx\n", "Default CBM", (1ull << info->cbm_len) - 1);
if (info->cdp_enabled)
printf("%5s%25s%16s%16s\n", "ID", "NAME", "CBM (code)", "CBM (data)");
else
printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
- return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled);
+ return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled, lvl);
}
-static int psr_cat_show(uint32_t domid)
+static int psr_cat_show(uint32_t domid, unsigned int lvl)
{
int i, nr;
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (lvl != 2 && lvl != 3) {
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
+ }
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, lvl);
if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ fprintf(stderr, "Failed to get %s cat info\n", (lvl == 3)?"L3":"L2");
return rc;
}
for (i = 0; i < nr; i++) {
- rc = psr_cat_print_socket(domid, info + i);
+ rc = psr_cat_print_socket(domid, info + i, lvl);
if (rc)
goto out;
}
@@ -533,11 +554,19 @@ int main_psr_cat_cbm_set(int argc, char **argv)
int main_psr_cat_show(int argc, char **argv)
{
- int opt;
+ int opt = 0;
uint32_t domid;
+ unsigned int lvl = 3;
- SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cat-show", 0) {
- /* No options */
+ static struct option opts[] = {
+ {"level", 1, 0, 'l'},
+ COMMON_LONG_OPTS
+ };
+
+ SWITCH_FOREACH_OPT(opt, "l:", opts, "psr-cat-show", 0) {
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
if (optind >= argc)
@@ -549,7 +578,7 @@ int main_psr_cat_show(int argc, char **argv)
return 2;
}
- return psr_cat_show(domid);
+ return psr_cat_show(domid, lvl);
}
int main_psr_hwinfo(int argc, char **argv)
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 22/23] tools: L2 CAT: support set cbm for L2 CAT.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (20 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 21/23] tools: L2 CAT: support show cbm " Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
2017-05-11 12:43 ` Wei Liu
2017-05-03 8:44 ` [PATCH v11 23/23] docs: add L2 CAT description in docs Yi Sun
22 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the xl/xc changes to support set CBM
for L2 CAT.
The new level option is introduced to original CAT setting
command in order to set CBM for specified level CAT.
- 'xl psr-cat-set' is updated to set cache capacity bitmasks(CBM)
for a domain according to input cache level.
root@:~$ xl psr-cat-set -l2 1 0x7f
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v11:
- revert change in v9 to handle the case to set both CODE and DATA for CDP
at same time. This must be handled in hypervisor.
v10:
- fix comments.
(suggested by Wei Liu)
v9:
- handle the case to set both CODE and DATA for CDP at same time.
For such case, user does not input '-c' or '-d' to set CDP cbm.
- move xl_cmdimpl.c changes into xl/xl_psr.c.
- move xl_cmdtable.c changes into xl/xl_cmdtable.c.
v6:
- rename 'psr-cat-cbm-set' to 'psr-cat-set'.
(suggested by Kevin Tian)
- return 'EXIT_FAILURE' for error case.
(suggested by Dario Faggioli)
- print error info when input level is wrong.
v4:
- create this patch because of codes architecture change.
---
tools/libxc/xc_psr.c | 3 +++
tools/xl/xl_cmdtable.c | 3 ++-
tools/xl/xl_psr.c | 33 +++++++++++++++++++++++----------
3 files changed, 28 insertions(+), 11 deletions(-)
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 04f5927..039b920 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -266,6 +266,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 539cb64..2c71a9f 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -546,11 +546,12 @@ struct cmd_spec cmd_table[] = {
},
#endif
#ifdef LIBXL_HAVE_PSR_CAT
- { "psr-cat-cbm-set",
+ { "psr-cat-set",
&main_psr_cat_cbm_set, 0, 1,
"Set cache capacity bitmasks(CBM) for a domain",
"[options] <Domain> <CBM>",
"-s <socket> Specify the socket to process, otherwise all sockets are processed\n"
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
"-c Set code CBM if CDP is supported\n"
"-d Set data CBM if CDP is supported\n"
},
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index 575f4a0..7309d4f 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -490,19 +490,21 @@ int main_psr_cat_cbm_set(int argc, char **argv)
char *value;
libxl_string_list socket_list;
unsigned long start, end;
- int i, j, len;
+ unsigned int i, j, len;
+ unsigned int lvl = 3;
static struct option opts[] = {
{"socket", 1, 0, 's'},
{"data", 0, 0, 'd'},
{"code", 0, 0, 'c'},
+ {"level", 1, 0, 'l'},
COMMON_LONG_OPTS
};
libxl_socket_bitmap_alloc(ctx, &target_map, 0);
libxl_bitmap_set_none(&target_map);
- SWITCH_FOREACH_OPT(opt, "s:cd", opts, "psr-cat-cbm-set", 2) {
+ SWITCH_FOREACH_OPT(opt, "s:l:cd", opts, "psr-cat-set", 2) {
case 's':
trim(isspace, optarg, &value);
split_string_into_string_list(value, ",", &socket_list);
@@ -522,24 +524,35 @@ int main_psr_cat_cbm_set(int argc, char **argv)
case 'c':
opt_code = 1;
break;
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
- if (opt_data && opt_code) {
- fprintf(stderr, "Cannot handle -c and -d at the same time\n");
- return -1;
- } else if (opt_data) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
- } else if (opt_code) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ if (lvl == 2)
+ type = LIBXL_PSR_CBM_TYPE_L2_CBM;
+ else if (lvl == 3) {
+ if (opt_data && opt_code) {
+ fprintf(stderr, "Cannot handle -c and -d at the same time\n");
+ return EXIT_FAILURE;
+ } else if (opt_data) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
+ } else if (opt_code) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ } else {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ }
} else {
type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
}
if (libxl_bitmap_is_empty(&target_map))
libxl_bitmap_set_any(&target_map);
if (argc != optind + 2) {
- help("psr-cat-cbm-set");
+ help("psr-cat-set");
return 2;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* [PATCH v11 23/23] docs: add L2 CAT description in docs.
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (21 preceding siblings ...)
2017-05-03 8:44 ` [PATCH v11 22/23] tools: L2 CAT: support set " Yi Sun
@ 2017-05-03 8:44 ` Yi Sun
22 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-05-03 8:44 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch adds L2 CAT description in related documents.
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
docs/man/xl.pod.1.in | 25 ++++++++++++++++++++++---
docs/misc/xl-psr.markdown | 18 ++++++++++++------
2 files changed, 34 insertions(+), 9 deletions(-)
diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 78bf884..db951c0 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -1716,6 +1716,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
a domain is attached to the monitoring service, monitoring data can be shown
for any of these monitoring types.
+There is no cache monitoring and memory bandwidth monitoring on L2 cache so
+far.
+
=over 4
=item B<psr-cmt-attach> [I<domain-id>]
@@ -1740,7 +1743,7 @@ monitor types are:
Intel Broadwell and later server platforms offer capabilities to configure and
make use of the Cache Allocation Technology (CAT) mechanisms, which enable more
-cache resources (i.e. L3 cache) to be made available for high priority
+cache resources (i.e. L3/L2 cache) to be made available for high priority
applications. In the Xen implementation, CAT is used to control cache allocation
on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
(CBM) for the domain.
@@ -1750,7 +1753,7 @@ Intel Broadwell and later server platforms also offer Code/Data Prioritization
applications. CDP is used on a per VM basis in the Xen implementation. To
specify code or data CBM for the domain, CDP feature must be enabled and CBM
type options need to be specified when setting CBM, and the type options (code
-and data) are mutually exclusive.
+and data) are mutually exclusive. There is no CDP support on L2 so far.
=over 4
@@ -1767,6 +1770,11 @@ B<OPTIONS>
Specify the socket to process, otherwise all sockets are processed.
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
=item B<-c>, B<--code>
Set code CBM when CDP is enabled.
@@ -1777,10 +1785,21 @@ Set data CBM when CDP is enabled.
=back
-=item B<psr-cat-show> [I<domain-id>]
+=item B<psr-cat-show> [I<OPTIONS>] [I<domain-id>]
Show CAT settings for a certain domain or all domains.
+B<OPTIONS>
+
+=over 4
+
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
+=back
+
=back
=head1 IGNORED FOR COMPATIBILITY WITH XM
diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
index c3c1e8e..04dd957 100644
--- a/docs/misc/xl-psr.markdown
+++ b/docs/misc/xl-psr.markdown
@@ -70,7 +70,7 @@ total-mem-bandwidth instead of cache-occupancy). E.g. after a `xl psr-cmt-attach
Cache Allocation Technology (CAT) is a new feature available on Intel
Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
-partition cache allocation (i.e. L3 cache) based on application priority or
+partition cache allocation (i.e. L3/L2 cache) based on application priority or
Class of Service (COS). Each COS is configured using capacity bitmasks (CBM)
which represent cache capacity and indicate the degree of overlap and
isolation between classes. System cache resource is divided into numbers of
@@ -107,7 +107,7 @@ System CAT information such as maximum COS and CBM length can be obtained by:
The simplest way to change a domain's CBM from its default is running:
-`xl psr-cat-cbm-set [OPTIONS] <domid> <cbm>`
+`xl psr-cat-set [OPTIONS] <domid> <cbm>`
where cbm is a number to represent the corresponding cache subset can be used.
A cbm is valid only when:
@@ -119,13 +119,19 @@ A cbm is valid only when:
In a multi-socket system, the same cbm will be set on each socket by default.
Per socket cbm can be specified with the `--socket SOCKET` option.
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
+
Setting the CBM may not be successful if insufficient COS is available. In
such case unused COS(es) may be freed by setting CBM of all related domains to
its default value(all-ones).
Per domain CBM settings can be shown by:
-`xl psr-cat-show`
+`xl psr-cat-show [OPTIONS] <domid>`
+
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
## Code and Data Prioritization (CDP)
@@ -172,13 +178,13 @@ options is invalid.
Example:
Setting code CBM for a domain:
-`xl psr-cat-cbm-set -c <domid> <cbm>`
+`xl psr-cat-set -c <domid> <cbm>`
Setting data CBM for a domain:
-`xl psr-cat-cbm-set -d <domid> <cbm>`
+`xl psr-cat-set -d <domid> <cbm>`
Setting the same code and data CBM for a domain:
-`xl psr-cat-cbm-set <domid> <cbm>`
+`xl psr-cat-set <domid> <cbm>`
## Reference
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 74+ messages in thread
* Re: [PATCH v11 20/23] tools: L2 CAT: support get HW info for L2 CAT.
2017-05-03 8:44 ` [PATCH v11 20/23] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
@ 2017-05-11 12:39 ` Wei Liu
0 siblings, 0 replies; 74+ messages in thread
From: Wei Liu @ 2017-05-11 12:39 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel, roger.pau
On Wed, May 03, 2017 at 04:44:20PM +0800, Yi Sun wrote:
> This patch implements xl/xc changes to support get HW info
> for L2 CAT.
>
> 'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
> info.
>
> Example(on machine which only supports L2 CAT):
> Cache Monitoring Technology (CMT):
> Enabled : 0
> Cache Allocation Technology (CAT): L2
> Socket ID : 0
> Maximum COS : 3
> CBM length : 8
> Default CBM : 0xff
>
> Signed-off-by: He Chen <he.chen@linux.intel.com>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 21/23] tools: L2 CAT: support show cbm for L2 CAT.
2017-05-03 8:44 ` [PATCH v11 21/23] tools: L2 CAT: support show cbm " Yi Sun
@ 2017-05-11 12:40 ` Wei Liu
0 siblings, 0 replies; 74+ messages in thread
From: Wei Liu @ 2017-05-11 12:40 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel, roger.pau
On Wed, May 03, 2017 at 04:44:21PM +0800, Yi Sun wrote:
> This patch implements changes in xl/xc changes to support
> showing CBM of L2 CAT.
>
> The new level option is introduced to original CAT showing
> command in order to show CBM for specified level CAT.
> - 'xl psr-cat-show' is updated to show CBM of a domain
> according to input cache level.
>
> Examples:
> root@:~$ xl psr-cat-show -l2 1
> Socket ID : 0
> Default CBM : 0xff
> ID NAME CBM
> 1 ubuntu14 0x7f
>
> Signed-off-by: He Chen <he.chen@linux.intel.com>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
Acked-by: Wei Liu <wei.liu2@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 22/23] tools: L2 CAT: support set cbm for L2 CAT.
2017-05-03 8:44 ` [PATCH v11 22/23] tools: L2 CAT: support set " Yi Sun
@ 2017-05-11 12:43 ` Wei Liu
0 siblings, 0 replies; 74+ messages in thread
From: Wei Liu @ 2017-05-11 12:43 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel, roger.pau
On Wed, May 03, 2017 at 04:44:22PM +0800, Yi Sun wrote:
> This patch implements the xl/xc changes to support set CBM
> for L2 CAT.
>
> The new level option is introduced to original CAT setting
> command in order to set CBM for specified level CAT.
> - 'xl psr-cat-set' is updated to set cache capacity bitmasks(CBM)
> for a domain according to input cache level.
>
> root@:~$ xl psr-cat-set -l2 1 0x7f
>
> root@:~$ xl psr-cat-show -l2 1
> Socket ID : 0
> Default CBM : 0xff
> ID NAME CBM
> 1 ubuntu14 0x7f
>
> Signed-off-by: He Chen <he.chen@linux.intel.com>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-05-03 8:44 ` [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
@ 2017-05-30 13:05 ` Jan Beulich
2017-05-31 2:44 ` Yi Sun
2017-05-31 9:36 ` Jan Beulich
1 sibling, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-30 13:05 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> +/*
> + * PSR features are managed per socket. Below structure defines the members
> + * used to manage these features.
> + * ref_lock - A lock to protect cos_ref.
> + * features - A feature node array used to manage all features enabled.
> + * cos_ref - A reference count array to record how many domains are using the
> + * COS ID. Every entry of cos_ref corresponds to one COS ID.
> + */
> +struct psr_socket_info {
> + bool feat_init;
The comment above lacks a line for this field.
> +static void free_socket_resources(unsigned int socket)
> +{
> + unsigned int i;
> + struct psr_socket_info *info = socket_info + socket;
> +
> + if ( !info )
> + return;
> +
> + /*
> + * Free resources of features. The global feature object, e.g. feat_l3_cat,
> + * may not be freed here if it is not added into array. It is simply being
> + * kept until the next CPU online attempt.
> + */
> + for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
> + {
> + xfree(info->features[i]);
> + info->features[i] = NULL;
> + }
If you iterate over arrays, please use ARRAY_SIZE() as the loop
boundary (and without knowing whether there are similar issues
elsewhere in the series, please consider the comment given for
all of it, just like any other comments usually are meant to apply
to similar issues elsewhere).
> +static void __init init_psr(void)
> +{
> + if ( opt_cos_max < 1 )
> + {
> + printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
> + return;
> + }
> +
> + socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
> +
> + if ( !socket_info )
> + {
> + printk(XENLOG_INFO "Failed to alloc socket_info!\n");
XENLOG_WARNING at least.
> static void psr_cpu_init(void)
> {
> + struct psr_socket_info *info;
> + unsigned int socket;
> + unsigned int cpu = smp_processor_id();
> + struct feat_node *feat;
> + struct cpuid_leaf regs;
> +
> + if ( !psr_alloc_feat_enabled() || !boot_cpu_has(X86_FEATURE_PQE) )
> + goto assoc_init;
> +
> + if ( boot_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
> + {
> + setup_clear_cpu_cap(X86_FEATURE_PQE);
> + goto assoc_init;
> + }
> +
> + socket = cpu_to_socket(cpu);
> + info = socket_info + socket;
> + if ( info->feat_init )
> + goto assoc_init;
> +
> + spin_lock_init(&info->ref_lock);
> +
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> + if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> + {
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> +
> + feat = feat_l3_cat;
> + feat_l3_cat = NULL;
> + l3_cat_props.type[0] = PSR_CBM_TYPE_L3;
Why is this not a static initializer? If it was, the whole l3_cat_props
could be const.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows.
2017-05-03 8:44 ` [PATCH v11 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
@ 2017-05-30 13:26 ` Jan Beulich
2017-05-31 6:37 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-30 13:26 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> +static unsigned int get_max_cos_max(const struct psr_socket_info *info)
> +{
> + unsigned int cos_max = 0, i;
> +
> + for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
> + {
> + const struct feat_node *feat = info->features[i];
> + if ( !feat )
Blank line between declaration(s) and statement(s) please.
> + continue;
> +
> + cos_max = max(feat->cos_max, cos_max);
And you're likely better off inverting the condition and dropping
the "continue".
> +static void psr_assoc_cos(uint64_t *reg, unsigned int cos,
> + uint64_t cos_mask)
> +{
> + *reg = (*reg & ~cos_mask) |
> + (((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
> +}
Indirection is normally only needed if a function needs to return
more than one value. Is there a reason you can't have this one
return the computed result?
> @@ -376,6 +412,14 @@ void psr_ctxt_switch_to(struct domain *d)
> if ( psr_cmt_enabled() )
> psr_assoc_rmid(®, d->arch.psr_rmid);
>
> + /* IDLE domain's 'psr_cos_ids' is NULL so we set default value for it. */
> + if ( psra->cos_mask )
> + psr_assoc_cos(®,
> + d->arch.psr_cos_ids ?
> + d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
> + 0,
While this doesn't really conflict with our coding style, it makes
reading harder than necessary. Please use either
if ( psra->cos_mask )
psr_assoc_cos(®,
(d->arch.psr_cos_ids ?
d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
0),
or
if ( psra->cos_mask )
psr_assoc_cos(®,
d->arch.psr_cos_ids
? d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())]
: 0,
to allow immediate recognition that it is a single argument's expression
that spans three lines.
As to the idle domain aspect - is there a strict need to write a new
value for the idle domain at all? I.e. can't you just skip the write in
that case, knowing you'll write a proper value anyway once the
next non-idle vCPU gets scheduled here? Which then raises the
question on d->arch.psr_cos_ids being NULL - is that strictly only
possible for the idle domain, or are there also other cases? This
determines how the if() condition should be re-written ...
> @@ -401,14 +445,37 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> return 0;
> }
>
> -int psr_domain_init(struct domain *d)
> +/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> +static void psr_free_cos(struct domain *d)
> +{
> + xfree(d->arch.psr_cos_ids);
> + d->arch.psr_cos_ids = NULL;
> +}
> +
> +static int psr_alloc_cos(struct domain *d)
> {
> + d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
> + if ( !d->arch.psr_cos_ids )
> + return -ENOMEM;
> +
> return 0;
> }
>
> +int psr_domain_init(struct domain *d)
> +{
> + /* Init to success value */
> + int ret = 0;
> +
> + if ( psr_alloc_feat_enabled() )
> + ret = psr_alloc_cos(d);
> +
> + return ret;
> +}
Along the lines of the above - do we really need to fail domain
creation if we can't alloc psr_cos_ids? Granted there'll be other
allocation failures, but from an abstract pov this is an optional
feature, and hence the domain could do fine without.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 06/23] x86: refactor psr: L3 CAT: implement get hw info flow.
2017-05-03 8:44 ` [PATCH v11 06/23] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
@ 2017-05-30 13:51 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-05-30 13:51 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> +int psr_get_info(unsigned int socket, enum cbm_type type,
> + uint32_t data[], unsigned int array_len)
> +{
> + const struct psr_socket_info *info = get_socket_info(socket);
> + const struct feat_node *feat;
> + enum psr_feat_type feat_type;
> +
> + ASSERT(data);
> +
> + if ( IS_ERR(info) )
> + return PTR_ERR(info);
> +
> + feat_type = psr_cbm_type_to_feat_type(type);
> + if ( feat_type >= ARRAY_SIZE(info->features) )
> + return -ENOENT;
> +
> + feat = info->features[feat_type];
> + if ( !feat || !feat_props[feat_type] )
> + return -ENOENT;
Isn't it a bug if feat_props[feat_type] is NULL when feat here is
non-NULL? If so, perhaps the better approach would be to have
separate checks:
if ( !feat )
return -ENOENT;
if ( !feat_props[feat_type] )
{
ASSERT_UNREACHABLE();
return -ENOENT;
}
at once documenting this interdependency.
With or without this change
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-05-03 8:44 ` [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
@ 2017-05-30 14:05 ` Jan Beulich
2017-05-31 7:30 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-30 14:05 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -476,23 +476,34 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
> return socket_info + socket;
> }
>
> +static struct feat_node *psr_get_feat_and_type(unsigned int socket,
> + enum cbm_type type,
> + enum psr_feat_type *feat_type)
> +{
> + const struct psr_socket_info *info = get_socket_info(socket);
> +
> + if ( IS_ERR(info) )
> + return ERR_PTR(PTR_ERR(info));
> +
> + *feat_type = psr_cbm_type_to_feat_type(type);
> + if ( *feat_type >= ARRAY_SIZE(info->features) )
> + return NULL;
Note how this return is not being taken care of by ...
> + return info->features[*feat_type];
> +}
> +
> int psr_get_info(unsigned int socket, enum cbm_type type,
> uint32_t data[], unsigned int array_len)
> {
> - const struct psr_socket_info *info = get_socket_info(socket);
> const struct feat_node *feat;
> enum psr_feat_type feat_type;
>
> ASSERT(data);
>
> - if ( IS_ERR(info) )
> - return PTR_ERR(info);
> -
> - feat_type = psr_cbm_type_to_feat_type(type);
> - if ( feat_type >= ARRAY_SIZE(info->features) )
> - return -ENOENT;
> + feat = psr_get_feat_and_type(socket, type, &feat_type);
> + if ( IS_ERR(feat) )
> + return PTR_ERR(feat);
... the check here. I think you want to alter the return above.
And of course I wonder why you replace code here that was
only introduced one or two patches earlier. Perhaps that earlier
patch should do things this way right away?
> - feat = info->features[feat_type];
> if ( !feat || !feat_props[feat_type] )
> return -ENOENT;
Afaics you need feat_type here only to get at the right feat_props[]
entry. If that's the case also for future callers of
psr_get_feat_and_type(), perhaps it would be better for it to
provide those two instead of the intermediate type? Of course
that would imply renaming the function. (This change would
clearly benefit the readability of psr_get_val() below.)
> @@ -502,9 +513,38 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> return -EINVAL;
> }
>
> -int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> - uint64_t *cbm, enum cbm_type type)
> +int psr_get_val(struct domain *d, unsigned int socket,
> + uint32_t *val, enum cbm_type type)
> {
> + const struct feat_node *feat;
> + enum psr_feat_type feat_type;
> + unsigned int cos, i;
> +
> + ASSERT(val);
> +
> + feat = psr_get_feat_and_type(socket, type, &feat_type);
> + if ( IS_ERR(feat) )
> + return PTR_ERR(feat);
> +
> + if ( !feat || !feat_props[feat_type] )
> + return -ENOENT;
> +
> + cos = d->arch.psr_cos_ids[socket];
> + /*
> + * If input cos exceeds current feature's cos_max, we should return its
> + * default value which is stored in cos 0. This case only happens
> + * when more than two features enabled concurrently and at least one
> + * features's cos_max is bigger than others. When a domain's working cos
> + * id is bigger than some features' cos_max, HW automatically works as
> + * default value for those features which cos_max is smaller.
> + */
> + if ( cos > feat->cos_max )
> + cos = 0;
> +
> + for ( i = 0; i < feat_props[feat_type]->cos_num; i++ )
> + if ( type == feat_props[feat_type]->type[i] )
> + *val = feat->cos_reg_val[cos * feat_props[feat_type]->cos_num + i];
> +
> return 0;
> }
Do you really want to return success here even if you didn't write
to *val? With the way the callers are coded, this is an (at least
latent) information leak at present.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-05-03 8:44 ` [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
@ 2017-05-30 14:32 ` Jan Beulich
2017-06-01 10:00 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-30 14:32 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -118,11 +118,13 @@ static const struct feat_props {
> * COS ID. Every entry of cos_ref corresponds to one COS ID.
> */
> struct psr_socket_info {
> - bool feat_init;
> - spinlock_t ref_lock;
> /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
> struct feat_node *features[PSR_SOCKET_FEAT_NUM];
> + bool feat_init;
> unsigned int cos_ref[MAX_COS_REG_CNT];
> + spinlock_t ref_lock;
This shuffling of fields seems unmotivated and is not being explained
in the description.
> @@ -178,6 +180,10 @@ static void free_socket_resources(unsigned int socket)
> }
>
> info->feat_init = false;
> +
> + memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
> +
> + memset(info->dom_ids, 0, ((DOMID_IDLE + 1) + 7) / 8);
bitmap_clear()
> @@ -449,11 +455,19 @@ void psr_ctxt_switch_to(struct domain *d)
>
> /* IDLE domain's 'psr_cos_ids' is NULL so we set default value for it. */
> if ( psra->cos_mask )
> - psr_assoc_cos(®,
> - d->arch.psr_cos_ids ?
> - d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
> - 0,
> - psra->cos_mask);
> + {
> + unsigned int socket = cpu_to_socket(smp_processor_id());
> + struct psr_socket_info *info = socket_info + socket;
> + unsigned int cos = 0;
> +
> + if ( d->arch.psr_cos_ids )
> + cos = d->arch.psr_cos_ids[socket];
> +
> + if ( unlikely(!test_bit(d->domain_id, info->dom_ids)) )
> + cos = 0;
I think a brief comment here would be helpful. I also think the two
if()-s would better be combined (after all to initialize cos to zero
above, so you simply need to avoid overwriting it in the first if().
> @@ -529,6 +543,10 @@ int psr_get_val(struct domain *d, unsigned int socket,
> if ( !feat || !feat_props[feat_type] )
> return -ENOENT;
>
> + if ( !test_bit(d->domain_id, socket_info[socket].dom_ids) &&
> + d->arch.psr_cos_ids[socket] )
> + d->arch.psr_cos_ids[socket] = 0;
What is the right side of the && good for?
Also the latest here it is clear that "dom_ids" isn't the best choice
for a name. What about "dom_set" or "domain_set"?
> +/* The whole set process is protected by domctl_lock. */
This needs to be re-considered: We intend to incrementally
reduce code ranges guarded by the domctl lock, and a good
first step might be to stop introducing further dependencies on
it in individual handlers. So the main question is: Do you need a
global lock here at all, or would a per-domain one suffice? In
the former case I think you should (re?)introduce your own,
while in the latter case you could probably use domain_lock(),
but since you don't require other per-domain activities to be
synchronized, having your own private would perhaps be
even better.
> +int psr_set_val(struct domain *d, unsigned int socket,
> + uint64_t new_val, enum cbm_type type)
> +{
> + unsigned int old_cos;
> + int cos, ret;
> + unsigned int *ref;
> + uint32_t *val_array, val;
> + struct psr_socket_info *info = get_socket_info(socket);
> + unsigned int array_len;
> + enum psr_feat_type feat_type;
> +
> + if ( IS_ERR(info) )
> + return PTR_ERR(info);
> +
> + if ( new_val != (uint32_t)new_val )
> + return -EINVAL;
> +
> + val = new_val;
Please switch this and the prior if(), using val instead of the cast
expression there.
> + feat_type = psr_cbm_type_to_feat_type(type);
> + if ( feat_type >= ARRAY_SIZE(info->features) ||
> + !info->features[feat_type] )
> + return -ENOENT;
Without seeing the code inside the functions you pass feat_type
to below it's not really clear whether you wouldn't better use
what is currently named psr_get_feat_and_type() here.
> + free_array:
> + xfree(val_array);
> + return ret;
> +
> + unlock_free_array:
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return ret;
> +}
I'm sure I've said so before - please don't duplicate error paths like
this. Here it's still easy to see all is fine, but what if each path gets
two or three more thing added. Please chain them together via goto.
> /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> static void psr_free_cos(struct domain *d)
> {
> + unsigned int socket, cos;
> +
> + ASSERT(socket_info);
> +
> + if ( !d->arch.psr_cos_ids )
> + return;
> +
> + /* Domain is destroied so its cos_ref should be decreased. */
destroyed
> + for ( socket = 0; socket < nr_sockets; socket++ )
> + {
> + struct psr_socket_info *info;
> +
> + /* cos 0 is default one which does not need be handled. */
> + cos = d->arch.psr_cos_ids[socket];
> + if ( cos == 0 )
> + continue;
Does this "doesn't need to be handled" even extend to ...
> + info = socket_info + socket;
> + spin_lock(&info->ref_lock);
> + ASSERT(info->cos_ref[cos]);
> + info->cos_ref[cos]--;
> + spin_unlock(&info->ref_lock);
> +
> + clear_bit(d->domain_id, info->dom_ids);
... this last part?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array.
2017-05-03 8:44 ` [PATCH v11 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
@ 2017-05-30 15:17 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-05-30 15:17 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> Only can one COS ID be used by one domain at one time. That means all enabled
> features' COS registers at this COS ID are valid for this domain at that
> time.
>
> When user updates a feature's value, we need make sure all other features'
> values are not affected. So, we firstly need gather an array which contains
> all features current values and replace the setting feature's value in array
> to new value.
>
> Then, we can try to find if there is a COS ID on which all features' COS
> registers values are same as the array. If we can find, we just use this COS
> ID. If fail to find, we need pick a new COS ID.
>
> This patch implements value array assembling flow.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
No specific comments on this one, but comments given on earlier
ones also apply here or - afaict - make changes necessary.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow.
2017-05-03 8:44 ` [PATCH v11 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
@ 2017-05-30 15:23 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-05-30 15:23 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> static int find_cos(const uint32_t val[], unsigned int array_len,
> enum psr_feat_type feat_type,
> const struct psr_socket_info *info)
> {
> + unsigned int cos, i;
> + const unsigned int *ref = info->cos_ref;
> + const struct feat_node *feat;
> + unsigned int cos_max;
Please can you declare all the "unsigned int"-s on one line? (i, btw,
should go into the for scope.)
Looks good otherwise, but again earlier comments apply.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-05-03 8:44 ` [PATCH v11 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
@ 2017-05-30 15:35 ` Jan Beulich
2017-06-05 8:10 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-30 15:35 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> +struct cos_write_info
> +{
> + unsigned int cos;
> + struct feat_node *feature;
> + uint32_t *val;
> + enum psr_feat_type feat_type;
> +};
> +
> +static void do_write_psr_msrs(void *data)
> +{
> + struct cos_write_info *info = data;
> + unsigned int cos = info->cos;
> + struct feat_node *feat = info->feature;
> + const struct feat_props *props = feat_props[info->feat_type];
> + unsigned int i;
> +
> + for ( i = 0; i < props->cos_num; i++ )
> + {
> + if ( feat->cos_reg_val[cos * props->cos_num + i] != info->val[i] )
> + {
> + feat->cos_reg_val[cos * props->cos_num + i] = info->val[i];
> + props->write_msr(cos, info->val[i], props->type[i]);
> + }
> + }
> +}
Again you're passing feat_type here only to get at props. Why
not pass props right away? Also I think it would make sense to
pull props->cos_num into a local variable.
> static int write_psr_msrs(unsigned int socket, unsigned int cos,
> uint32_t val[], unsigned int array_len,
> enum psr_feat_type feat_type)
> {
> - return -ENOENT;
> + unsigned int i;
> + struct psr_socket_info *info = get_socket_info(socket);
> + struct cos_write_info data =
> + {
> + .cos = cos,
> + .feature = info->features[feat_type],
> + .feat_type = feat_type,
> + };
> +
> + if ( cos > info->features[feat_type]->cos_max )
> + return -EINVAL;
> +
> + /* Skip to the feature's value head. */
> + for ( i = 0; i < feat_type; i++ )
> + {
> + if ( !info->features[i] )
> + continue;
This is inconsistent with checks done elsewhere, where you also
check feat_props[feat_type] against NULL. I've made a comment
regarding whether both checks are wanted in a uniform or non-
uniform way pretty early in the series. Whatever is selected
should then be used consistently.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-05-30 13:05 ` Jan Beulich
@ 2017-05-31 2:44 ` Yi Sun
2017-05-31 6:02 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-31 2:44 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-30 07:05:18, Jan Beulich wrote:
> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> > +/*
> > + * PSR features are managed per socket. Below structure defines the members
> > + * used to manage these features.
> > + * ref_lock - A lock to protect cos_ref.
> > + * features - A feature node array used to manage all features enabled.
> > + * cos_ref - A reference count array to record how many domains are using the
> > + * COS ID. Every entry of cos_ref corresponds to one COS ID.
> > + */
> > +struct psr_socket_info {
> > + bool feat_init;
>
> The comment above lacks a line for this field.
>
Will add it. Thanks!
> > +static void free_socket_resources(unsigned int socket)
> > +{
> > + unsigned int i;
> > + struct psr_socket_info *info = socket_info + socket;
> > +
> > + if ( !info )
> > + return;
> > +
> > + /*
> > + * Free resources of features. The global feature object, e.g. feat_l3_cat,
> > + * may not be freed here if it is not added into array. It is simply being
> > + * kept until the next CPU online attempt.
> > + */
> > + for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
> > + {
> > + xfree(info->features[i]);
> > + info->features[i] = NULL;
> > + }
>
> If you iterate over arrays, please use ARRAY_SIZE() as the loop
> boundary (and without knowing whether there are similar issues
> elsewhere in the series, please consider the comment given for
> all of it, just like any other comments usually are meant to apply
> to similar issues elsewhere).
>
Ok, will check all loops.
> > +static void __init init_psr(void)
> > +{
> > + if ( opt_cos_max < 1 )
> > + {
> > + printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
> > + return;
> > + }
> > +
> > + socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
> > +
> > + if ( !socket_info )
> > + {
> > + printk(XENLOG_INFO "Failed to alloc socket_info!\n");
>
> XENLOG_WARNING at least.
>
Ok.
> > static void psr_cpu_init(void)
> > {
> > + struct psr_socket_info *info;
> > + unsigned int socket;
> > + unsigned int cpu = smp_processor_id();
> > + struct feat_node *feat;
> > + struct cpuid_leaf regs;
> > +
> > + if ( !psr_alloc_feat_enabled() || !boot_cpu_has(X86_FEATURE_PQE) )
> > + goto assoc_init;
> > +
> > + if ( boot_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
> > + {
> > + setup_clear_cpu_cap(X86_FEATURE_PQE);
> > + goto assoc_init;
> > + }
> > +
> > + socket = cpu_to_socket(cpu);
> > + info = socket_info + socket;
> > + if ( info->feat_init )
> > + goto assoc_init;
> > +
> > + spin_lock_init(&info->ref_lock);
> > +
> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> > + if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> > + {
> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> > +
> > + feat = feat_l3_cat;
> > + feat_l3_cat = NULL;
> > + l3_cat_props.type[0] = PSR_CBM_TYPE_L3;
>
> Why is this not a static initializer? If it was, the whole l3_cat_props
> could be const.
>
Good point. Will modify it.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-05-31 2:44 ` Yi Sun
@ 2017-05-31 6:02 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 6:02 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 31.05.17 at 04:44, <yi.y.sun@linux.intel.com> wrote:
> On 17-05-30 07:05:18, Jan Beulich wrote:
>> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> > +static void free_socket_resources(unsigned int socket)
>> > +{
>> > + unsigned int i;
>> > + struct psr_socket_info *info = socket_info + socket;
>> > +
>> > + if ( !info )
>> > + return;
>> > +
>> > + /*
>> > + * Free resources of features. The global feature object, e.g. feat_l3_cat,
>> > + * may not be freed here if it is not added into array. It is simply being
>> > + * kept until the next CPU online attempt.
>> > + */
>> > + for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
>> > + {
>> > + xfree(info->features[i]);
>> > + info->features[i] = NULL;
>> > + }
>>
>> If you iterate over arrays, please use ARRAY_SIZE() as the loop
>> boundary (and without knowing whether there are similar issues
>> elsewhere in the series, please consider the comment given for
>> all of it, just like any other comments usually are meant to apply
>> to similar issues elsewhere).
>>
> Ok, will check all loops.
And btw, as in a number of cases you're accessing to identically
dimensioned arrays, picking either one for the loop boundary is
fine provided either one of the arrays also uses ARRAY_SIZE() to
set its own size or you have suitable BUILD_BUG_ON()s in such
loops.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows.
2017-05-30 13:26 ` Jan Beulich
@ 2017-05-31 6:37 ` Yi Sun
2017-05-31 6:57 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-31 6:37 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-30 07:26:55, Jan Beulich wrote:
> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> > +static unsigned int get_max_cos_max(const struct psr_socket_info *info)
> > +{
> > + unsigned int cos_max = 0, i;
> > +
> > + for ( i = 0; i < PSR_SOCKET_FEAT_NUM; i++ )
> > + {
> > + const struct feat_node *feat = info->features[i];
> > + if ( !feat )
>
> Blank line between declaration(s) and statement(s) please.
>
> > + continue;
> > +
> > + cos_max = max(feat->cos_max, cos_max);
>
> And you're likely better off inverting the condition and dropping
> the "continue".
>
Will modify the above points.
> > +static void psr_assoc_cos(uint64_t *reg, unsigned int cos,
> > + uint64_t cos_mask)
> > +{
> > + *reg = (*reg & ~cos_mask) |
> > + (((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
> > +}
>
> Indirection is normally only needed if a function needs to return
> more than one value. Is there a reason you can't have this one
> return the computed result?
>
This is inherited from old code. I think it is to avoid indentation issue in
caller below. Anyway, I will modify it according to your comment.
> > @@ -376,6 +412,14 @@ void psr_ctxt_switch_to(struct domain *d)
> > if ( psr_cmt_enabled() )
> > psr_assoc_rmid(®, d->arch.psr_rmid);
> >
> > + /* IDLE domain's 'psr_cos_ids' is NULL so we set default value for it. */
> > + if ( psra->cos_mask )
> > + psr_assoc_cos(®,
> > + d->arch.psr_cos_ids ?
> > + d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
> > + 0,
>
> While this doesn't really conflict with our coding style, it makes
> reading harder than necessary. Please use either
>
> if ( psra->cos_mask )
> psr_assoc_cos(®,
> (d->arch.psr_cos_ids ?
> d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
> 0),
>
> or
>
> if ( psra->cos_mask )
> psr_assoc_cos(®,
> d->arch.psr_cos_ids
> ? d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())]
> : 0,
>
> to allow immediate recognition that it is a single argument's expression
> that spans three lines.
>
Thank you!
> As to the idle domain aspect - is there a strict need to write a new
> value for the idle domain at all? I.e. can't you just skip the write in
> that case, knowing you'll write a proper value anyway once the
> next non-idle vCPU gets scheduled here? Which then raises the
> question on d->arch.psr_cos_ids being NULL - is that strictly only
> possible for the idle domain, or are there also other cases? This
> determines how the if() condition should be re-written ...
>
I agree with you that IDLE domain can be skipped. But considering that some
domains' 'psr_cos_ids' may fail to be allocated, we have to set default value
for these domains. So, I think we should keep this but skip IDLE domain in
'psr_ctxt_switch_to' caller.
> > @@ -401,14 +445,37 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> > return 0;
> > }
> >
> > -int psr_domain_init(struct domain *d)
> > +/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> > +static void psr_free_cos(struct domain *d)
> > +{
> > + xfree(d->arch.psr_cos_ids);
> > + d->arch.psr_cos_ids = NULL;
> > +}
> > +
> > +static int psr_alloc_cos(struct domain *d)
> > {
> > + d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
> > + if ( !d->arch.psr_cos_ids )
> > + return -ENOMEM;
> > +
> > return 0;
> > }
> >
> > +int psr_domain_init(struct domain *d)
> > +{
> > + /* Init to success value */
> > + int ret = 0;
> > +
> > + if ( psr_alloc_feat_enabled() )
> > + ret = psr_alloc_cos(d);
> > +
> > + return ret;
> > +}
>
> Along the lines of the above - do we really need to fail domain
> creation if we can't alloc psr_cos_ids? Granted there'll be other
> allocation failures, but from an abstract pov this is an optional
> feature, and hence the domain could do fine without.
>
Ok, will modiy related codes.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows.
2017-05-31 6:37 ` Yi Sun
@ 2017-05-31 6:57 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 6:57 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 31.05.17 at 08:37, <yi.y.sun@linux.intel.com> wrote:
> On 17-05-30 07:26:55, Jan Beulich wrote:
>> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> > @@ -376,6 +412,14 @@ void psr_ctxt_switch_to(struct domain *d)
>> > if ( psr_cmt_enabled() )
>> > psr_assoc_rmid(®, d->arch.psr_rmid);
>> >
>> > + /* IDLE domain's 'psr_cos_ids' is NULL so we set default value for it. */
>> > + if ( psra->cos_mask )
>> > + psr_assoc_cos(®,
>> > + d->arch.psr_cos_ids ?
>> > + d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
>> > + 0,
>>
>> As to the idle domain aspect - is there a strict need to write a new
>> value for the idle domain at all? I.e. can't you just skip the write in
>> that case, knowing you'll write a proper value anyway once the
>> next non-idle vCPU gets scheduled here? Which then raises the
>> question on d->arch.psr_cos_ids being NULL - is that strictly only
>> possible for the idle domain, or are there also other cases? This
>> determines how the if() condition should be re-written ...
>>
> I agree with you that IDLE domain can be skipped. But considering that some
> domains' 'psr_cos_ids' may fail to be allocated, we have to set default value
> for these domains. So, I think we should keep this but skip IDLE domain in
> 'psr_ctxt_switch_to' caller.
There don't need to be two conditionals, i.e. using the one here
is fine. But the comment then shouldn't give the impression the
idle domain is the only case to consider.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-05-30 14:05 ` Jan Beulich
@ 2017-05-31 7:30 ` Yi Sun
2017-05-31 7:45 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-31 7:30 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-30 08:05:02, Jan Beulich wrote:
> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -476,23 +476,34 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
> > return socket_info + socket;
> > }
> >
> > +static struct feat_node *psr_get_feat_and_type(unsigned int socket,
> > + enum cbm_type type,
> > + enum psr_feat_type *feat_type)
> > +{
> > + const struct psr_socket_info *info = get_socket_info(socket);
> > +
> > + if ( IS_ERR(info) )
> > + return ERR_PTR(PTR_ERR(info));
> > +
> > + *feat_type = psr_cbm_type_to_feat_type(type);
> > + if ( *feat_type >= ARRAY_SIZE(info->features) )
> > + return NULL;
>
> Note how this return is not being taken care of by ...
>
> > + return info->features[*feat_type];
> > +}
> > +
> > int psr_get_info(unsigned int socket, enum cbm_type type,
> > uint32_t data[], unsigned int array_len)
> > {
> > - const struct psr_socket_info *info = get_socket_info(socket);
> > const struct feat_node *feat;
> > enum psr_feat_type feat_type;
> >
> > ASSERT(data);
> >
> > - if ( IS_ERR(info) )
> > - return PTR_ERR(info);
> > -
> > - feat_type = psr_cbm_type_to_feat_type(type);
> > - if ( feat_type >= ARRAY_SIZE(info->features) )
> > - return -ENOENT;
> > + feat = psr_get_feat_and_type(socket, type, &feat_type);
> > + if ( IS_ERR(feat) )
> > + return PTR_ERR(feat);
>
> ... the check here. I think you want to alter the return above.
>
This NULL is taken care by below code:
if ( !feat || !feat_props[feat_type] )
The returned errors are handled separately. For PTR_ERR, it is handled
above. For NULL, it is handled below.
I checked IS_ERR, I think it can handle the NULL case. The NULL will not
be treated as an error.
> And of course I wonder why you replace code here that was
> only introduced one or two patches earlier. Perhaps that earlier
> patch should do things this way right away?
>
Because the helper function 'psr_get_feat_and_type' is only used by
'psr_get_info' if we implement it in previous patch. This seems
unnecessary. So, I introduce this helper function in this patch.
Shall I move it to previous patch?
> > - feat = info->features[feat_type];
> > if ( !feat || !feat_props[feat_type] )
> > return -ENOENT;
>
> Afaics you need feat_type here only to get at the right feat_props[]
> entry. If that's the case also for future callers of
> psr_get_feat_and_type(), perhaps it would be better for it to
> provide those two instead of the intermediate type? Of course
> that would imply renaming the function. (This change would
> clearly benefit the readability of psr_get_val() below.)
>
Ok, got it.
> > @@ -502,9 +513,38 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> > return -EINVAL;
> > }
> >
> > -int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> > - uint64_t *cbm, enum cbm_type type)
> > +int psr_get_val(struct domain *d, unsigned int socket,
> > + uint32_t *val, enum cbm_type type)
> > {
> > + const struct feat_node *feat;
> > + enum psr_feat_type feat_type;
> > + unsigned int cos, i;
> > +
> > + ASSERT(val);
> > +
> > + feat = psr_get_feat_and_type(socket, type, &feat_type);
> > + if ( IS_ERR(feat) )
> > + return PTR_ERR(feat);
> > +
> > + if ( !feat || !feat_props[feat_type] )
> > + return -ENOENT;
> > +
> > + cos = d->arch.psr_cos_ids[socket];
> > + /*
> > + * If input cos exceeds current feature's cos_max, we should return its
> > + * default value which is stored in cos 0. This case only happens
> > + * when more than two features enabled concurrently and at least one
> > + * features's cos_max is bigger than others. When a domain's working cos
> > + * id is bigger than some features' cos_max, HW automatically works as
> > + * default value for those features which cos_max is smaller.
> > + */
> > + if ( cos > feat->cos_max )
> > + cos = 0;
> > +
> > + for ( i = 0; i < feat_props[feat_type]->cos_num; i++ )
> > + if ( type == feat_props[feat_type]->type[i] )
> > + *val = feat->cos_reg_val[cos * feat_props[feat_type]->cos_num + i];
> > +
> > return 0;
> > }
>
> Do you really want to return success here even if you didn't write
> to *val? With the way the callers are coded, this is an (at least
> latent) information leak at present.
>
Will fix it. Thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-05-31 7:30 ` Yi Sun
@ 2017-05-31 7:45 ` Jan Beulich
2017-05-31 8:05 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 7:45 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 31.05.17 at 09:30, <yi.y.sun@linux.intel.com> wrote:
> On 17-05-30 08:05:02, Jan Beulich wrote:
>> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>>
>> > --- a/xen/arch/x86/psr.c
>> > +++ b/xen/arch/x86/psr.c
>> > @@ -476,23 +476,34 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
>> > return socket_info + socket;
>> > }
>> >
>> > +static struct feat_node *psr_get_feat_and_type(unsigned int socket,
>> > + enum cbm_type type,
>> > + enum psr_feat_type *feat_type)
>> > +{
>> > + const struct psr_socket_info *info = get_socket_info(socket);
>> > +
>> > + if ( IS_ERR(info) )
>> > + return ERR_PTR(PTR_ERR(info));
>> > +
>> > + *feat_type = psr_cbm_type_to_feat_type(type);
>> > + if ( *feat_type >= ARRAY_SIZE(info->features) )
>> > + return NULL;
>>
>> Note how this return is not being taken care of by ...
>>
>> > + return info->features[*feat_type];
>> > +}
>> > +
>> > int psr_get_info(unsigned int socket, enum cbm_type type,
>> > uint32_t data[], unsigned int array_len)
>> > {
>> > - const struct psr_socket_info *info = get_socket_info(socket);
>> > const struct feat_node *feat;
>> > enum psr_feat_type feat_type;
>> >
>> > ASSERT(data);
>> >
>> > - if ( IS_ERR(info) )
>> > - return PTR_ERR(info);
>> > -
>> > - feat_type = psr_cbm_type_to_feat_type(type);
>> > - if ( feat_type >= ARRAY_SIZE(info->features) )
>> > - return -ENOENT;
>> > + feat = psr_get_feat_and_type(socket, type, &feat_type);
>> > + if ( IS_ERR(feat) )
>> > + return PTR_ERR(feat);
>>
>> ... the check here. I think you want to alter the return above.
>>
> This NULL is taken care by below code:
> if ( !feat || !feat_props[feat_type] )
Oh, I see.
> The returned errors are handled separately. For PTR_ERR, it is handled
> above. For NULL, it is handled below.
>
> I checked IS_ERR, I think it can handle the NULL case. The NULL will not
> be treated as an error.
"it can handle the NULL case" is rather ambiguous: The NULL
case normally (including the case here) also is an error, and
IS_ERR() _does not_ detect this error. Hence using it one the
result of a function that may return NULL is at least
questionable (IS_ERR_OR_NULL() is intended to be used in such
cases).
So while indeed the code is correct as is, I'd still like to ask you
to make the suggested change so that the code also ends up
being visibly correct at the first glance.
>> And of course I wonder why you replace code here that was
>> only introduced one or two patches earlier. Perhaps that earlier
>> patch should do things this way right away?
>>
> Because the helper function 'psr_get_feat_and_type' is only used by
> 'psr_get_info' if we implement it in previous patch. This seems
> unnecessary. So, I introduce this helper function in this patch.
> Shall I move it to previous patch?
I'd prefer if you did. When taking such decisions, please also consider
the amount of churn you cause as well as reviewability.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-05-31 7:45 ` Jan Beulich
@ 2017-05-31 8:05 ` Yi Sun
2017-05-31 8:10 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-05-31 8:05 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-31 01:45:45, Jan Beulich wrote:
> >>> On 31.05.17 at 09:30, <yi.y.sun@linux.intel.com> wrote:
> > On 17-05-30 08:05:02, Jan Beulich wrote:
> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> >>
> >> > --- a/xen/arch/x86/psr.c
> >> > +++ b/xen/arch/x86/psr.c
> >> > @@ -476,23 +476,34 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
> >> > return socket_info + socket;
> >> > }
> >> >
> >> > +static struct feat_node *psr_get_feat_and_type(unsigned int socket,
> >> > + enum cbm_type type,
> >> > + enum psr_feat_type *feat_type)
> >> > +{
> >> > + const struct psr_socket_info *info = get_socket_info(socket);
> >> > +
> >> > + if ( IS_ERR(info) )
> >> > + return ERR_PTR(PTR_ERR(info));
> >> > +
> >> > + *feat_type = psr_cbm_type_to_feat_type(type);
> >> > + if ( *feat_type >= ARRAY_SIZE(info->features) )
> >> > + return NULL;
> >>
> >> Note how this return is not being taken care of by ...
> >>
> >> > + return info->features[*feat_type];
> >> > +}
> >> > +
> >> > int psr_get_info(unsigned int socket, enum cbm_type type,
> >> > uint32_t data[], unsigned int array_len)
> >> > {
> >> > - const struct psr_socket_info *info = get_socket_info(socket);
> >> > const struct feat_node *feat;
> >> > enum psr_feat_type feat_type;
> >> >
> >> > ASSERT(data);
> >> >
> >> > - if ( IS_ERR(info) )
> >> > - return PTR_ERR(info);
> >> > -
> >> > - feat_type = psr_cbm_type_to_feat_type(type);
> >> > - if ( feat_type >= ARRAY_SIZE(info->features) )
> >> > - return -ENOENT;
> >> > + feat = psr_get_feat_and_type(socket, type, &feat_type);
> >> > + if ( IS_ERR(feat) )
> >> > + return PTR_ERR(feat);
> >>
> >> ... the check here. I think you want to alter the return above.
> >>
> > This NULL is taken care by below code:
> > if ( !feat || !feat_props[feat_type] )
>
> Oh, I see.
>
> > The returned errors are handled separately. For PTR_ERR, it is handled
> > above. For NULL, it is handled below.
> >
> > I checked IS_ERR, I think it can handle the NULL case. The NULL will not
> > be treated as an error.
>
> "it can handle the NULL case" is rather ambiguous: The NULL
> case normally (including the case here) also is an error, and
> IS_ERR() _does not_ detect this error. Hence using it one the
> result of a function that may return NULL is at least
> questionable (IS_ERR_OR_NULL() is intended to be used in such
> cases).
>
> So while indeed the code is correct as is, I'd still like to ask you
> to make the suggested change so that the code also ends up
> being visibly correct at the first glance.
>
Thank you! Will use 'IS_ERR_OR_NULL()' to check result.
> >> And of course I wonder why you replace code here that was
> >> only introduced one or two patches earlier. Perhaps that earlier
> >> patch should do things this way right away?
> >>
> > Because the helper function 'psr_get_feat_and_type' is only used by
> > 'psr_get_info' if we implement it in previous patch. This seems
> > unnecessary. So, I introduce this helper function in this patch.
> > Shall I move it to previous patch?
>
> I'd prefer if you did. When taking such decisions, please also consider
> the amount of churn you cause as well as reviewability.
>
Got it, thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-05-31 8:05 ` Yi Sun
@ 2017-05-31 8:10 ` Jan Beulich
2017-06-01 3:14 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 8:10 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 31.05.17 at 10:05, <yi.y.sun@linux.intel.com> wrote:
> On 17-05-31 01:45:45, Jan Beulich wrote:
>> >>> On 31.05.17 at 09:30, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-05-30 08:05:02, Jan Beulich wrote:
>> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> >>
>> >> > --- a/xen/arch/x86/psr.c
>> >> > +++ b/xen/arch/x86/psr.c
>> >> > @@ -476,23 +476,34 @@ static struct psr_socket_info
> *get_socket_info(unsigned int socket)
>> >> > return socket_info + socket;
>> >> > }
>> >> >
>> >> > +static struct feat_node *psr_get_feat_and_type(unsigned int socket,
>> >> > + enum cbm_type type,
>> >> > + enum psr_feat_type
> *feat_type)
>> >> > +{
>> >> > + const struct psr_socket_info *info = get_socket_info(socket);
>> >> > +
>> >> > + if ( IS_ERR(info) )
>> >> > + return ERR_PTR(PTR_ERR(info));
>> >> > +
>> >> > + *feat_type = psr_cbm_type_to_feat_type(type);
>> >> > + if ( *feat_type >= ARRAY_SIZE(info->features) )
>> >> > + return NULL;
>> >>
>> >> Note how this return is not being taken care of by ...
>> >>
>> >> > + return info->features[*feat_type];
>> >> > +}
>> >> > +
>> >> > int psr_get_info(unsigned int socket, enum cbm_type type,
>> >> > uint32_t data[], unsigned int array_len)
>> >> > {
>> >> > - const struct psr_socket_info *info = get_socket_info(socket);
>> >> > const struct feat_node *feat;
>> >> > enum psr_feat_type feat_type;
>> >> >
>> >> > ASSERT(data);
>> >> >
>> >> > - if ( IS_ERR(info) )
>> >> > - return PTR_ERR(info);
>> >> > -
>> >> > - feat_type = psr_cbm_type_to_feat_type(type);
>> >> > - if ( feat_type >= ARRAY_SIZE(info->features) )
>> >> > - return -ENOENT;
>> >> > + feat = psr_get_feat_and_type(socket, type, &feat_type);
>> >> > + if ( IS_ERR(feat) )
>> >> > + return PTR_ERR(feat);
>> >>
>> >> ... the check here. I think you want to alter the return above.
>> >>
>> > This NULL is taken care by below code:
>> > if ( !feat || !feat_props[feat_type] )
>>
>> Oh, I see.
>>
>> > The returned errors are handled separately. For PTR_ERR, it is handled
>> > above. For NULL, it is handled below.
>> >
>> > I checked IS_ERR, I think it can handle the NULL case. The NULL will not
>> > be treated as an error.
>>
>> "it can handle the NULL case" is rather ambiguous: The NULL
>> case normally (including the case here) also is an error, and
>> IS_ERR() _does not_ detect this error. Hence using it one the
>> result of a function that may return NULL is at least
>> questionable (IS_ERR_OR_NULL() is intended to be used in such
>> cases).
>>
>> So while indeed the code is correct as is, I'd still like to ask you
>> to make the suggested change so that the code also ends up
>> being visibly correct at the first glance.
>>
> Thank you! Will use 'IS_ERR_OR_NULL()' to check result.
But that's not what I did suggest, and doing so will make the
handling at the checking site more clumsy.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-05-03 8:44 ` [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
2017-05-30 13:05 ` Jan Beulich
@ 2017-05-31 9:36 ` Jan Beulich
2017-06-01 3:18 ` Yi Sun
1 sibling, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 9:36 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> +static void cat_init_feature(const struct cpuid_leaf *regs,
> + struct feat_node *feat,
> + struct psr_socket_info *info,
> + enum psr_feat_type type)
> +{
> + /* No valid value so do not enable feature. */
> + if ( !regs->a || !regs->d )
> + return;
> +
> + feat->cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
> + feat->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
> +
> + switch ( type )
> + {
> + case PSR_SOCKET_L3_CAT:
> + /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> + feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
> +
> + break;
> +
> + default:
> + return;
> + }
> +
> + /* Add this feature into array. */
> + info->features[type] = feat;
> +
> + if ( !opt_cpu_info )
> + return;
> +
> + printk(XENLOG_INFO "%s CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
> + ((type == PSR_SOCKET_L3_CAT) ? "L3" : "L2"),
> + cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
> +}
Btw, I don't think there should be any mention of L2 CAT here just yet.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-05-03 8:44 ` [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
@ 2017-05-31 9:37 ` Jan Beulich
2017-06-02 7:26 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 9:37 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> @@ -150,11 +151,28 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
> * array creation. It is used to transiently store a spare node.
> */
> static struct feat_node *feat_l3_cat;
> +static struct feat_node *feat_l3_cdp;
>
> /* Common functions */
> #define cat_default_val(len) (0xffffffff >> (32 - (len)))
>
> /*
> + * get_cdp_data - get DATA COS register value from input COS ID.
> + * @feat: the feature node.
> + * @cos: the COS ID.
> + */
> +#define get_cdp_data(feat, cos) \
> + ( (feat)->cos_reg_val[(cos) * 2] )
Stray blanks inside parentheses.
> +/*
> + * get_cdp_code - get CODE COS register value from input COS ID.
> + * @feat: the feature node.
> + * @cos: the COS ID.
> + */
> +#define get_cdp_code(feat, cos) \
> + ( (feat)->cos_reg_val[(cos) * 2 + 1] )
Same here. In fact, while I'm not going to ask to remove the
parentheses, this is the one case where an expression a macro
expands to does not strictly need an outer pair of parentheses,
as suffix expressions have highest precedence anyway.
> @@ -249,6 +267,25 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
>
> break;
>
> + case PSR_SOCKET_L3_CDP:
> + {
> + unsigned long val;
As MSR values are specifically 64-bit ones, I think uint64_t would
be more appropriate here. Depending on intended later additions to
this function it may also be worthwhile making this a switch-wide
variable.
> + /* Cut half of cos_max when CDP is enabled. */
> + feat->cos_max >>= 1;
> +
> + /* We only write mask1 since mask0 is always all ones by default. */
Is this, btw, just reset state or even guaranteed after offlining
and re-onlining a CPU?
> + wrmsrl(MSR_IA32_PSR_L3_MASK(1), cat_default_val(feat->cbm_len));
> + rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
> + wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
1u at the very least, perhaps even 1ull.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 14/23] x86: refactor psr: CDP: implement get hw info flow.
2017-05-03 8:44 ` [PATCH v11 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
@ 2017-05-31 9:40 ` Jan Beulich
2017-06-05 8:09 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 9:40 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -207,7 +207,9 @@ static void free_socket_resources(unsigned int socket)
> memset(info->dom_ids, 0, ((DOMID_IDLE + 1) + 7) / 8);
> }
>
> -static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> +static enum psr_feat_type psr_cbm_type_to_feat_type(
> + const struct psr_socket_info *info,
> + enum cbm_type type)
Couldn't you avoid adding this new parameter by checking ...
> @@ -215,7 +217,18 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> {
> case PSR_CBM_TYPE_L3:
> feat_type = PSR_SOCKET_L3_CAT;
> +
> + /* If type is L3 CAT but we cannot find it in feature array, try CDP. */
> + if ( !info->features[feat_type] )
... the props array entry here?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-05-03 8:44 ` [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
@ 2017-05-31 9:44 ` Jan Beulich
2017-06-02 7:59 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 9:44 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -352,9 +352,21 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
> return true;
> }
>
> +static void l3_cdp_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
> +{
> + /* Data */
> + if ( type == PSR_CBM_TYPE_L3_DATA )
> + wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(cos), val);
> +
> + /* Code */
> + if ( type == PSR_CBM_TYPE_L3_CODE )
> + wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(cos), val);
> +}
With l3_cat_write_msr() ignoring "type" altogether I think this wants
to be an if/else pair (or even a conditional expression for the first
wrmsrl() argument).
> @@ -765,7 +777,8 @@ static int insert_val_into_array(uint32_t val[],
>
> /* Value setting position is same as feature array. */
> for ( i = 0; i < props->cos_num; i++ )
> - if ( type == props->type[i] )
> + if ( type == props->type[i] ||
> + (feat_type == PSR_SOCKET_L3_CDP && type == PSR_CBM_TYPE_L3) )
Didn't the earlier patch take care of doing this substitution? Non-
feature-specific code clearly shouldn't have such special cases if
at all avoidable.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 18/23] x86: L2 CAT: implement get value flow.
2017-05-03 8:44 ` [PATCH v11 18/23] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-05-31 9:51 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 9:51 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> This patch implements L2 CAT get value interface in domctl.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 19/23] x86: L2 CAT: implement set value flow.
2017-05-03 8:44 ` [PATCH v11 19/23] x86: L2 CAT: implement set " Yi Sun
@ 2017-05-31 9:52 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-05-31 9:52 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> This patch implements L2 CAT set value related callback function
> and domctl interface.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-05-31 8:10 ` Jan Beulich
@ 2017-06-01 3:14 ` Yi Sun
0 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-06-01 3:14 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-31 02:10:27, Jan Beulich wrote:
> >>> On 31.05.17 at 10:05, <yi.y.sun@linux.intel.com> wrote:
> > On 17-05-31 01:45:45, Jan Beulich wrote:
> >> >>> On 31.05.17 at 09:30, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-05-30 08:05:02, Jan Beulich wrote:
> >> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> >> >>
> >> >> > --- a/xen/arch/x86/psr.c
> >> >> > +++ b/xen/arch/x86/psr.c
> >> >> > @@ -476,23 +476,34 @@ static struct psr_socket_info
> > *get_socket_info(unsigned int socket)
> >> >> > return socket_info + socket;
> >> >> > }
> >> >> >
> >> >> > +static struct feat_node *psr_get_feat_and_type(unsigned int socket,
> >> >> > + enum cbm_type type,
> >> >> > + enum psr_feat_type
> > *feat_type)
> >> >> > +{
> >> >> > + const struct psr_socket_info *info = get_socket_info(socket);
> >> >> > +
> >> >> > + if ( IS_ERR(info) )
> >> >> > + return ERR_PTR(PTR_ERR(info));
> >> >> > +
> >> >> > + *feat_type = psr_cbm_type_to_feat_type(type);
> >> >> > + if ( *feat_type >= ARRAY_SIZE(info->features) )
> >> >> > + return NULL;
> >> >>
> >> >> Note how this return is not being taken care of by ...
> >> >>
> >> >> > + return info->features[*feat_type];
> >> >> > +}
> >> >> > +
> >> >> > int psr_get_info(unsigned int socket, enum cbm_type type,
> >> >> > uint32_t data[], unsigned int array_len)
> >> >> > {
> >> >> > - const struct psr_socket_info *info = get_socket_info(socket);
> >> >> > const struct feat_node *feat;
> >> >> > enum psr_feat_type feat_type;
> >> >> >
> >> >> > ASSERT(data);
> >> >> >
> >> >> > - if ( IS_ERR(info) )
> >> >> > - return PTR_ERR(info);
> >> >> > -
> >> >> > - feat_type = psr_cbm_type_to_feat_type(type);
> >> >> > - if ( feat_type >= ARRAY_SIZE(info->features) )
> >> >> > - return -ENOENT;
> >> >> > + feat = psr_get_feat_and_type(socket, type, &feat_type);
> >> >> > + if ( IS_ERR(feat) )
> >> >> > + return PTR_ERR(feat);
> >> >>
> >> >> ... the check here. I think you want to alter the return above.
> >> >>
> >> > This NULL is taken care by below code:
> >> > if ( !feat || !feat_props[feat_type] )
> >>
> >> Oh, I see.
> >>
> >> > The returned errors are handled separately. For PTR_ERR, it is handled
> >> > above. For NULL, it is handled below.
> >> >
> >> > I checked IS_ERR, I think it can handle the NULL case. The NULL will not
> >> > be treated as an error.
> >>
> >> "it can handle the NULL case" is rather ambiguous: The NULL
> >> case normally (including the case here) also is an error, and
> >> IS_ERR() _does not_ detect this error. Hence using it one the
> >> result of a function that may return NULL is at least
> >> questionable (IS_ERR_OR_NULL() is intended to be used in such
> >> cases).
> >>
> >> So while indeed the code is correct as is, I'd still like to ask you
> >> to make the suggested change so that the code also ends up
> >> being visibly correct at the first glance.
> >>
> > Thank you! Will use 'IS_ERR_OR_NULL()' to check result.
>
> But that's not what I did suggest, and doing so will make the
> handling at the checking site more clumsy.
>
The 'psr_get_feat_and_type' make things complex here. I'd like to discard
it and directly implement the functionality of it in 'psr_get_info' and
'psr_get_val' to make codes clear.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-05-31 9:36 ` Jan Beulich
@ 2017-06-01 3:18 ` Yi Sun
2017-06-01 6:57 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-06-01 3:18 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau
On 17-05-31 03:36:43, Jan Beulich wrote:
> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> > +static void cat_init_feature(const struct cpuid_leaf *regs,
> > + struct feat_node *feat,
> > + struct psr_socket_info *info,
> > + enum psr_feat_type type)
> > +{
> > + /* No valid value so do not enable feature. */
> > + if ( !regs->a || !regs->d )
> > + return;
> > +
> > + feat->cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
> > + feat->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
> > +
> > + switch ( type )
> > + {
> > + case PSR_SOCKET_L3_CAT:
> > + /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> > + feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
> > +
> > + break;
> > +
> > + default:
> > + return;
> > + }
> > +
> > + /* Add this feature into array. */
> > + info->features[type] = feat;
> > +
> > + if ( !opt_cpu_info )
> > + return;
> > +
> > + printk(XENLOG_INFO "%s CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
> > + ((type == PSR_SOCKET_L3_CAT) ? "L3" : "L2"),
> > + cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
> > +}
>
> Btw, I don't think there should be any mention of L2 CAT here just yet.
>
So I think you would like me to move this into L2 CAT patch, right?
> Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-06-01 3:18 ` Yi Sun
@ 2017-06-01 6:57 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-06-01 6:57 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 01.06.17 at 05:18, <yi.y.sun@linux.intel.com> wrote:
> On 17-05-31 03:36:43, Jan Beulich wrote:
>> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> > +static void cat_init_feature(const struct cpuid_leaf *regs,
>> > + struct feat_node *feat,
>> > + struct psr_socket_info *info,
>> > + enum psr_feat_type type)
>> > +{
>> > + /* No valid value so do not enable feature. */
>> > + if ( !regs->a || !regs->d )
>> > + return;
>> > +
>> > + feat->cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
>> > + feat->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
>> > +
>> > + switch ( type )
>> > + {
>> > + case PSR_SOCKET_L3_CAT:
>> > + /* cos=0 is reserved as default cbm(all bits within cbm_len are
> 1). */
>> > + feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
>> > +
>> > + break;
>> > +
>> > + default:
>> > + return;
>> > + }
>> > +
>> > + /* Add this feature into array. */
>> > + info->features[type] = feat;
>> > +
>> > + if ( !opt_cpu_info )
>> > + return;
>> > +
>> > + printk(XENLOG_INFO "%s CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
>> > + ((type == PSR_SOCKET_L3_CAT) ? "L3" : "L2"),
>> > + cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
>> > +}
>>
>> Btw, I don't think there should be any mention of L2 CAT here just yet.
>>
> So I think you would like me to move this into L2 CAT patch, right?
Of course - that's where it belongs.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-05-30 14:32 ` Jan Beulich
@ 2017-06-01 10:00 ` Yi Sun
2017-06-01 10:45 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-06-01 10:00 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-30 08:32:59, Jan Beulich wrote:
> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -118,11 +118,13 @@ static const struct feat_props {
> > * COS ID. Every entry of cos_ref corresponds to one COS ID.
> > */
> > struct psr_socket_info {
> > - bool feat_init;
> > - spinlock_t ref_lock;
> > /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
> > struct feat_node *features[PSR_SOCKET_FEAT_NUM];
> > + bool feat_init;
> > unsigned int cos_ref[MAX_COS_REG_CNT];
> > + spinlock_t ref_lock;
>
> This shuffling of fields seems unmotivated and is not being explained
> in the description.
>
Per your comment in v10, such movement may avoid false cacheline conflicts.
The comment is below.
Also please try to space apart the two locks, to avoid false cacheline
conflicts (e.g. the new lock may well go immediately before the array
it pairs with).
> > @@ -178,6 +180,10 @@ static void free_socket_resources(unsigned int socket)
> > }
> >
> > info->feat_init = false;
> > +
> > + memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
> > +
> > + memset(info->dom_ids, 0, ((DOMID_IDLE + 1) + 7) / 8);
>
> bitmap_clear()
>
I searched the codes and found 'bitmap_clear' is only defined in tools/. There
is no such definition in hypervisor. So, I did not use it.
> > @@ -449,11 +455,19 @@ void psr_ctxt_switch_to(struct domain *d)
> >
> > /* IDLE domain's 'psr_cos_ids' is NULL so we set default value for it. */
> > if ( psra->cos_mask )
> > - psr_assoc_cos(®,
> > - d->arch.psr_cos_ids ?
> > - d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
> > - 0,
> > - psra->cos_mask);
> > + {
> > + unsigned int socket = cpu_to_socket(smp_processor_id());
> > + struct psr_socket_info *info = socket_info + socket;
> > + unsigned int cos = 0;
> > +
> > + if ( d->arch.psr_cos_ids )
> > + cos = d->arch.psr_cos_ids[socket];
> > +
> > + if ( unlikely(!test_bit(d->domain_id, info->dom_ids)) )
> > + cos = 0;
>
> I think a brief comment here would be helpful. I also think the two
> if()-s would better be combined (after all to initialize cos to zero
> above, so you simply need to avoid overwriting it in the first if().
>
Ok, will add comment and merge the two if()-s.
> > @@ -529,6 +543,10 @@ int psr_get_val(struct domain *d, unsigned int socket,
> > if ( !feat || !feat_props[feat_type] )
> > return -ENOENT;
> >
> > + if ( !test_bit(d->domain_id, socket_info[socket].dom_ids) &&
> > + d->arch.psr_cos_ids[socket] )
> > + d->arch.psr_cos_ids[socket] = 0;
>
> What is the right side of the && good for?
>
The purpose is to avoid unnecessary restore action. But yes, this is a
redundant check.
> Also the latest here it is clear that "dom_ids" isn't the best choice
> for a name. What about "dom_set" or "domain_set"?
>
I prefer 'dom_set'.
> > +/* The whole set process is protected by domctl_lock. */
>
> This needs to be re-considered: We intend to incrementally
> reduce code ranges guarded by the domctl lock, and a good
> first step might be to stop introducing further dependencies on
> it in individual handlers. So the main question is: Do you need a
> global lock here at all, or would a per-domain one suffice? In
> the former case I think you should (re?)introduce your own,
> while in the latter case you could probably use domain_lock(),
> but since you don't require other per-domain activities to be
> synchronized, having your own private would perhaps be
> even better.
>
The domctl_lock is used protect resources used in below functions:
1. psr_domain_init
2. psr_domain_free
3. psr_get_info
4. psr_get_val
5. psr_set_val
Per analysis, only 'd->arch.psr_cos_ids' should be protected in these
functions and psr_domain_init/free do not need lock because domain
cannot be scheduled when they are called.
So, I think 'domain_lock' is enough.
> > +int psr_set_val(struct domain *d, unsigned int socket,
> > + uint64_t new_val, enum cbm_type type)
> > +{
> > + unsigned int old_cos;
> > + int cos, ret;
> > + unsigned int *ref;
> > + uint32_t *val_array, val;
> > + struct psr_socket_info *info = get_socket_info(socket);
> > + unsigned int array_len;
> > + enum psr_feat_type feat_type;
> > +
> > + if ( IS_ERR(info) )
> > + return PTR_ERR(info);
> > +
> > + if ( new_val != (uint32_t)new_val )
> > + return -EINVAL;
> > +
> > + val = new_val;
>
> Please switch this and the prior if(), using val instead of the cast
> expression there.
>
Got it.
> > + feat_type = psr_cbm_type_to_feat_type(type);
> > + if ( feat_type >= ARRAY_SIZE(info->features) ||
> > + !info->features[feat_type] )
> > + return -ENOENT;
>
> Without seeing the code inside the functions you pass feat_type
> to below it's not really clear whether you wouldn't better use
> what is currently named psr_get_feat_and_type() here.
>
'psr_get_feat_and_type' will be removed. So, I would like to keep codes here.
What is your opinion?
> > + free_array:
> > + xfree(val_array);
> > + return ret;
> > +
> > + unlock_free_array:
> > + spin_unlock(&info->ref_lock);
> > + xfree(val_array);
> > + return ret;
> > +}
>
> I'm sure I've said so before - please don't duplicate error paths like
> this. Here it's still easy to see all is fine, but what if each path gets
> two or three more thing added. Please chain them together via goto.
>
To make things clear, I wrote below codes. How about them?
unlock_free_array:
spin_unlock(&info->ref_lock);
free_array:
xfree(val_array);
return ret;
> > /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> > static void psr_free_cos(struct domain *d)
> > {
> > + unsigned int socket, cos;
> > +
> > + ASSERT(socket_info);
> > +
> > + if ( !d->arch.psr_cos_ids )
> > + return;
> > +
> > + /* Domain is destroied so its cos_ref should be decreased. */
>
> destroyed
>
> > + for ( socket = 0; socket < nr_sockets; socket++ )
> > + {
> > + struct psr_socket_info *info;
> > +
> > + /* cos 0 is default one which does not need be handled. */
> > + cos = d->arch.psr_cos_ids[socket];
> > + if ( cos == 0 )
> > + continue;
>
> Does this "doesn't need to be handled" even extend to ...
>
> > + info = socket_info + socket;
> > + spin_lock(&info->ref_lock);
> > + ASSERT(info->cos_ref[cos]);
> > + info->cos_ref[cos]--;
> > + spin_unlock(&info->ref_lock);
> > +
> > + clear_bit(d->domain_id, info->dom_ids);
>
> ... this last part?
>
Hmm, I should clear it no matter cos is 0 or not. Thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-01 10:00 ` Yi Sun
@ 2017-06-01 10:45 ` Jan Beulich
2017-06-02 2:49 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-06-01 10:45 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 01.06.17 at 12:00, <yi.y.sun@linux.intel.com> wrote:
> On 17-05-30 08:32:59, Jan Beulich wrote:
>> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> > --- a/xen/arch/x86/psr.c
>> > +++ b/xen/arch/x86/psr.c
>> > @@ -118,11 +118,13 @@ static const struct feat_props {
>> > * COS ID. Every entry of cos_ref corresponds to one COS ID.
>> > */
>> > struct psr_socket_info {
>> > - bool feat_init;
>> > - spinlock_t ref_lock;
>> > /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
>> > struct feat_node *features[PSR_SOCKET_FEAT_NUM];
>> > + bool feat_init;
>> > unsigned int cos_ref[MAX_COS_REG_CNT];
>> > + spinlock_t ref_lock;
>>
>> This shuffling of fields seems unmotivated and is not being explained
>> in the description.
>>
> Per your comment in v10, such movement may avoid false cacheline conflicts.
> The comment is below.
> Also please try to space apart the two locks, to avoid false cacheline
> conflicts (e.g. the new lock may well go immediately before the array
> it pairs with).
Well - where is the second lock here?
>> > @@ -178,6 +180,10 @@ static void free_socket_resources(unsigned int socket)
>> > }
>> >
>> > info->feat_init = false;
>> > +
>> > + memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
>> > +
>> > + memset(info->dom_ids, 0, ((DOMID_IDLE + 1) + 7) / 8);
>>
>> bitmap_clear()
>>
> I searched the codes and found 'bitmap_clear' is only defined in tools/.
> There
> is no such definition in hypervisor. So, I did not use it.
bitmap_zero()
>> > + feat_type = psr_cbm_type_to_feat_type(type);
>> > + if ( feat_type >= ARRAY_SIZE(info->features) ||
>> > + !info->features[feat_type] )
>> > + return -ENOENT;
>>
>> Without seeing the code inside the functions you pass feat_type
>> to below it's not really clear whether you wouldn't better use
>> what is currently named psr_get_feat_and_type() here.
>>
> 'psr_get_feat_and_type' will be removed. So, I would like to keep codes
> here.
> What is your opinion?
We'll see how it ends up being.
>> > + free_array:
>> > + xfree(val_array);
>> > + return ret;
>> > +
>> > + unlock_free_array:
>> > + spin_unlock(&info->ref_lock);
>> > + xfree(val_array);
>> > + return ret;
>> > +}
>>
>> I'm sure I've said so before - please don't duplicate error paths like
>> this. Here it's still easy to see all is fine, but what if each path gets
>> two or three more thing added. Please chain them together via goto.
>>
> To make things clear, I wrote below codes. How about them?
> unlock_free_array:
> spin_unlock(&info->ref_lock);
>
> free_array:
> xfree(val_array);
> return ret;
I don't think that'll be okay for the case which previously fell
through to free_array.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-01 10:45 ` Jan Beulich
@ 2017-06-02 2:49 ` Yi Sun
2017-06-06 7:43 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-06-02 2:49 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-01 04:45:58, Jan Beulich wrote:
> >>> On 01.06.17 at 12:00, <yi.y.sun@linux.intel.com> wrote:
> > On 17-05-30 08:32:59, Jan Beulich wrote:
> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> >> > --- a/xen/arch/x86/psr.c
> >> > +++ b/xen/arch/x86/psr.c
> >> > @@ -118,11 +118,13 @@ static const struct feat_props {
> >> > * COS ID. Every entry of cos_ref corresponds to one COS ID.
> >> > */
> >> > struct psr_socket_info {
> >> > - bool feat_init;
> >> > - spinlock_t ref_lock;
> >> > /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
> >> > struct feat_node *features[PSR_SOCKET_FEAT_NUM];
> >> > + bool feat_init;
> >> > unsigned int cos_ref[MAX_COS_REG_CNT];
> >> > + spinlock_t ref_lock;
> >>
> >> This shuffling of fields seems unmotivated and is not being explained
> >> in the description.
> >>
> > Per your comment in v10, such movement may avoid false cacheline conflicts.
> > The comment is below.
> > Also please try to space apart the two locks, to avoid false cacheline
> > conflicts (e.g. the new lock may well go immediately before the array
> > it pairs with).
>
> Well - where is the second lock here?
>
I thought 'feat_init' has same effect. But I should be wrong.
Then, I want to define the structure as below:
struct psr_socket_info {
bool feat_init;
/* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
struct feat_node *features[PSR_SOCKET_FEAT_NUM];
spinlock_t ref_lock;
unsigned int cos_ref[MAX_COS_REG_CNT];
/* Every bit corresponds to a domain. Index is domain_id. */
DECLARE_BITMAP(dom_ids, DOMID_IDLE + 1);
};
> >> > + free_array:
> >> > + xfree(val_array);
> >> > + return ret;
> >> > +
> >> > + unlock_free_array:
> >> > + spin_unlock(&info->ref_lock);
> >> > + xfree(val_array);
> >> > + return ret;
> >> > +}
> >>
> >> I'm sure I've said so before - please don't duplicate error paths like
> >> this. Here it's still easy to see all is fine, but what if each path gets
> >> two or three more thing added. Please chain them together via goto.
> >>
> > To make things clear, I wrote below codes. How about them?
> > unlock_free_array:
> > spin_unlock(&info->ref_lock);
> >
> > free_array:
> > xfree(val_array);
> > return ret;
>
> I don't think that'll be okay for the case which previously fell
> through to free_array.
>
I tried to understand your meaning. Do you mean below codes?
set_bit(d->domain_id, info->dom_ids); //Success path.
goto free_array;
unlock_free_array:
spin_unlock(&info->ref_lock);
free_array:
xfree(val_array);
return ret;
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-05-31 9:37 ` Jan Beulich
@ 2017-06-02 7:26 ` Yi Sun
2017-06-06 7:45 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-06-02 7:26 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-31 03:37:48, Jan Beulich wrote:
> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> > @@ -150,11 +151,28 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
> > @@ -249,6 +267,25 @@ static void cat_init_feature(const struct cpuid_leaf *regs,
> >
> > break;
> >
> > + case PSR_SOCKET_L3_CDP:
> > + {
> > + unsigned long val;
>
> As MSR values are specifically 64-bit ones, I think uint64_t would
> be more appropriate here. Depending on intended later additions to
> this function it may also be worthwhile making this a switch-wide
> variable.
>
This variable is only used in this case. Will change it to uint64_t.
> > + /* Cut half of cos_max when CDP is enabled. */
> > + feat->cos_max >>= 1;
> > +
> > + /* We only write mask1 since mask0 is always all ones by default. */
>
> Is this, btw, just reset state or even guaranteed after offlining
> and re-onlining a CPU?
>
Below MSRs are all per socket. So, we just need reset them when socket is
online.
> > + wrmsrl(MSR_IA32_PSR_L3_MASK(1), cat_default_val(feat->cbm_len));
> > + rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
> > + wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
>
> 1u at the very least, perhaps even 1ull.
>
Thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-05-31 9:44 ` Jan Beulich
@ 2017-06-02 7:59 ` Yi Sun
2017-06-06 7:48 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-06-02 7:59 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-31 03:44:31, Jan Beulich wrote:
> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -352,9 +352,21 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
> > return true;
> > }
> >
> > +static void l3_cdp_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
> > +{
> > + /* Data */
> > + if ( type == PSR_CBM_TYPE_L3_DATA )
> > + wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(cos), val);
> > +
> > + /* Code */
> > + if ( type == PSR_CBM_TYPE_L3_CODE )
> > + wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(cos), val);
> > +}
>
> With l3_cat_write_msr() ignoring "type" altogether I think this wants
> to be an if/else pair (or even a conditional expression for the first
> wrmsrl() argument).
>
Ok, thanks!
> > @@ -765,7 +777,8 @@ static int insert_val_into_array(uint32_t val[],
> >
> > /* Value setting position is same as feature array. */
> > for ( i = 0; i < props->cos_num; i++ )
> > - if ( type == props->type[i] )
> > + if ( type == props->type[i] ||
> > + (feat_type == PSR_SOCKET_L3_CDP && type == PSR_CBM_TYPE_L3) )
>
> Didn't the earlier patch take care of doing this substitution? Non-
> feature-specific code clearly shouldn't have such special cases if
> at all avoidable.
>
User can set both DATA and CODE to same value at same time with below command:
xl psr-cat-set dom_id 0x3ff
Because no '-c' or '-d' is input, the cbm type will be 'PSR_CBM_TYPE_L3'.
To handle this case, we have to add a special case here. If the cbm tyep is
'PSR_CBM_TYPE_L3' and the feature type is CDP, we set both DATA and CODE. This
should be the simplest way to handle this case.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 14/23] x86: refactor psr: CDP: implement get hw info flow.
2017-05-31 9:40 ` Jan Beulich
@ 2017-06-05 8:09 ` Yi Sun
0 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-06-05 8:09 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-31 03:40:40, Jan Beulich wrote:
> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -207,7 +207,9 @@ static void free_socket_resources(unsigned int socket)
> > memset(info->dom_ids, 0, ((DOMID_IDLE + 1) + 7) / 8);
> > }
> >
> > -static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> > +static enum psr_feat_type psr_cbm_type_to_feat_type(
> > + const struct psr_socket_info *info,
> > + enum cbm_type type)
>
> Couldn't you avoid adding this new parameter by checking ...
>
> > @@ -215,7 +217,18 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> > {
> > case PSR_CBM_TYPE_L3:
> > feat_type = PSR_SOCKET_L3_CAT;
> > +
> > + /* If type is L3 CAT but we cannot find it in feature array, try CDP. */
> > + if ( !info->features[feat_type] )
>
> ... the props array entry here?
>
Sure, thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-05-30 15:35 ` Jan Beulich
@ 2017-06-05 8:10 ` Yi Sun
0 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-06-05 8:10 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-05-30 09:35:53, Jan Beulich wrote:
> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> > +struct cos_write_info
> > +{
> > + unsigned int cos;
> > + struct feat_node *feature;
> > + uint32_t *val;
> > + enum psr_feat_type feat_type;
> > +};
> > +
> > +static void do_write_psr_msrs(void *data)
> > +{
> > + struct cos_write_info *info = data;
> > + unsigned int cos = info->cos;
> > + struct feat_node *feat = info->feature;
> > + const struct feat_props *props = feat_props[info->feat_type];
> > + unsigned int i;
> > +
> > + for ( i = 0; i < props->cos_num; i++ )
> > + {
> > + if ( feat->cos_reg_val[cos * props->cos_num + i] != info->val[i] )
> > + {
> > + feat->cos_reg_val[cos * props->cos_num + i] = info->val[i];
> > + props->write_msr(cos, info->val[i], props->type[i]);
> > + }
> > + }
> > +}
>
> Again you're passing feat_type here only to get at props. Why
> not pass props right away? Also I think it would make sense to
> pull props->cos_num into a local variable.
>
Have modified these according to your comments. Thanks!
> > static int write_psr_msrs(unsigned int socket, unsigned int cos,
> > uint32_t val[], unsigned int array_len,
> > enum psr_feat_type feat_type)
> > {
> > - return -ENOENT;
> > + unsigned int i;
> > + struct psr_socket_info *info = get_socket_info(socket);
> > + struct cos_write_info data =
> > + {
> > + .cos = cos,
> > + .feature = info->features[feat_type],
> > + .feat_type = feat_type,
> > + };
> > +
> > + if ( cos > info->features[feat_type]->cos_max )
> > + return -EINVAL;
> > +
> > + /* Skip to the feature's value head. */
> > + for ( i = 0; i < feat_type; i++ )
> > + {
> > + if ( !info->features[i] )
> > + continue;
>
> This is inconsistent with checks done elsewhere, where you also
> check feat_props[feat_type] against NULL. I've made a comment
> regarding whether both checks are wanted in a uniform or non-
> uniform way pretty early in the series. Whatever is selected
> should then be used consistently.
>
Have changed it. Thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-02 2:49 ` Yi Sun
@ 2017-06-06 7:43 ` Jan Beulich
2017-06-06 8:18 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-06-06 7:43 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 02.06.17 at 04:49, <yi.y.sun@linux.intel.com> wrote:
> On 17-06-01 04:45:58, Jan Beulich wrote:
>> >>> On 01.06.17 at 12:00, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-05-30 08:32:59, Jan Beulich wrote:
>> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> >> > --- a/xen/arch/x86/psr.c
>> >> > +++ b/xen/arch/x86/psr.c
>> >> > @@ -118,11 +118,13 @@ static const struct feat_props {
>> >> > * COS ID. Every entry of cos_ref corresponds to one COS ID.
>> >> > */
>> >> > struct psr_socket_info {
>> >> > - bool feat_init;
>> >> > - spinlock_t ref_lock;
>> >> > /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
>> >> > struct feat_node *features[PSR_SOCKET_FEAT_NUM];
>> >> > + bool feat_init;
>> >> > unsigned int cos_ref[MAX_COS_REG_CNT];
>> >> > + spinlock_t ref_lock;
>> >>
>> >> This shuffling of fields seems unmotivated and is not being explained
>> >> in the description.
>> >>
>> > Per your comment in v10, such movement may avoid false cacheline conflicts.
>> > The comment is below.
>> > Also please try to space apart the two locks, to avoid false cacheline
>> > conflicts (e.g. the new lock may well go immediately before the array
>> > it pairs with).
>>
>> Well - where is the second lock here?
>>
> I thought 'feat_init' has same effect. But I should be wrong.
>
> Then, I want to define the structure as below:
>
> struct psr_socket_info {
> bool feat_init;
> /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
> struct feat_node *features[PSR_SOCKET_FEAT_NUM];
> spinlock_t ref_lock;
> unsigned int cos_ref[MAX_COS_REG_CNT];
> /* Every bit corresponds to a domain. Index is domain_id. */
> DECLARE_BITMAP(dom_ids, DOMID_IDLE + 1);
> };
I've outlined my expectation to the ordering of fields before. The
above broadly matches that, so would be fine. What I'd like to ask
though is that fields don't get moved around without reason during
the series. Insert new fields at their intended final place unless
there's an actual reason to move them later.
>> >> > + free_array:
>> >> > + xfree(val_array);
>> >> > + return ret;
>> >> > +
>> >> > + unlock_free_array:
>> >> > + spin_unlock(&info->ref_lock);
>> >> > + xfree(val_array);
>> >> > + return ret;
>> >> > +}
>> >>
>> >> I'm sure I've said so before - please don't duplicate error paths like
>> >> this. Here it's still easy to see all is fine, but what if each path gets
>> >> two or three more thing added. Please chain them together via goto.
>> >>
>> > To make things clear, I wrote below codes. How about them?
>> > unlock_free_array:
>> > spin_unlock(&info->ref_lock);
>> >
>> > free_array:
>> > xfree(val_array);
>> > return ret;
>>
>> I don't think that'll be okay for the case which previously fell
>> through to free_array.
>>
> I tried to understand your meaning. Do you mean below codes?
>
> set_bit(d->domain_id, info->dom_ids); //Success path.
> goto free_array;
>
> unlock_free_array:
> spin_unlock(&info->ref_lock);
>
> free_array:
> xfree(val_array);
> return ret;
Coming close: Once again, using "goto" on error paths is half way
acceptable to me, while using it anywhere else normally isn't.
Hence you want the "unlock_free_array" path "goto free_array;"
rather than the normal (success) one. Alternatively you might use
a local variable to signal whether to release the lock.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-02 7:26 ` Yi Sun
@ 2017-06-06 7:45 ` Jan Beulich
2017-06-06 8:13 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-06-06 7:45 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 02.06.17 at 09:26, <yi.y.sun@linux.intel.com> wrote:
> On 17-05-31 03:37:48, Jan Beulich wrote:
>> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> > + /* Cut half of cos_max when CDP is enabled. */
>> > + feat->cos_max >>= 1;
>> > +
>> > + /* We only write mask1 since mask0 is always all ones by default. */
>>
>> Is this, btw, just reset state or even guaranteed after offlining
>> and re-onlining a CPU?
>>
> Below MSRs are all per socket. So, we just need reset them when socket is
> online.
Which I hope you've understood then means the comment and
presumably also the code here need further refinement.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-02 7:59 ` Yi Sun
@ 2017-06-06 7:48 ` Jan Beulich
2017-06-06 8:22 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-06-06 7:48 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 02.06.17 at 09:59, <yi.y.sun@linux.intel.com> wrote:
> On 17-05-31 03:44:31, Jan Beulich wrote:
>> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> > @@ -765,7 +777,8 @@ static int insert_val_into_array(uint32_t val[],
>> >
>> > /* Value setting position is same as feature array. */
>> > for ( i = 0; i < props->cos_num; i++ )
>> > - if ( type == props->type[i] )
>> > + if ( type == props->type[i] ||
>> > + (feat_type == PSR_SOCKET_L3_CDP && type == PSR_CBM_TYPE_L3) )
>>
>> Didn't the earlier patch take care of doing this substitution? Non-
>> feature-specific code clearly shouldn't have such special cases if
>> at all avoidable.
>>
> User can set both DATA and CODE to same value at same time with below
> command:
> xl psr-cat-set dom_id 0x3ff
>
> Because no '-c' or '-d' is input, the cbm type will be 'PSR_CBM_TYPE_L3'.
>
> To handle this case, we have to add a special case here. If the cbm tyep is
> 'PSR_CBM_TYPE_L3' and the feature type is CDP, we set both DATA and CODE. This
> should be the simplest way to handle this case.
Simplest or not, it is not really appropriate to have such special cases
here. Along the lines of the earlier abstractions I've recommended
(and which, at least afaic, made the overall series quite a bit more
comprehensible), please re-consider how this can be done without
having special case logic here (I can't immediately suggest an option,
I'm sorry).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-06 7:45 ` Jan Beulich
@ 2017-06-06 8:13 ` Yi Sun
2017-06-06 8:38 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-06-06 8:13 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-06 01:45:11, Jan Beulich wrote:
> >>> On 02.06.17 at 09:26, <yi.y.sun@linux.intel.com> wrote:
> > On 17-05-31 03:37:48, Jan Beulich wrote:
> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> >> > + /* Cut half of cos_max when CDP is enabled. */
> >> > + feat->cos_max >>= 1;
> >> > +
> >> > + /* We only write mask1 since mask0 is always all ones by default. */
> >>
> >> Is this, btw, just reset state or even guaranteed after offlining
> >> and re-onlining a CPU?
> >>
> > Below MSRs are all per socket. So, we just need reset them when socket is
> > online.
>
> Which I hope you've understood then means the comment and
> presumably also the code here need further refinement.
>
Spec states it below. So, the mask0 is guranteed.
"the default mask in IA32_L3_MASK_0 - which is all “1”s (on reset)"
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-06 7:43 ` Jan Beulich
@ 2017-06-06 8:18 ` Yi Sun
2017-06-06 8:39 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-06-06 8:18 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-06 01:43:00, Jan Beulich wrote:
> >>> On 02.06.17 at 04:49, <yi.y.sun@linux.intel.com> wrote:
> > On 17-06-01 04:45:58, Jan Beulich wrote:
> >> >>> On 01.06.17 at 12:00, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-05-30 08:32:59, Jan Beulich wrote:
> >> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> >> >> > --- a/xen/arch/x86/psr.c
> >> >> > +++ b/xen/arch/x86/psr.c
> >> >> > @@ -118,11 +118,13 @@ static const struct feat_props {
> >> >> > * COS ID. Every entry of cos_ref corresponds to one COS ID.
> >> >> > */
> >> >> > struct psr_socket_info {
> >> >> > - bool feat_init;
> >> >> > - spinlock_t ref_lock;
> >> >> > /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
> >> >> > struct feat_node *features[PSR_SOCKET_FEAT_NUM];
> >> >> > + bool feat_init;
> >> >> > unsigned int cos_ref[MAX_COS_REG_CNT];
> >> >> > + spinlock_t ref_lock;
> >> >>
> >> >> This shuffling of fields seems unmotivated and is not being explained
> >> >> in the description.
> >> >>
> >> > Per your comment in v10, such movement may avoid false cacheline conflicts.
> >> > The comment is below.
> >> > Also please try to space apart the two locks, to avoid false cacheline
> >> > conflicts (e.g. the new lock may well go immediately before the array
> >> > it pairs with).
> >>
> >> Well - where is the second lock here?
> >>
> > I thought 'feat_init' has same effect. But I should be wrong.
> >
> > Then, I want to define the structure as below:
> >
> > struct psr_socket_info {
> > bool feat_init;
> > /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
> > struct feat_node *features[PSR_SOCKET_FEAT_NUM];
> > spinlock_t ref_lock;
> > unsigned int cos_ref[MAX_COS_REG_CNT];
> > /* Every bit corresponds to a domain. Index is domain_id. */
> > DECLARE_BITMAP(dom_ids, DOMID_IDLE + 1);
> > };
>
> I've outlined my expectation to the ordering of fields before. The
> above broadly matches that, so would be fine. What I'd like to ask
> though is that fields don't get moved around without reason during
> the series. Insert new fields at their intended final place unless
> there's an actual reason to move them later.
>
Ok, I will be careful about this to put the new field to its final
place when implement it.
> >> >> > + free_array:
> >> >> > + xfree(val_array);
> >> >> > + return ret;
> >> >> > +
> >> >> > + unlock_free_array:
> >> >> > + spin_unlock(&info->ref_lock);
> >> >> > + xfree(val_array);
> >> >> > + return ret;
> >> >> > +}
> >> >>
> >> >> I'm sure I've said so before - please don't duplicate error paths like
> >> >> this. Here it's still easy to see all is fine, but what if each path gets
> >> >> two or three more thing added. Please chain them together via goto.
> >> >>
> >> > To make things clear, I wrote below codes. How about them?
> >> > unlock_free_array:
> >> > spin_unlock(&info->ref_lock);
> >> >
> >> > free_array:
> >> > xfree(val_array);
> >> > return ret;
> >>
> >> I don't think that'll be okay for the case which previously fell
> >> through to free_array.
> >>
> > I tried to understand your meaning. Do you mean below codes?
> >
> > set_bit(d->domain_id, info->dom_ids); //Success path.
> > goto free_array;
> >
> > unlock_free_array:
> > spin_unlock(&info->ref_lock);
> >
> > free_array:
> > xfree(val_array);
> > return ret;
>
> Coming close: Once again, using "goto" on error paths is half way
> acceptable to me, while using it anywhere else normally isn't.
> Hence you want the "unlock_free_array" path "goto free_array;"
> rather than the normal (success) one. Alternatively you might use
> a local variable to signal whether to release the lock.
>
How about this which can avoid a local variable?
set_bit(d->domain_id, info->dom_ids);
free_array:
xfree(val_array);
return ret;
unlock_free_array:
spin_unlock(&info->ref_lock);
goto free_array;
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-06 7:48 ` Jan Beulich
@ 2017-06-06 8:22 ` Yi Sun
2017-06-06 8:43 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-06-06 8:22 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-06 01:48:13, Jan Beulich wrote:
> >>> On 02.06.17 at 09:59, <yi.y.sun@linux.intel.com> wrote:
> > On 17-05-31 03:44:31, Jan Beulich wrote:
> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> >> > @@ -765,7 +777,8 @@ static int insert_val_into_array(uint32_t val[],
> >> >
> >> > /* Value setting position is same as feature array. */
> >> > for ( i = 0; i < props->cos_num; i++ )
> >> > - if ( type == props->type[i] )
> >> > + if ( type == props->type[i] ||
> >> > + (feat_type == PSR_SOCKET_L3_CDP && type == PSR_CBM_TYPE_L3) )
> >>
> >> Didn't the earlier patch take care of doing this substitution? Non-
> >> feature-specific code clearly shouldn't have such special cases if
> >> at all avoidable.
> >>
> > User can set both DATA and CODE to same value at same time with below
> > command:
> > xl psr-cat-set dom_id 0x3ff
> >
> > Because no '-c' or '-d' is input, the cbm type will be 'PSR_CBM_TYPE_L3'.
> >
> > To handle this case, we have to add a special case here. If the cbm tyep is
> > 'PSR_CBM_TYPE_L3' and the feature type is CDP, we set both DATA and CODE. This
> > should be the simplest way to handle this case.
>
> Simplest or not, it is not really appropriate to have such special cases
> here. Along the lines of the earlier abstractions I've recommended
> (and which, at least afaic, made the overall series quite a bit more
> comprehensible), please re-consider how this can be done without
> having special case logic here (I can't immediately suggest an option,
> I'm sorry).
>
How about a callback function here to handle this insertion? For L3/L2 CAT,
use a function just to assign new_val to val[]. For CDP, in its callback
function, check 'type' to decide insert new_val to both DATA and CODE or just
one item according to type.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-06 8:13 ` Yi Sun
@ 2017-06-06 8:38 ` Jan Beulich
2017-06-07 1:31 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-06-06 8:38 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 06.06.17 at 10:13, <yi.y.sun@linux.intel.com> wrote:
> On 17-06-06 01:45:11, Jan Beulich wrote:
>> >>> On 02.06.17 at 09:26, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-05-31 03:37:48, Jan Beulich wrote:
>> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> >> > + /* Cut half of cos_max when CDP is enabled. */
>> >> > + feat->cos_max >>= 1;
>> >> > +
>> >> > + /* We only write mask1 since mask0 is always all ones by default. */
>> >>
>> >> Is this, btw, just reset state or even guaranteed after offlining
>> >> and re-onlining a CPU?
>> >>
>> > Below MSRs are all per socket. So, we just need reset them when socket is
>> > online.
>>
>> Which I hope you've understood then means the comment and
>> presumably also the code here need further refinement.
>>
> Spec states it below. So, the mask0 is guranteed.
> "the default mask in IA32_L3_MASK_0 - which is all “1”s (on reset)"
Sigh. I did ask very clearly (and this is still visible above) about
the case where the CPU did _not_ undergo a reset cycle.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-06-06 8:18 ` Yi Sun
@ 2017-06-06 8:39 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-06-06 8:39 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 06.06.17 at 10:18, <yi.y.sun@linux.intel.com> wrote:
> On 17-06-06 01:43:00, Jan Beulich wrote:
>> >>> On 02.06.17 at 04:49, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-06-01 04:45:58, Jan Beulich wrote:
>> >> >>> On 01.06.17 at 12:00, <yi.y.sun@linux.intel.com> wrote:
>> >> > On 17-05-30 08:32:59, Jan Beulich wrote:
>> >> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> >> >> > + free_array:
>> >> >> > + xfree(val_array);
>> >> >> > + return ret;
>> >> >> > +
>> >> >> > + unlock_free_array:
>> >> >> > + spin_unlock(&info->ref_lock);
>> >> >> > + xfree(val_array);
>> >> >> > + return ret;
>> >> >> > +}
>> >> >>
>> >> >> I'm sure I've said so before - please don't duplicate error paths like
>> >> >> this. Here it's still easy to see all is fine, but what if each path gets
>> >> >> two or three more thing added. Please chain them together via goto.
>> >> >>
>> >> > To make things clear, I wrote below codes. How about them?
>> >> > unlock_free_array:
>> >> > spin_unlock(&info->ref_lock);
>> >> >
>> >> > free_array:
>> >> > xfree(val_array);
>> >> > return ret;
>> >>
>> >> I don't think that'll be okay for the case which previously fell
>> >> through to free_array.
>> >>
>> > I tried to understand your meaning. Do you mean below codes?
>> >
>> > set_bit(d->domain_id, info->dom_ids); //Success path.
>> > goto free_array;
>> >
>> > unlock_free_array:
>> > spin_unlock(&info->ref_lock);
>> >
>> > free_array:
>> > xfree(val_array);
>> > return ret;
>>
>> Coming close: Once again, using "goto" on error paths is half way
>> acceptable to me, while using it anywhere else normally isn't.
>> Hence you want the "unlock_free_array" path "goto free_array;"
>> rather than the normal (success) one. Alternatively you might use
>> a local variable to signal whether to release the lock.
>>
> How about this which can avoid a local variable?
>
> set_bit(d->domain_id, info->dom_ids);
>
> free_array:
> xfree(val_array);
> return ret;
>
> unlock_free_array:
> spin_unlock(&info->ref_lock);
> goto free_array;
Yes, that's what I've been asking for.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-06 8:22 ` Yi Sun
@ 2017-06-06 8:43 ` Jan Beulich
2017-06-06 10:43 ` Yi Sun
0 siblings, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-06-06 8:43 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 06.06.17 at 10:22, <yi.y.sun@linux.intel.com> wrote:
> On 17-06-06 01:48:13, Jan Beulich wrote:
>> >>> On 02.06.17 at 09:59, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-05-31 03:44:31, Jan Beulich wrote:
>> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> >> > @@ -765,7 +777,8 @@ static int insert_val_into_array(uint32_t val[],
>> >> >
>> >> > /* Value setting position is same as feature array. */
>> >> > for ( i = 0; i < props->cos_num; i++ )
>> >> > - if ( type == props->type[i] )
>> >> > + if ( type == props->type[i] ||
>> >> > + (feat_type == PSR_SOCKET_L3_CDP && type == PSR_CBM_TYPE_L3) )
>> >>
>> >> Didn't the earlier patch take care of doing this substitution? Non-
>> >> feature-specific code clearly shouldn't have such special cases if
>> >> at all avoidable.
>> >>
>> > User can set both DATA and CODE to same value at same time with below
>> > command:
>> > xl psr-cat-set dom_id 0x3ff
>> >
>> > Because no '-c' or '-d' is input, the cbm type will be 'PSR_CBM_TYPE_L3'.
>> >
>> > To handle this case, we have to add a special case here. If the cbm tyep is
>> > 'PSR_CBM_TYPE_L3' and the feature type is CDP, we set both DATA and CODE. This
>> > should be the simplest way to handle this case.
>>
>> Simplest or not, it is not really appropriate to have such special cases
>> here. Along the lines of the earlier abstractions I've recommended
>> (and which, at least afaic, made the overall series quite a bit more
>> comprehensible), please re-consider how this can be done without
>> having special case logic here (I can't immediately suggest an option,
>> I'm sorry).
>>
> How about a callback function here to handle this insertion? For L3/L2 CAT,
> use a function just to assign new_val to val[]. For CDP, in its callback
> function, check 'type' to decide insert new_val to both DATA and CODE or just
> one item according to type.
Well, I'm not sure what to say. The history of this series tells me
that you suggesting a new callback is likely to be not better than
having open coded special case logic here. IOW neither is a good
(or should I say preferred) solution here, and I'm relatively
certain (as I had been with all the other callbacks that are now
gone) that there is a reasonably clean solution without either, by
simply using suitable abstracted data structures. As expressed
back then, even if I can't immediately suggest how to make this
work, I'm still insisting that you at least try to come up with a
clean solution here.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-06 8:43 ` Jan Beulich
@ 2017-06-06 10:43 ` Yi Sun
2017-06-06 10:49 ` Jan Beulich
0 siblings, 1 reply; 74+ messages in thread
From: Yi Sun @ 2017-06-06 10:43 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau
On 17-06-06 02:43:42, Jan Beulich wrote:
> >>> On 06.06.17 at 10:22, <yi.y.sun@linux.intel.com> wrote:
> > On 17-06-06 01:48:13, Jan Beulich wrote:
> >> >>> On 02.06.17 at 09:59, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-05-31 03:44:31, Jan Beulich wrote:
> >> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> >> >> > @@ -765,7 +777,8 @@ static int insert_val_into_array(uint32_t val[],
> >> >> >
> >> >> > /* Value setting position is same as feature array. */
> >> >> > for ( i = 0; i < props->cos_num; i++ )
> >> >> > - if ( type == props->type[i] )
> >> >> > + if ( type == props->type[i] ||
> >> >> > + (feat_type == PSR_SOCKET_L3_CDP && type == PSR_CBM_TYPE_L3) )
> >> >>
> >> >> Didn't the earlier patch take care of doing this substitution? Non-
> >> >> feature-specific code clearly shouldn't have such special cases if
> >> >> at all avoidable.
> >> >>
> >> > User can set both DATA and CODE to same value at same time with below
> >> > command:
> >> > xl psr-cat-set dom_id 0x3ff
> >> >
> >> > Because no '-c' or '-d' is input, the cbm type will be 'PSR_CBM_TYPE_L3'.
> >> >
> >> > To handle this case, we have to add a special case here. If the cbm tyep is
> >> > 'PSR_CBM_TYPE_L3' and the feature type is CDP, we set both DATA and CODE. This
> >> > should be the simplest way to handle this case.
> >>
> >> Simplest or not, it is not really appropriate to have such special cases
> >> here. Along the lines of the earlier abstractions I've recommended
> >> (and which, at least afaic, made the overall series quite a bit more
> >> comprehensible), please re-consider how this can be done without
> >> having special case logic here (I can't immediately suggest an option,
> >> I'm sorry).
> >>
> > How about a callback function here to handle this insertion? For L3/L2 CAT,
> > use a function just to assign new_val to val[]. For CDP, in its callback
> > function, check 'type' to decide insert new_val to both DATA and CODE or just
> > one item according to type.
>
> Well, I'm not sure what to say. The history of this series tells me
> that you suggesting a new callback is likely to be not better than
> having open coded special case logic here. IOW neither is a good
> (or should I say preferred) solution here, and I'm relatively
> certain (as I had been with all the other callbacks that are now
> gone) that there is a reasonably clean solution without either, by
> simply using suitable abstracted data structures. As expressed
> back then, even if I can't immediately suggest how to make this
> work, I'm still insisting that you at least try to come up with a
> clean solution here.
>
Ok, I should think more. :)
Then, please check below solution.
This case only happens in CDP mode that the input cbm_type corresponds to L3
CAT but current feat_type is CDP. In all other modes, the input cbm_type
corresponds to its own mode. So, maybe we can implement codes as below.
//Add an input parameter 'bool strict'
static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type, boot strict)
{
...
switch ( type )
{
case PSR_CBM_TYPE_L3:
feat_type = PSR_SOCKET_L3_CAT;
/*
* If type is L3 CAT but we cannot find it in feat_props array,
* try CDP.
*/
if ( !feat_props[feat_type] && !strict )
feat_type = PSR_SOCKET_L3_CDP;
break;
case PSR_CBM_TYPE_L3_DATA:
case PSR_CBM_TYPE_L3_CODE:
feat_type = PSR_SOCKET_L3_CDP;
break;
...
}
//Input feat_type is PSR_SOCKET_L3_CDP, type is PSR_CBM_TYPE_L3
static int insert_val_into_array(feat_type, type)
{
...
for ( i = 0; i < props->cos_num; i++ )
{
if ( type == props->type[i] ||
feat_type != psr_cbm_type_to_feat_type(type, true) )
{
val[i] = new_val;
ret = 0;
}
}
return ret;
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-06-06 10:43 ` Yi Sun
@ 2017-06-06 10:49 ` Jan Beulich
0 siblings, 0 replies; 74+ messages in thread
From: Jan Beulich @ 2017-06-06 10:49 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 06.06.17 at 12:43, <yi.y.sun@linux.intel.com> wrote:
> Ok, I should think more. :)
>
> Then, please check below solution.
>
> This case only happens in CDP mode that the input cbm_type corresponds to L3
> CAT but current feat_type is CDP. In all other modes, the input cbm_type
> corresponds to its own mode. So, maybe we can implement codes as below.
>
> //Add an input parameter 'bool strict'
> static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type, boot
> strict)
> {
> ...
> switch ( type )
> {
> case PSR_CBM_TYPE_L3:
> feat_type = PSR_SOCKET_L3_CAT;
>
> /*
> * If type is L3 CAT but we cannot find it in feat_props array,
> * try CDP.
> */
> if ( !feat_props[feat_type] && !strict )
> feat_type = PSR_SOCKET_L3_CDP;
>
> break;
>
> case PSR_CBM_TYPE_L3_DATA:
> case PSR_CBM_TYPE_L3_CODE:
> feat_type = PSR_SOCKET_L3_CDP;
> break;
> ...
> }
>
> //Input feat_type is PSR_SOCKET_L3_CDP, type is PSR_CBM_TYPE_L3
> static int insert_val_into_array(feat_type, type)
> {
> ...
> for ( i = 0; i < props->cos_num; i++ )
> {
> if ( type == props->type[i] ||
> feat_type != psr_cbm_type_to_feat_type(type, true) )
> {
> val[i] = new_val;
> ret = 0;
> }
> }
>
> return ret;
> }
Thanks, this looks better, but I will want to see this in context of
the next version of the patch series before making up a final
opinion.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-06 8:38 ` Jan Beulich
@ 2017-06-07 1:31 ` Yi Sun
2017-06-07 7:28 ` Yi Sun
2017-06-07 8:14 ` Jan Beulich
0 siblings, 2 replies; 74+ messages in thread
From: Yi Sun @ 2017-06-07 1:31 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-06 02:38:27, Jan Beulich wrote:
> >>> On 06.06.17 at 10:13, <yi.y.sun@linux.intel.com> wrote:
> > On 17-06-06 01:45:11, Jan Beulich wrote:
> >> >>> On 02.06.17 at 09:26, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-05-31 03:37:48, Jan Beulich wrote:
> >> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> >> >> > + /* Cut half of cos_max when CDP is enabled. */
> >> >> > + feat->cos_max >>= 1;
> >> >> > +
> >> >> > + /* We only write mask1 since mask0 is always all ones by default. */
> >> >>
> >> >> Is this, btw, just reset state or even guaranteed after offlining
> >> >> and re-onlining a CPU?
> >> >>
> >> > Below MSRs are all per socket. So, we just need reset them when socket is
> >> > online.
> >>
> >> Which I hope you've understood then means the comment and
> >> presumably also the code here need further refinement.
> >>
> > Spec states it below. So, the mask0 is guranteed.
> > "the default mask in IA32_L3_MASK_0 - which is all “1”s (on reset)"
>
> Sigh. I did ask very clearly (and this is still visible above) about
> the case where the CPU did _not_ undergo a reset cycle.
>
Sorry, I think I must mis-understand your question. SW never modifies value
of COS ID 0 mask register. So, the value in mask0 will not be changed after
socket first online (mask0 is set to default by HW this time). So, we think
it is not necessary to restore mask0 value to default here.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-07 1:31 ` Yi Sun
@ 2017-06-07 7:28 ` Yi Sun
2017-06-07 8:14 ` Jan Beulich
1 sibling, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-06-07 7:28 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau
On 17-06-07 09:31:02, Yi Sun wrote:
> On 17-06-06 02:38:27, Jan Beulich wrote:
> > >>> On 06.06.17 at 10:13, <yi.y.sun@linux.intel.com> wrote:
> > > On 17-06-06 01:45:11, Jan Beulich wrote:
> > >> >>> On 02.06.17 at 09:26, <yi.y.sun@linux.intel.com> wrote:
> > >> > On 17-05-31 03:37:48, Jan Beulich wrote:
> > >> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> > >> >> > + /* Cut half of cos_max when CDP is enabled. */
> > >> >> > + feat->cos_max >>= 1;
> > >> >> > +
> > >> >> > + /* We only write mask1 since mask0 is always all ones by default. */
> > >> >>
> > >> >> Is this, btw, just reset state or even guaranteed after offlining
> > >> >> and re-onlining a CPU?
> > >> >>
> > >> > Below MSRs are all per socket. So, we just need reset them when socket is
> > >> > online.
> > >>
> > >> Which I hope you've understood then means the comment and
> > >> presumably also the code here need further refinement.
> > >>
> > > Spec states it below. So, the mask0 is guranteed.
> > > "the default mask in IA32_L3_MASK_0 - which is all “1”s (on reset)"
> >
> > Sigh. I did ask very clearly (and this is still visible above) about
> > the case where the CPU did _not_ undergo a reset cycle.
> >
> Sorry, I think I must mis-understand your question. SW never modifies value
> of COS ID 0 mask register. So, the value in mask0 will not be changed after
> socket first online (mask0 is set to default by HW this time). So, we think
> it is not necessary to restore mask0 value to default here.
>
But considering something else (e.g. FW) may change this MSR value during
socket offline-online cycle, we may need to restore mask0 here too. Is that
your intention? Thanks!
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-07 1:31 ` Yi Sun
2017-06-07 7:28 ` Yi Sun
@ 2017-06-07 8:14 ` Jan Beulich
2017-06-07 9:00 ` Yi Sun
1 sibling, 1 reply; 74+ messages in thread
From: Jan Beulich @ 2017-06-07 8:14 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 07.06.17 at 03:31, <yi.y.sun@linux.intel.com> wrote:
> On 17-06-06 02:38:27, Jan Beulich wrote:
>> >>> On 06.06.17 at 10:13, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-06-06 01:45:11, Jan Beulich wrote:
>> >> >>> On 02.06.17 at 09:26, <yi.y.sun@linux.intel.com> wrote:
>> >> > On 17-05-31 03:37:48, Jan Beulich wrote:
>> >> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
>> >> >> > + /* Cut half of cos_max when CDP is enabled. */
>> >> >> > + feat->cos_max >>= 1;
>> >> >> > +
>> >> >> > + /* We only write mask1 since mask0 is always all ones by default. */
>> >> >>
>> >> >> Is this, btw, just reset state or even guaranteed after offlining
>> >> >> and re-onlining a CPU?
>> >> >>
>> >> > Below MSRs are all per socket. So, we just need reset them when socket is
>> >> > online.
>> >>
>> >> Which I hope you've understood then means the comment and
>> >> presumably also the code here need further refinement.
>> >>
>> > Spec states it below. So, the mask0 is guranteed.
>> > "the default mask in IA32_L3_MASK_0 - which is all “1”s (on reset)"
>>
>> Sigh. I did ask very clearly (and this is still visible above) about
>> the case where the CPU did _not_ undergo a reset cycle.
>>
> Sorry, I think I must mis-understand your question. SW never modifies value
> of COS ID 0 mask register. So, the value in mask0 will not be changed after
> socket first online (mask0 is set to default by HW this time). So, we think
> it is not necessary to restore mask0 value to default here.
Is the register read-only? If so, writing it would indeed be
unnecessary (and likely wrong, as it might raise #GP then
instead of being ignored). If not, firmware may have written it,
so you'll want to write it even during boot time CPU bringup.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
* Re: [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-06-07 8:14 ` Jan Beulich
@ 2017-06-07 9:00 ` Yi Sun
0 siblings, 0 replies; 74+ messages in thread
From: Yi Sun @ 2017-06-07 9:00 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-06-07 02:14:26, Jan Beulich wrote:
> >>> On 07.06.17 at 03:31, <yi.y.sun@linux.intel.com> wrote:
> > On 17-06-06 02:38:27, Jan Beulich wrote:
> >> >>> On 06.06.17 at 10:13, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-06-06 01:45:11, Jan Beulich wrote:
> >> >> >>> On 02.06.17 at 09:26, <yi.y.sun@linux.intel.com> wrote:
> >> >> > On 17-05-31 03:37:48, Jan Beulich wrote:
> >> >> >> >>> On 03.05.17 at 10:44, <yi.y.sun@linux.intel.com> wrote:
> >> >> >> > + /* Cut half of cos_max when CDP is enabled. */
> >> >> >> > + feat->cos_max >>= 1;
> >> >> >> > +
> >> >> >> > + /* We only write mask1 since mask0 is always all ones by default. */
> >> >> >>
> >> >> >> Is this, btw, just reset state or even guaranteed after offlining
> >> >> >> and re-onlining a CPU?
> >> >> >>
> >> >> > Below MSRs are all per socket. So, we just need reset them when socket is
> >> >> > online.
> >> >>
> >> >> Which I hope you've understood then means the comment and
> >> >> presumably also the code here need further refinement.
> >> >>
> >> > Spec states it below. So, the mask0 is guranteed.
> >> > "the default mask in IA32_L3_MASK_0 - which is all “1”s (on reset)"
> >>
> >> Sigh. I did ask very clearly (and this is still visible above) about
> >> the case where the CPU did _not_ undergo a reset cycle.
> >>
> > Sorry, I think I must mis-understand your question. SW never modifies value
> > of COS ID 0 mask register. So, the value in mask0 will not be changed after
> > socket first online (mask0 is set to default by HW this time). So, we think
> > it is not necessary to restore mask0 value to default here.
>
> Is the register read-only? If so, writing it would indeed be
> unnecessary (and likely wrong, as it might raise #GP then
> instead of being ignored). If not, firmware may have written it,
> so you'll want to write it even during boot time CPU bringup.
>
No, it is writable. I will restore mask0 to default value here.
Furthermore, for L3/L2 CAT mask0, we may need to restore mask0
too.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 74+ messages in thread
end of thread, other threads:[~2017-06-07 9:01 UTC | newest]
Thread overview: 74+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-03 8:44 [PATCH v11 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-05-03 8:44 ` [PATCH v11 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-05-03 8:44 ` [PATCH v11 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
2017-05-03 8:44 ` [PATCH v11 03/23] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
2017-05-03 8:44 ` [PATCH v11 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
2017-05-30 13:05 ` Jan Beulich
2017-05-31 2:44 ` Yi Sun
2017-05-31 6:02 ` Jan Beulich
2017-05-31 9:36 ` Jan Beulich
2017-06-01 3:18 ` Yi Sun
2017-06-01 6:57 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
2017-05-30 13:26 ` Jan Beulich
2017-05-31 6:37 ` Yi Sun
2017-05-31 6:57 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 06/23] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
2017-05-30 13:51 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
2017-05-30 14:05 ` Jan Beulich
2017-05-31 7:30 ` Yi Sun
2017-05-31 7:45 ` Jan Beulich
2017-05-31 8:05 ` Yi Sun
2017-05-31 8:10 ` Jan Beulich
2017-06-01 3:14 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
2017-05-30 14:32 ` Jan Beulich
2017-06-01 10:00 ` Yi Sun
2017-06-01 10:45 ` Jan Beulich
2017-06-02 2:49 ` Yi Sun
2017-06-06 7:43 ` Jan Beulich
2017-06-06 8:18 ` Yi Sun
2017-06-06 8:39 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
2017-05-30 15:17 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
2017-05-30 15:23 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
2017-05-03 8:44 ` [PATCH v11 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
2017-05-30 15:35 ` Jan Beulich
2017-06-05 8:10 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
2017-05-31 9:37 ` Jan Beulich
2017-06-02 7:26 ` Yi Sun
2017-06-06 7:45 ` Jan Beulich
2017-06-06 8:13 ` Yi Sun
2017-06-06 8:38 ` Jan Beulich
2017-06-07 1:31 ` Yi Sun
2017-06-07 7:28 ` Yi Sun
2017-06-07 8:14 ` Jan Beulich
2017-06-07 9:00 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
2017-05-31 9:40 ` Jan Beulich
2017-06-05 8:09 ` Yi Sun
2017-05-03 8:44 ` [PATCH v11 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
2017-05-31 9:44 ` Jan Beulich
2017-06-02 7:59 ` Yi Sun
2017-06-06 7:48 ` Jan Beulich
2017-06-06 8:22 ` Yi Sun
2017-06-06 8:43 ` Jan Beulich
2017-06-06 10:43 ` Yi Sun
2017-06-06 10:49 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
2017-05-03 8:44 ` [PATCH v11 17/23] x86: L2 CAT: implement get hw info flow Yi Sun
2017-05-03 8:44 ` [PATCH v11 18/23] x86: L2 CAT: implement get value flow Yi Sun
2017-05-31 9:51 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 19/23] x86: L2 CAT: implement set " Yi Sun
2017-05-31 9:52 ` Jan Beulich
2017-05-03 8:44 ` [PATCH v11 20/23] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
2017-05-11 12:39 ` Wei Liu
2017-05-03 8:44 ` [PATCH v11 21/23] tools: L2 CAT: support show cbm " Yi Sun
2017-05-11 12:40 ` Wei Liu
2017-05-03 8:44 ` [PATCH v11 22/23] tools: L2 CAT: support set " Yi Sun
2017-05-11 12:43 ` Wei Liu
2017-05-03 8:44 ` [PATCH v11 23/23] docs: add L2 CAT description in docs Yi Sun
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.