* [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c
@ 2017-02-15 8:49 Yi Sun
2017-02-15 8:49 ` [PATCH v8 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
` (26 more replies)
0 siblings, 27 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
Hi all,
We plan to bring a new PSR (Platform Shared Resource) feature called
Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
Besides the L2 CAT implementaion, we refactor the psr.c to make it more
flexible to add new features and fulfill the principle, open for extension
but closed for modification. We abstract the general operations of all
features and encapsulate them into a structure. Then, the development
of new feature is simple to mainly implement these callback functions.
The patch set can be found at:
https://github.com/yisun-git/xen.git l2_cat_v8
v7:
- patch 1:
- change revision info.
- add content int 'Areas for improvement'.
- patch 4:
- remove unused parameter of psr_cpu_prepare.
- add comments in cpu_fini_work.
- coding style fix.
- patch 12:
- modify 'write_msr' callback function to 'void' because we have to set
all features' cbm. When input cos exceeds some features' cos_max, just
skip them but not break the iteration.
- patch 16:
- modify 'l3_cdp_write_msr' to 'void'.
- patch 20:
- modify 'l2_cat_write_msr' to 'void'.
Yi Sun (24):
docs: create Cache Allocation Technology (CAT) and Code and Data
Prioritization (CDP) feature document
x86: refactor psr: remove L3 CAT/CDP codes.
x86: refactor psr: implement main data structures.
x86: refactor psr: implement CPU init and free flow.
x86: refactor psr: implement Domain init/free and schedule flows.
x86: refactor psr: implement get hw info flow.
x86: refactor psr: implement get value flow.
x86: refactor psr: set value: implement framework.
x86: refactor psr: set value: assemble features value array.
x86: refactor psr: set value: implement cos finding flow.
x86: refactor psr: set value: implement cos id picking flow.
x86: refactor psr: set value: implement write msr flow.
x86: refactor psr: implement CPU init and free flow for CDP.
x86: refactor psr: implement get hw info flow for CDP.
x86: refactor psr: implement get value flow for CDP.
x86: refactor psr: implement set value callback functions for CDP.
x86: L2 CAT: implement CPU init and free flow.
x86: L2 CAT: implement get hw info flow.
x86: L2 CAT: implement get value flow.
x86: L2 CAT: implement set value flow.
tools: L2 CAT: support get HW info for L2 CAT.
tools: L2 CAT: support show cbm for L2 CAT.
tools: L2 CAT: support set cbm for L2 CAT.
docs: add L2 CAT description in docs.
docs/features/intel_psr_cat_cdp.pandoc | 469 +++++++++
docs/man/xl.pod.1.in | 25 +-
docs/misc/xl-psr.markdown | 10 +-
tools/libxc/include/xenctrl.h | 7 +-
tools/libxc/xc_psr.c | 45 +-
tools/libxl/libxl.h | 9 +
tools/libxl/libxl_psr.c | 19 +-
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 169 +++-
tools/libxl/xl_cmdtable.c | 6 +-
xen/arch/x86/cpuid.c | 6 -
xen/arch/x86/domctl.c | 49 +-
xen/arch/x86/psr.c | 1620 ++++++++++++++++++++++++++------
xen/arch/x86/sysctl.c | 45 +-
xen/include/asm-x86/msr-index.h | 1 +
xen/include/asm-x86/processor.h | 7 +
xen/include/asm-x86/psr.h | 29 +-
xen/include/public/domctl.h | 2 +
xen/include/public/sysctl.h | 6 +
19 files changed, 2112 insertions(+), 413 deletions(-)
create mode 100644 docs/features/intel_psr_cat_cdp.pandoc
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v8 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-15 16:49 ` Konrad Rzeszutek Wilk
2017-02-26 17:40 ` Wei Liu
2017-02-15 8:49 ` [PATCH v8 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
` (25 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch creates CAT and CDP feature document in doc/features/. It describes
key points to implement L3 CAT/CDP and L2 CAT which is described in details in
Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES".
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v8:
- change revision info.
- add content int 'Areas for improvement'.
---
docs/features/intel_psr_cat_cdp.pandoc | 469 +++++++++++++++++++++++++++++++++
1 file changed, 469 insertions(+)
create mode 100644 docs/features/intel_psr_cat_cdp.pandoc
diff --git a/docs/features/intel_psr_cat_cdp.pandoc b/docs/features/intel_psr_cat_cdp.pandoc
new file mode 100644
index 0000000..f46b41b
--- /dev/null
+++ b/docs/features/intel_psr_cat_cdp.pandoc
@@ -0,0 +1,469 @@
+% Intel Cache Allocation Technology and Code and Data Prioritization Features
+% Revision 1.8
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+ Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+ Component(s): Hypervisor, toolstack
+
+ Hardware: L3 CAT: Haswell and beyond CPUs
+ CDP : Broadwell and beyond CPUs
+ L2 CAT: Atom codename Goldmont and beyond CPUs
+---------------- ----------------------------------------------------
+
+# Terminology
+
+* CAT Cache Allocation Technology
+* CBM Capacity BitMasks
+* CDP Code and Data Prioritization
+* COS/CLOS Class of Service
+* MSRs Machine Specific Registers
+* PSR Intel Platform Shared Resource
+
+# Overview
+
+Intel provides a set of allocation capabilities including Cache Allocatation
+Technology (CAT) and Code and Data Prioritization (CDP).
+
+CAT allows an OS or hypervisor to control allocation of a CPU's shared cache
+based on application/domain priority or Class of Service (COS). Each COS is
+configured using capacity bitmasks (CBMs) which represent cache capacity and
+indicate the degree of overlap and isolation between classes. Once CAT is co-
+nfigured, the processor allows access to portions of cache according to the
+established COS. Intel Xeon processor E5 v4 family (and some others) introduce
+capabilities to configure and make use of the CAT mechanism on the L3 cache.
+Intel Goldmont processor provides support for control over the L2 cache.
+
+Code and Data Prioritization (CDP) Technology is an extension of CAT. CDP
+enables isolation and separate prioritization of code and data fetches to
+the L3 cache in a SW configurable manner, which can enable workload priorit-
+ization and tuning of cache capacity to the characteristics of the workload.
+CDP extends CAT by providing separate code and data masks per Class of Service
+(COS). When SW configures to enable CDP, L3 CAT is disabled.
+
+# User details
+
+* Feature Enabling:
+
+ Add "psr=cat" to boot line parameter to enable all supported level CAT featu-
+ res. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW.
+
+* xl interfaces:
+
+ 1. `psr-cat-show [OPTIONS] domain-id`:
+
+ Show L2 CAT or L3 CAT/CDP CBM of the domain designated by Xen domain-id.
+
+ Option `-l`:
+ `-l2`: Show cbm for L2 cache.
+ `-l3`: Show cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ 2. `psr-cat-set [OPTIONS] domain-id cbm`:
+
+ Set L2 CAT or L3 CAT/CDP CBM to the domain designated by Xen domain-id.
+
+ Option `-s`: Specify the socket to process, otherwise all sockets are
+ processed.
+
+ Option `-l`:
+ `-l2`: Specify cbm for L2 cache.
+ `-l3`: Specify cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ Option `-c` or `-d`:
+ `-c`: Set L3 CDP code cbm.
+ `-d`: Set L3 CDP data cbm.
+
+ 3. `psr-hwinfo [OPTIONS]`:
+
+ Show CMT & L2 CAT & L3 CAT/CDP HW information on every socket.
+
+ Option `-m, --cmt`: Show Cache Monitoring Technology (CMT) hardware info.
+
+ Option `-a, --cat`: Show CAT/CDP hardware info.
+
+# Technical details
+
+L3 CAT/CDP and L2 CAT are all members of Intel PSR features, they share the base
+PSR infrastructure in Xen.
+
+## Hardware perspective
+
+ CAT/CDP defines a range of MSRs to assign different cache access patterns
+ which are known as CBMs, each CBM is associated with a COS.
+
+ ```
+ E.g. L2 CAT:
+ +----------------------------+----------------+
+ IA32_PQR_ASSOC | MSR (per socket) | Address |
+ +----+---+-------+ +----------------------------+----------------+
+ | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
+ +----+---+-------+ +----------------------------+----------------+
+ └-------------> | ... | ... |
+ +----------------------------+----------------+
+ | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
+ +----------------------------+----------------+
+ ```
+
+ L3 CAT/CDP uses a range of MSRs from 0xC90 ~ 0xC90+n (n<128).
+
+ L2 CAT uses a range of MSRs from 0xD10 ~ 0xD10+n (n<64), following the L3
+ CAT/CDP MSRs, setting different L2 cache accessing patterns from L3 cache is
+ supported.
+
+ Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the
+ hardware indicating the cache space a domain should be limited to as well as
+ providing an indication of overlap and isolation in the CAT-capable cache from
+ other domains contending for the cache.
+
+ Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
+ note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
+ 0FF0H, 003CH, etc.).
+
+ ```
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Default Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Overlapped Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | | | | | Isolated Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | | |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+ ```
+
+ We can get the CBM length through CPUID. The default value of CBM is calcul-
+ ated by `(1ull << cbm_len) - 1`. That is a fully open bitmask, all ones bitm-
+ ask. The COS[0] always stores the default value without change.
+
+ There is a `IA32_PQR_ASSOC` register which stores the COS ID of the VCPU. HW
+ enforces cache allocation according to the corresponding CBM.
+
+## The relationship between L3 CAT/CDP and L2 CAT
+
+ HW may support all features. By default, CDP is disabled on the processor.
+ If the L3 CAT MSRs are used without enabling CDP, the processor operates in
+ a traditional CAT-only mode. When CDP is enabled:
+ * the CAT mask MSRs are re-mapped into interleaved pairs of mask MSRs for
+ data or code fetches.
+ * the range of COS for CAT is re-indexed, with the lower-half of the COS
+ range available for CDP.
+
+ L2 CAT is independent of L3 CAT/CDP, which means L2 CAT can be enabled while
+ L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are both enabled.
+
+ As a requirement, the bits of CBM of CAT/CDP must be continuous.
+
+ N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same associate
+ register `IA32_PQR_ASSOC`, which means one COS is associated with a pair of
+ L2 CAT CBM and L3 CAT/CDP CBM.
+
+ Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or other
+ PSR features in future). In some cases, a domain is permitted to have a COS
+ that is beyond one (or more) of PSR features but within the others. For
+ instance, let's assume the max COS of L2 CAT is 8 but the max COS of L3
+ CAT is 16, when a domain is assigned 9 as COS, the L3 CAT CBM associated to
+ COS 9 would be enforced, but for L2 CAT, the HW works as default value is
+ set since COS 9 is beyond the max COS (8) of L2 CAT.
+
+## Design Overview
+
+* Core COS/CBM association
+
+ When enforcing CAT/CDP, all cores of domains have the same default COS (COS0)
+ which is associated with the fully open CBM (all ones bitmask) to access all
+ cache. The default COS is used only in hypervisor and is transparent to tool
+ stack and user.
+
+ System administrator can change PSR allocation policy at runtime by tool stack.
+ Since L2 CAT shares COS with L3 CAT/CDP, a COS corresponds to a 2-tuple, like
+ [L2 CBM, L3 CBM] with only-CAT enabled, when CDP is enabled, one COS correspo-
+ nds to a 3-tuple, like [L2 CBM, L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT
+ nor L3 CDP is enabled, things would be easier, one COS corresponds to one L2
+ CBM.
+
+* VCPU schedule
+
+ When context switch happens, the COS of VCPU is written to per-thread MSR
+ `IA32_PQR_ASSOC`, and then hardware enforces cache allocation according to
+ the corresponding CBM.
+
+* Multi-sockets
+
+ Different sockets may have different CAT/CDP capability (e.g. max COS) alth-
+ ough it is consistent on the same socket. So the capability of per-socket CAT/
+ CDP is specified.
+
+ 'psr-cat-set' can set CBM for one domain per socket. On each socket, we main-
+ tain a COS array for all domains. One domain uses one COS at one time. One COS
+ stores the CBM of the domain to work. So, when a VCPU of the domain is migrat-
+ ed from socket 1 to socket 2, it follows configuration on socket 2.
+
+ E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets do-
+ main 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
+ is migrated from socket 1 to 2, the COS ID used is 7, that means 0x3f is the
+ CBM to work for this domain 1 now.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+ 1. Boot line parameter "psr=cat" enables L2 CAT and L3 CAT if hardware suppo-
+ rted. "psr=cdp" enables CDP if hardware supported.
+
+ 2. SYSCTL:
+ - XEN_SYSCTL_PSR_CAT_get_l3_info: Get L3 CAT/CDP information.
+ - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
+
+ 3. DOMCTL:
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: Get L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: Set L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE: Get CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: Set CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA: Get CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: Set CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
+
+* xl interfaces:
+
+ 1. psr-cat-show -lX domain-id
+ Show LX cbm for a domain.
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
+
+ 2. psr-cat-set -lX domain-id cbm
+ Set LX cbm for a domain.
+ => XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
+
+ 3. psr-hwinfo
+ Show PSR HW information, including L3 CAT/CDP/L2 CAT
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info
+
+* Key data structure:
+
+ 1. Feature HW info
+
+ ```
+ struct psr_cat_hw_info {
+ unsigned int cbm_len;
+ unsigned int cos_max;
+ };
+ ```
+
+ - Member `cbm_len`
+
+ `cbm_len` is one of the hardware info of CAT. It means the max number
+ of bits to set.
+
+ - Member `cos_max`
+
+ `cos_max` is one of the hardware info of CAT. It means the max number
+ of COS registers.
+
+ 2. Feature list node
+
+ ```
+ struct feat_node {
+ enum psr_feat_type feature;
+ struct feat_ops ops;
+ struct psr_cat_hw_info info;
+ uint64_t cos_reg_val[MAX_COS_REG_NUM];
+ struct list_head list;
+ };
+ ```
+
+ When a PSR enforcement feature is enabled, it will be added into a
+ feature list. The head of the list is created in psr initialization.
+
+ - Member `feature`
+
+ `feature` is an integer number, to indicate which feature the list entry
+ corresponds to.
+
+ - Member `ops`
+
+ `ops` maintains a callback function list of the feature. It will be introduced
+ in details later at [Feature operation functions structure].
+
+ - Member `info`
+
+ `info` maintains the feature HW information which are provided to psr_hwinfo
+ command.
+
+ - Member `cos_reg_val`
+
+ `cos_reg_val` is an array to maintain the value set in all COS registers of
+ the feature. The array is indexed by COS ID.
+
+ 3. Per-socket PSR features information structure
+
+ ```
+ struct psr_socket_info {
+ unsigned int feat_mask;
+ unsigned int nr_feat;
+ struct list_head feat_list;
+ unsigned int cos_ref[MAX_COS_REG_NUM];
+ spinlock_t ref_lock;
+ };
+ ```
+
+ We collect all PSR allocation features information of a socket in this
+ `struct psr_socket_info`.
+
+ - Member `feat_mask`
+
+ `feat_mask` is a bitmap, to indicate which feature is enabled on current
+ socket. See values defined in 'enum psr_feat_type'. E.g.
+
+ bit 0: L3 CAT status.
+ bit 1: L3 CDP status.
+ bit 2: L2 CAT status.
+
+ - Member `nr_feat`
+
+ `nr_feat` means the number of PSR features enabled.
+
+ - Member `cos_ref`
+
+ `cos_ref` is an array which maintains the reference of one COS. It maps
+ to cos_reg_val[MAX_COS_REG_NUM] in `struct feat_node`. If one COS is
+ used by one domain, the corresponding reference will increase by one. If
+ a domain releases the COS, the reference will decrease by one. The array
+ is indexed by COS ID.
+
+ 4. Feature operation functions structure
+
+ ```
+ struct feat_ops {
+ unsigned int (*get_cos_max)(const struct feat_node *feat);
+ int (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], uint32_t array_len);
+ int (*get_val)(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val);
+ unsigned int (*get_cos_num)(const struct feat_node *feat);
+ int (*get_old_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos);
+ int (*set_new_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos,
+ enum cbm_type type,
+ uint64_t m);
+ int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
+ unsigned int cos, bool *found);
+ unsigned int (*fits_cos_max)(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos);
+ int (*write_msr)(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat);
+ };
+ ```
+
+ We abstract above callback functions to encapsulate the feature specific
+ behaviors into them. Then, it is easy to add a new feature. We just need:
+ 1) Implement such ops and callback functions for every feature.
+ 2) Register the ops into `struct feat_node`.
+ 3) Add the feature into feature list during CPU initialization.
+
+# Limitations
+
+CAT/CDP can only work on HW which enables it(check by CPUID). So far, there is
+no HW which enables both L2 CAT and L3 CAT/CDP. But SW implementation has cons-
+idered such scenario to enable both L2 CAT and L3 CAT/CDP.
+
+# Testing
+
+We can execute above xl commands to verify L2 CAT and L3 CAT/CDP on different
+HWs support them.
+
+For example:
+ root@:~$ xl psr-hwinfo --cat
+ Cache Allocation Technology (CAT): L2
+ Socket ID : 0
+ Maximum COS : 3
+ CBM length : 8
+ Default CBM : 0xff
+
+ root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
+
+ root@:~$ xl psr-cat-show -l2 1
+ Socket ID : 0
+ Default CBM : 0xff
+ ID NAME CBM
+ 1 ubuntu14 0x7f
+
+# Areas for improvement
+
+A hexadecimal number is used to set/show CBM for a domain now. Although this
+is convenient to cover overlap/isolated bitmask requirement, it is not user-
+friendly.
+
+To improve this, the libxl interfaces can be wrapped in libvirt to provide more
+usr-friendly interfaces to user, e.g. a percentage number of the cache to set
+and show.
+
+# Known issues
+
+N/A
+
+# References
+
+"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date Revision Version Notes
+---------- -------- -------- -------------------------------------------
+2016-08-12 1.0 Xen 4.9 Design document written
+2017-02-13 1.7 Xen 4.9 Changes:
+ 1. Modify the design document to cover L3
+ CAT/CDP and L2 CAT;
+ 2. Fix typos;
+ 3. Amend description of `feat_mask` to make
+ it clearer;
+ 4. Other minor changes.
+2017-02-15 1.8 Xen 4.9 Changes:
+ 1. Add content in 'Areas for improvement';
+ 2. Adjust revision number.
+---------- -------- -------- -------------------------------------------
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 02/24] x86: refactor psr: remove L3 CAT/CDP codes.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-02-15 8:49 ` [PATCH v8 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:40 ` Wei Liu
2017-02-15 8:49 ` [PATCH v8 03/24] x86: refactor psr: implement main data structures Yi Sun
` (24 subsequent siblings)
26 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
The current cache allocation codes in psr.c do not consider
future features addition and are not friendly to extend.
To make psr.c be more flexible to add new features and fulfill
the program principle, open for extension but closed for
modification, we have to refactor the psr.c:
1. Analyze cache allocation features and abstract general data
structures.
2. Analyze the init and all other functions flow, abstract all
steps that different features may have different implementations.
Make these steps be callback functions and register feature
specific fuctions. Then, the main processes will not be changed
when introducing a new feature.
Because the quantity of refactor codes is big and the logics are
changed a lot, it will cause reviewers confused if just change
old codes. Reviewers have to understand both old codes and new
implementations. After review iterations from V1 to V3, Jan has
proposed to remove all old cache allocation codes firstly, then
implement new codes step by step. This will help to make codes
be more easily reviewable.
There is no construction without destruction. So, this patch
removes all current L3 CAT/CDP codes in psr.c. The following
patches will introduce the new mechanism.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
xen/arch/x86/psr.c | 470 +----------------------------------------------------
1 file changed, 5 insertions(+), 465 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0b5073c..96a8589 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -23,24 +23,6 @@
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
-struct psr_cat_cbm {
- union {
- uint64_t cbm;
- struct {
- uint64_t code;
- uint64_t data;
- };
- };
- unsigned int ref;
-};
-
-struct psr_cat_socket_info {
- unsigned int cbm_len;
- unsigned int cos_max;
- struct psr_cat_cbm *cos_to_cbm;
- spinlock_t cbm_lock;
-};
-
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -48,26 +30,11 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
-static unsigned long *__read_mostly cat_socket_enable;
-static struct psr_cat_socket_info *__read_mostly cat_socket_info;
-static unsigned long *__read_mostly cdp_socket_enable;
-
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
-static unsigned int __read_mostly opt_cos_max = 255;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
-static struct psr_cat_cbm *temp_cos_to_cbm;
-
-static unsigned int get_socket_cpu(unsigned int socket)
-{
- if ( likely(socket < nr_sockets) )
- return cpumask_any(socket_cpumask[socket]);
-
- return nr_cpu_ids;
-}
-
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -107,9 +74,6 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
- if ( val_str && !strcmp(s, "cos_max") )
- opt_cos_max = simple_strtoul(val_str, NULL, 0);
-
s = ss + 1;
} while ( ss );
}
@@ -213,16 +177,7 @@ static inline void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( cat_socket_info )
- {
- unsigned int socket = cpu_to_socket(smp_processor_id());
-
- if ( test_bit(socket, cat_socket_enable) )
- psra->cos_mask = ((1ull << get_count_order(
- cat_socket_info[socket].cos_max)) - 1) << 32;
- }
-
- if ( psr_cmt_enabled() || psra->cos_mask )
+ if ( psr_cmt_enabled() )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -231,12 +186,6 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
-static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
- uint64_t cos_mask)
-{
- *reg = (*reg & ~cos_mask) | (((uint64_t)cos << 32) & cos_mask);
-}
-
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -245,459 +194,54 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
- if ( psra->cos_mask )
- psr_assoc_cos(®, d->arch.psr_cos_ids ?
- d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
- 0, psra->cos_mask);
-
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
psra->val = reg;
}
}
-static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
-{
- if ( !cat_socket_info )
- return ERR_PTR(-ENODEV);
-
- if ( socket >= nr_sockets )
- return ERR_PTR(-ENOTSOCK);
-
- if ( !test_bit(socket, cat_socket_enable) )
- return ERR_PTR(-ENOENT);
-
- return cat_socket_info + socket;
-}
-
-static inline bool_t cdp_is_enabled(unsigned int socket)
-{
- return cdp_socket_enable && test_bit(socket, cdp_socket_enable);
-}
int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
uint32_t *cos_max, uint32_t *flags)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- *cbm_len = info->cbm_len;
- *cos_max = info->cos_max;
-
- *flags = 0;
- if ( cdp_is_enabled(socket) )
- *flags |= XEN_SYSCTL_PSR_CAT_L3_CDP;
-
return 0;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
- bool_t cdp_enabled = cdp_is_enabled(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- if ( cdp_enabled )
- return -EXDEV;
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].code;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].data;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- }
-
- return 0;
-}
-
-static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
-{
- unsigned int first_bit, zero_bit;
-
- /* Set bits should only in the range of [0, cbm_len). */
- if ( cbm & (~0ull << cbm_len) )
- return 0;
-
- /* At least one bit need to be set. */
- if ( cbm == 0 )
- return 0;
-
- first_bit = find_first_bit(&cbm, cbm_len);
- zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
-
- /* Set bits should be contiguous. */
- if ( zero_bit < cbm_len &&
- find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
- return 0;
-
- return 1;
-}
-
-struct cos_cbm_info
-{
- unsigned int cos;
- bool_t cdp;
- uint64_t cbm_code;
- uint64_t cbm_data;
-};
-
-static void do_write_l3_cbm(void *data)
-{
- struct cos_cbm_info *info = data;
-
- if ( info->cdp )
- {
- wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(info->cos), info->cbm_code);
- wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(info->cos), info->cbm_data);
- }
- else
- wrmsrl(MSR_IA32_PSR_L3_MASK(info->cos), info->cbm_code);
-}
-
-static int write_l3_cbm(unsigned int socket, unsigned int cos,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp)
-{
- struct cos_cbm_info info =
- {
- .cos = cos,
- .cbm_code = cbm_code,
- .cbm_data = cbm_data,
- .cdp = cdp,
- };
-
- if ( socket == cpu_to_socket(smp_processor_id()) )
- do_write_l3_cbm(&info);
- else
- {
- unsigned int cpu = get_socket_cpu(socket);
-
- if ( cpu >= nr_cpu_ids )
- return -ENOTSOCK;
- on_selected_cpus(cpumask_of(cpu), do_write_l3_cbm, &info, 1);
- }
-
return 0;
}
-static int find_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
-{
- unsigned int cos;
-
- for ( cos = 0; cos <= cos_max; cos++ )
- {
- if ( (map[cos].ref || cos == 0) &&
- ((!cdp_enabled && map[cos].cbm == cbm_code) ||
- (cdp_enabled && map[cos].code == cbm_code &&
- map[cos].data == cbm_data)) )
- return cos;
- }
-
- return -ENOENT;
-}
-
-static int pick_avail_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- unsigned int old_cos)
-{
- unsigned int cos;
-
- /* If old cos is referred only by the domain, then use it. */
- if ( map[old_cos].ref == 1 && old_cos != 0 )
- return old_cos;
-
- /* Find an unused one other than cos0. */
- for ( cos = 1; cos <= cos_max; cos++ )
- if ( map[cos].ref == 0 )
- return cos;
-
- return -ENOENT;
-}
-
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type)
{
- unsigned int old_cos, cos_max;
- int cos, ret;
- uint64_t cbm_data, cbm_code;
- bool_t cdp_enabled = cdp_is_enabled(socket);
- struct psr_cat_cbm *map;
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- if ( !psr_check_cbm(info->cbm_len, cbm) )
- return -EINVAL;
-
- if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
- type == PSR_CBM_TYPE_L3_DATA) )
- return -ENXIO;
-
- cos_max = info->cos_max;
- old_cos = d->arch.psr_cos_ids[socket];
- map = info->cos_to_cbm;
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- cbm_code = cbm;
- cbm_data = cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- cbm_code = cbm;
- cbm_data = map[old_cos].data;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- cbm_code = map[old_cos].code;
- cbm_data = cbm;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- return -EINVAL;
- }
-
- spin_lock(&info->cbm_lock);
- cos = find_cos(map, cos_max, cbm_code, cbm_data, cdp_enabled);
- if ( cos >= 0 )
- {
- if ( cos == old_cos )
- {
- spin_unlock(&info->cbm_lock);
- return 0;
- }
- }
- else
- {
- cos = pick_avail_cos(map, cos_max, old_cos);
- if ( cos < 0 )
- {
- spin_unlock(&info->cbm_lock);
- return cos;
- }
-
- /* We try to avoid writing MSR. */
- if ( (cdp_enabled &&
- (map[cos].code != cbm_code || map[cos].data != cbm_data)) ||
- (!cdp_enabled && map[cos].cbm != cbm_code) )
- {
- ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
- if ( ret )
- {
- spin_unlock(&info->cbm_lock);
- return ret;
- }
- map[cos].code = cbm_code;
- map[cos].data = cbm_data;
- }
- }
-
- map[cos].ref++;
- map[old_cos].ref--;
- spin_unlock(&info->cbm_lock);
-
- d->arch.psr_cos_ids[socket] = cos;
-
return 0;
}
-/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
-static void psr_free_cos(struct domain *d)
-{
- unsigned int socket;
- unsigned int cos;
- struct psr_cat_socket_info *info;
-
- if( !d->arch.psr_cos_ids )
- return;
-
- for_each_set_bit(socket, cat_socket_enable, nr_sockets)
- {
- if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
- continue;
-
- info = cat_socket_info + socket;
- spin_lock(&info->cbm_lock);
- info->cos_to_cbm[cos].ref--;
- spin_unlock(&info->cbm_lock);
- }
-
- xfree(d->arch.psr_cos_ids);
- d->arch.psr_cos_ids = NULL;
-}
-
int psr_domain_init(struct domain *d)
{
- if ( cat_socket_info )
- {
- d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
- if ( !d->arch.psr_cos_ids )
- return -ENOMEM;
- }
-
return 0;
}
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
- psr_free_cos(d);
-}
-
-static int cat_cpu_prepare(unsigned int cpu)
-{
- if ( !cat_socket_info )
- return 0;
-
- if ( temp_cos_to_cbm == NULL &&
- (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
- opt_cos_max + 1UL)) == NULL )
- return -ENOMEM;
-
- return 0;
-}
-
-static void cat_cpu_init(void)
-{
- unsigned int eax, ebx, ecx, edx;
- struct psr_cat_socket_info *info;
- unsigned int socket;
- unsigned int cpu = smp_processor_id();
- uint64_t val;
- const struct cpuinfo_x86 *c = cpu_data + cpu;
-
- if ( !cpu_has(c, X86_FEATURE_PQE) || c->cpuid_level < PSR_CPUID_LEVEL_CAT )
- return;
-
- socket = cpu_to_socket(cpu);
- if ( test_bit(socket, cat_socket_enable) )
- return;
-
- cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
- if ( ebx & PSR_RESOURCE_TYPE_L3 )
- {
- cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
- info = cat_socket_info + socket;
- info->cbm_len = (eax & 0x1f) + 1;
- info->cos_max = min(opt_cos_max, edx & 0xffff);
-
- info->cos_to_cbm = temp_cos_to_cbm;
- temp_cos_to_cbm = NULL;
- /* cos=0 is reserved as default cbm(all ones). */
- info->cos_to_cbm[0].cbm = (1ull << info->cbm_len) - 1;
-
- spin_lock_init(&info->cbm_lock);
-
- set_bit(socket, cat_socket_enable);
-
- if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
- cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
- {
- info->cos_to_cbm[0].code = (1ull << info->cbm_len) - 1;
- info->cos_to_cbm[0].data = (1ull << info->cbm_len) - 1;
-
- /* We only write mask1 since mask0 is always all ones by default. */
- wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->cbm_len) - 1);
-
- rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
- wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
-
- /* Cut half of cos_max when CDP is enabled. */
- info->cos_max >>= 1;
-
- set_bit(socket, cdp_socket_enable);
- }
- printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
- socket, info->cos_max, info->cbm_len,
- cdp_is_enabled(socket) ? "on" : "off");
- }
-}
-
-static void cat_cpu_fini(unsigned int cpu)
-{
- unsigned int socket = cpu_to_socket(cpu);
-
- if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
- {
- struct psr_cat_socket_info *info = cat_socket_info + socket;
-
- if ( info->cos_to_cbm )
- {
- xfree(info->cos_to_cbm);
- info->cos_to_cbm = NULL;
- }
-
- if ( cdp_is_enabled(socket) )
- clear_bit(socket, cdp_socket_enable);
-
- clear_bit(socket, cat_socket_enable);
- }
-}
-
-static void __init psr_cat_free(void)
-{
- xfree(cat_socket_enable);
- cat_socket_enable = NULL;
- xfree(cat_socket_info);
- cat_socket_info = NULL;
-}
-
-static void __init init_psr_cat(void)
-{
- if ( opt_cos_max < 1 )
- {
- printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
- return;
- }
-
- cat_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
- cat_socket_info = xzalloc_array(struct psr_cat_socket_info, nr_sockets);
- cdp_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
-
- if ( !cat_socket_enable || !cat_socket_info )
- psr_cat_free();
}
static int psr_cpu_prepare(unsigned int cpu)
{
- return cat_cpu_prepare(cpu);
+ return 0;
}
static void psr_cpu_init(void)
{
- if ( cat_socket_info )
- cat_cpu_init();
-
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
- if ( cat_socket_info )
- cat_cpu_fini(cpu);
+ return;
}
static int cpu_callback(
@@ -738,14 +282,10 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- if ( opt_psr & PSR_CAT )
- init_psr_cat();
-
- if ( psr_cpu_prepare(0) )
- psr_cat_free();
+ psr_cpu_prepare(0);
psr_cpu_init();
- if ( psr_cmt_enabled() || cat_socket_info )
+ if ( psr_cmt_enabled() )
register_cpu_notifier(&cpu_nfb);
return 0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 03/24] x86: refactor psr: implement main data structures.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-02-15 8:49 ` [PATCH v8 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-02-15 8:49 ` [PATCH v8 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-28 11:58 ` Roger Pau Monné
2017-02-15 8:49 ` [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
` (23 subsequent siblings)
26 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
To construct an extendible framework, we need analyze PSR features
and abstract the common things and feature specific things. Then,
encapsulate them into different data structures.
By analyzing PSR features, we can get below map.
+------+------+------+
--------->| Dom0 | Dom1 | ... |
| +------+------+------+
| |
|Dom ID | cos_id of domain
| V
| +-----------------------------------------------------------------------------+
User --------->| PSR |
Socket ID | +--------------+---------------+---------------+ |
| | Socket0 Info | Socket 1 Info | ... | |
| +--------------+---------------+---------------+ |
| | cos_id=0 cos_id=1 ... |
| | +-----------------------+-----------------------+-----------+ |
| |->Ref : | ref 0 | ref 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L3 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L2 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------+-----------+-----------+-----------+-----------+ |
| |->CDP : | cos0 code | cos0 data | cos1 code | cos1 data | ... | |
| +-----------+-----------+-----------+-----------+-----------+ |
+-----------------------------------------------------------------------------+
So, we need define a socket info data structure, 'struct
psr_socket_info' to manage information per socket. It contains a
reference count array according to COS ID and a feature list to
manage all features enabled. Every entry of the reference count
array is used to record how many domains are using the COS registers
according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
Dom1 uses COS_ID=1 registers of both features to save CBM values, like
below.
+-------+-------+-------+-----+
| COS 0 | COS 1 | COS 2 | ... |
+-------+-------+-------+-----+
L3 CAT | 0x7ff | 0x1ff | ... | ... |
+-------+-------+-------+-----+
L2 CAT | 0xff | 0xff | ... | ... |
+-------+-------+-------+-----+
If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
That means, both Dom1 and Dom2 use same COS registers(ID=1) to save same
L3/L2 values. So, the value ref[1] is 2 which means 2 domains are using
COS_ID 1.
To manage a feature, we need define a feature node data structure,
'struct feat_node', to manage feature's specific HW info, its callback
functions (all feature's specific behaviors are encapsulated into these
callback functions), and an array of all COS registers values of this
feature.
CDP is a special feature which uses two entries of the array
for one COS ID. So, the number of CDP COS registers is the half of L3
CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
it is enabled. CDP uses the COS registers array as below.
+-----------+-----------+-----------+-----------+-----------+
CDP cos_reg_val[] index: | 0 | 1 | 2 | 3 | ... |
+-----------+-----------+-----------+-----------+-----------+
value: | cos0 code | cos0 data | cos1 code | cos1 data | ... |
+-----------+-----------+-----------+-----------+-----------+
For more details, please refer SDM and patches to implement 'get value' and
'set value'.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
xen/arch/x86/psr.c | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 107 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 96a8589..5acd9ca 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -13,16 +13,122 @@
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
* more details.
*/
-#include <xen/init.h>
#include <xen/cpu.h>
#include <xen/err.h>
+#include <xen/init.h>
+#include <xen/list.h>
#include <xen/sched.h>
#include <asm/psr.h>
+/*
+ * Terminology:
+ * - CAT Cache Allocation Technology
+ * - CBM Capacity BitMasks
+ * - CDP Code and Data Prioritization
+ * - COS/CLOS Class of Service. Also mean COS registers.
+ * - COS_MAX Max number of COS for the feature (minus 1)
+ * - MSRs Machine Specific Registers
+ * - PSR Intel Platform Shared Resource
+ */
+
#define PSR_CMT (1<<0)
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
+/*
+ * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
+ * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
+ * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
+ *
+ * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for
+ * up to 64 L2 CAT COS. The COS_ID=[0,63].
+ *
+ * So, the maximum COS register count of one feature is 128.
+ */
+#define MAX_COS_REG_CNT 128
+
+/*
+ * PSR features are managed per socket. Below structure defines the members
+ * used to manage these features.
+ * feat_mask - Mask used to record features enabled on socket. There may be
+ * some features enabled at same time.
+ * nr_feat - Record how many features enabled.
+ * feat_list - A list used to manage all features enabled.
+ * cos_ref - A reference count array to record how many domains are using the
+ * COS_ID.
+ * Every entry of cos_ref corresponds to one COS ID.
+ * ref_lock - A lock to protect cos_ref.
+ */
+struct psr_socket_info {
+ /*
+ * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
+ * psr_feat_type' means the bit position.
+ * bit 0: L3 CAT
+ * bit 1: L3 CDP
+ * bit 2: L2 CAT
+ */
+ unsigned int feat_mask;
+ unsigned int nr_feat;
+ struct list_head feat_list;
+ unsigned int cos_ref[MAX_COS_REG_CNT];
+ spinlock_t ref_lock;
+};
+
+enum psr_feat_type {
+ PSR_SOCKET_L3_CAT = 0,
+ PSR_SOCKET_L3_CDP,
+ PSR_SOCKET_L2_CAT,
+};
+
+/* CAT/CDP HW info data structure. */
+struct psr_cat_hw_info {
+ unsigned int cbm_len;
+ unsigned int cos_max;
+};
+
+/* Encapsulate feature specific HW info here. */
+struct feat_hw_info {
+ union {
+ struct psr_cat_hw_info l3_cat_info;
+ };
+};
+
+struct feat_node;
+
+/*
+ * This structure defines feature operation callback functions. Every feature
+ * enabled MUST implement such callback functions and register them to ops.
+ *
+ * Feature specific behaviors will be encapsulated into these callback
+ * functions. Then, the main flows will not be changed when introducing a new
+ * feature.
+ */
+struct feat_ops {
+ /* get_cos_max is used to get feature's cos_max. */
+ unsigned int (*get_cos_max)(const struct feat_node *feat);
+};
+
+/*
+ * This structure represents one feature.
+ * feature - Which feature it is.
+ * feat_ops - Feature operation callback functions.
+ * info - Feature HW info.
+ * cos_reg_val - Array to store the values of COS registers. One entry stores
+ * the value of one COS register.
+ * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
+ * For CDP, two entries correspond to one COS_ID. E.g.
+ * COS_ID=0 corresponds to cos_reg_val[0] (Data) and
+ * cos_reg_val[1] (Code).
+ * list - Feature list.
+ */
+struct feat_node {
+ enum psr_feat_type feature;
+ struct feat_ops ops;
+ struct feat_hw_info info;
+ uint64_t cos_reg_val[MAX_COS_REG_CNT];
+ struct list_head list;
+};
+
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (2 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 03/24] x86: refactor psr: implement main data structures Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:41 ` Wei Liu
2017-03-08 14:56 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
` (22 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the CPU init and free flow including L3 CAT
initialization and feature list free.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v8:
- remove unused parameter of psr_cpu_prepare.
- add comments in cpu_fini_work.
- coding style fix.
---
xen/arch/x86/cpuid.c | 6 --
xen/arch/x86/psr.c | 181 +++++++++++++++++++++++++++++++++++++++-
xen/include/asm-x86/processor.h | 7 ++
3 files changed, 184 insertions(+), 10 deletions(-)
diff --git a/xen/arch/x86/cpuid.c b/xen/arch/x86/cpuid.c
index e0a387e..e3e92dd 100644
--- a/xen/arch/x86/cpuid.c
+++ b/xen/arch/x86/cpuid.c
@@ -34,12 +34,6 @@ static void cpuid_leaf(uint32_t leaf, struct cpuid_leaf *data)
cpuid(leaf, &data->a, &data->b, &data->c, &data->d);
}
-static void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
- struct cpuid_leaf *data)
-{
- cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
-}
-
static void sanitise_featureset(uint32_t *fs)
{
/* for_each_set_bit() uses unsigned longs. Extend with zeroes. */
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5acd9ca..10d268a 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -19,6 +19,7 @@
#include <xen/list.h>
#include <xen/sched.h>
#include <asm/psr.h>
+#include <asm/x86_emulate.h>
/*
* Terminology:
@@ -35,6 +36,9 @@
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
+#define CAT_CBM_LEN_MASK 0x1f
+#define CAT_COS_MAX_MASK 0xffff
+
/*
* Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
* the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
@@ -136,11 +140,87 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
+static struct psr_socket_info *__read_mostly socket_info;
+
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
+static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
+/*
+ * Declare global feature list entry for every feature to facilitate the
+ * feature list creation. It will be allocated in psr_cpu_prepare() and
+ * inserted into feature list in cpu_init_work(). It is protected by
+ * cpu_add_remove_lock spinlock.
+ */
+static struct feat_node *feat_l3_cat;
+
+/* Common functions. */
+static void free_feature(struct psr_socket_info *info)
+{
+ struct feat_node *feat, *next;
+
+ if ( !info )
+ return;
+
+ /*
+ * Free resources of features. But we do not free global feature list
+ * entry, like feat_l3_cat. Although it may cause a few memory leak,
+ * it is OK simplify things.
+ */
+ list_for_each_entry_safe(feat, next, &info->feat_list, list)
+ {
+ if ( !feat )
+ return;
+
+ __clear_bit(feat->feature, &info->feat_mask);
+ list_del(&feat->list);
+ xfree(feat);
+ }
+}
+
+/* L3 CAT functions implementation. */
+static void l3_cat_init_feature(struct cpuid_leaf regs,
+ struct feat_node *feat,
+ struct psr_socket_info *info)
+{
+ struct psr_cat_hw_info l3_cat = { };
+ unsigned int socket;
+
+ /* No valid value so do not enable feature. */
+ if ( !regs.a || !regs.d )
+ return;
+
+ l3_cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
+ l3_cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
+
+ /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+ feat->cos_reg_val[0] = (1ull << l3_cat.cbm_len) - 1;
+
+ feat->feature = PSR_SOCKET_L3_CAT;
+ ASSERT(!test_bit(PSR_SOCKET_L3_CAT, &info->feat_mask));
+ __set_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
+
+ feat->info.l3_cat_info = l3_cat;
+
+ info->nr_feat++;
+
+ /* Add this feature into list. */
+ list_add_tail(&feat->list, &info->feat_list);
+
+ socket = cpu_to_socket(smp_processor_id());
+ if ( !opt_cpu_info )
+ return;
+
+ printk(XENLOG_INFO "L3 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ socket, feat->info.l3_cat_info.cos_max,
+ feat->info.l3_cat_info.cbm_len);
+}
+
+static const struct feat_ops l3_cat_ops = {
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -180,6 +260,9 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
+ if ( val_str && !strcmp(s, "cos_max") )
+ opt_cos_max = simple_strtoul(val_str, NULL, 0);
+
s = ss + 1;
} while ( ss );
}
@@ -335,18 +418,104 @@ void psr_domain_free(struct domain *d)
psr_free_rmid(d);
}
-static int psr_cpu_prepare(unsigned int cpu)
+static void cpu_init_work(void)
+{
+ struct psr_socket_info *info;
+ unsigned int socket;
+ unsigned int cpu = smp_processor_id();
+ struct feat_node *feat;
+ struct cpuid_leaf regs = { .a = 0, .b = 0, .c = 0, .d = 0 };
+
+ if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
+ return;
+ else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
+ {
+ __clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
+ return;
+ }
+
+ socket = cpu_to_socket(cpu);
+ info = socket_info + socket;
+ if ( info->feat_mask )
+ return;
+
+ INIT_LIST_HEAD(&info->feat_list);
+ spin_lock_init(&info->ref_lock);
+
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L3 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
+
+ feat = feat_l3_cat;
+ /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
+ feat_l3_cat = NULL;
+ feat->ops = l3_cat_ops;
+
+ l3_cat_init_feature(regs, feat, info);
+ }
+}
+
+static void cpu_fini_work(unsigned int cpu)
{
+ unsigned int socket = cpu_to_socket(cpu);
+
+ /*
+ * We only free when we are the last CPU in the socket. The socket_cpumask
+ * is cleared prior to this notification code by remove_siblinginfo().
+ */
+ if ( socket_cpumask[socket] && cpumask_empty(socket_cpumask[socket]) )
+ free_feature(socket_info + socket);
+}
+
+static void __init init_psr(void)
+{
+ if ( opt_cos_max < 1 )
+ {
+ printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
+ return;
+ }
+
+ socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
+
+ if ( !socket_info )
+ {
+ printk(XENLOG_INFO "Failed to alloc socket_info!\n");
+ return;
+ }
+}
+
+static void __init psr_free(void)
+{
+ xfree(socket_info);
+ socket_info = NULL;
+}
+
+static int psr_cpu_prepare(void)
+{
+ if ( !socket_info )
+ return 0;
+
+ /* Malloc memory for the global feature head here. */
+ if ( feat_l3_cat == NULL &&
+ (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
static void psr_cpu_init(void)
{
+ if ( socket_info )
+ cpu_init_work();
+
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
+ if ( socket_info )
+ cpu_fini_work(cpu);
return;
}
@@ -359,7 +528,7 @@ static int cpu_callback(
switch ( action )
{
case CPU_UP_PREPARE:
- rc = psr_cpu_prepare(cpu);
+ rc = psr_cpu_prepare();
break;
case CPU_STARTING:
psr_cpu_init();
@@ -388,10 +557,14 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- psr_cpu_prepare(0);
+ if ( opt_psr & PSR_CAT )
+ init_psr();
+
+ if ( psr_cpu_prepare() )
+ psr_free();
psr_cpu_init();
- if ( psr_cmt_enabled() )
+ if ( psr_cmt_enabled() || socket_info )
register_cpu_notifier(&cpu_nfb);
return 0;
diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h
index 7735bc2..eb0a1e9 100644
--- a/xen/include/asm-x86/processor.h
+++ b/xen/include/asm-x86/processor.h
@@ -14,6 +14,7 @@
#include <asm/types.h>
#include <asm/cpufeature.h>
#include <asm/desc.h>
+#include <asm/x86_emulate.h>
#endif
#include <asm/x86-defns.h>
@@ -259,6 +260,12 @@ static always_inline unsigned int cpuid_count_ebx(
return ebx;
}
+static always_inline void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
+ struct cpuid_leaf *data)
+{
+ cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
+}
+
static inline unsigned long read_cr0(void)
{
unsigned long cr0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 05/24] x86: refactor psr: implement Domain init/free and schedule flows.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (3 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:41 ` Wei Liu
2017-03-08 15:04 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 06/24] x86: refactor psr: implement get hw info flow Yi Sun
` (21 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the Domain init/free and schedule flows.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
xen/arch/x86/psr.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 61 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 10d268a..798c614 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -51,6 +51,8 @@
*/
#define MAX_COS_REG_CNT 128
+#define PSR_ASSOC_REG_SHIFT 32
+
/*
* PSR features are managed per socket. Below structure defines the members
* used to manage these features.
@@ -218,7 +220,13 @@ static void l3_cat_init_feature(struct cpuid_leaf regs,
feat->info.l3_cat_info.cbm_len);
}
+static unsigned int l3_cat_get_cos_max(const struct feat_node *feat)
+{
+ return feat->info.l3_cat_info.cos_max;
+}
+
static const struct feat_ops l3_cat_ops = {
+ .get_cos_max = l3_cat_get_cos_max,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -362,11 +370,33 @@ void psr_free_rmid(struct domain *d)
d->arch.psr_rmid = 0;
}
+static inline unsigned int get_max_cos_max(const struct psr_socket_info *info)
+{
+ const struct feat_node *feat;
+ unsigned int cos_max = 0;
+
+ list_for_each_entry(feat, &info->feat_list, list)
+ cos_max = max(feat->ops.get_cos_max(feat), cos_max);
+
+ return cos_max;
+}
+
static inline void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( psr_cmt_enabled() )
+ if ( socket_info )
+ {
+ unsigned int socket = cpu_to_socket(smp_processor_id());
+ const struct psr_socket_info *info = socket_info + socket;
+ unsigned int cos_max = get_max_cos_max(info);
+
+ if ( info->feat_mask )
+ psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
+ PSR_ASSOC_REG_SHIFT;
+ }
+
+ if ( psr_cmt_enabled() || psra->cos_mask )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -375,6 +405,13 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
+static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
+ uint64_t cos_mask)
+{
+ *reg = (*reg & ~cos_mask) |
+ (((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
+}
+
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -383,6 +420,11 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
+ if ( psra->cos_mask )
+ psr_assoc_cos(®, d->arch.psr_cos_ids ?
+ d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
+ 0, psra->cos_mask);
+
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
@@ -408,14 +450,32 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
return 0;
}
+/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
+static void psr_free_cos(struct domain *d)
+{
+ if( !d->arch.psr_cos_ids )
+ return;
+
+ xfree(d->arch.psr_cos_ids);
+ d->arch.psr_cos_ids = NULL;
+}
+
int psr_domain_init(struct domain *d)
{
+ if ( socket_info )
+ {
+ d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
+ if ( !d->arch.psr_cos_ids )
+ return -ENOMEM;
+ }
+
return 0;
}
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
+ psr_free_cos(d);
}
static void cpu_init_work(void)
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 06/24] x86: refactor psr: implement get hw info flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (4 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:41 ` Wei Liu
` (2 more replies)
2017-02-15 8:49 ` [PATCH v8 07/24] x86: refactor psr: implement get value flow Yi Sun
` (20 subsequent siblings)
26 siblings, 3 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements get HW info flow including L3 CAT callback
function.
It also changes sysctl interface to make it more general.
With this patch, 'psr-hwinfo' can work for L3 CAT.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
xen/arch/x86/psr.c | 75 +++++++++++++++++++++++++++++++++++++++++++++--
xen/arch/x86/sysctl.c | 14 +++++----
xen/include/asm-x86/psr.h | 19 +++++++-----
3 files changed, 93 insertions(+), 15 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 798c614..8af59d9 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -84,6 +84,7 @@ enum psr_feat_type {
PSR_SOCKET_L3_CAT = 0,
PSR_SOCKET_L3_CDP,
PSR_SOCKET_L2_CAT,
+ PSR_SOCKET_UNKNOWN = 0xFFFF,
};
/* CAT/CDP HW info data structure. */
@@ -112,6 +113,9 @@ struct feat_node;
struct feat_ops {
/* get_cos_max is used to get feature's cos_max. */
unsigned int (*get_cos_max)(const struct feat_node *feat);
+ /* get_feat_info is used to get feature HW info. */
+ bool (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len);
};
/*
@@ -182,6 +186,24 @@ static void free_feature(struct psr_socket_info *info)
}
}
+static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
+{
+ enum psr_feat_type feat_type;
+
+ /* Judge if feature is enabled. */
+ switch ( type )
+ {
+ case PSR_CBM_TYPE_L3:
+ feat_type = PSR_SOCKET_L3_CAT;
+ break;
+ default:
+ feat_type = PSR_SOCKET_UNKNOWN;
+ break;
+ }
+
+ return feat_type;
+}
+
/* L3 CAT functions implementation. */
static void l3_cat_init_feature(struct cpuid_leaf regs,
struct feat_node *feat,
@@ -225,8 +247,22 @@ static unsigned int l3_cat_get_cos_max(const struct feat_node *feat)
return feat->info.l3_cat_info.cos_max;
}
+static bool l3_cat_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len)
+{
+ if ( !data || 3 > array_len )
+ return false;
+
+ data[CBM_LEN] = feat->info.l3_cat_info.cbm_len;
+ data[COS_MAX] = feat->info.l3_cat_info.cos_max;
+ data[PSR_FLAG] = 0;
+
+ return true;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
+ .get_feat_info = l3_cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -432,10 +468,43 @@ void psr_ctxt_switch_to(struct domain *d)
}
}
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags)
+static struct psr_socket_info *get_socket_info(unsigned int socket)
{
- return 0;
+ if ( !socket_info )
+ return ERR_PTR(-ENODEV);
+
+ if ( socket >= nr_sockets )
+ return ERR_PTR(-ERANGE);
+
+ if ( !socket_info[socket].feat_mask )
+ return ERR_PTR(-ENOENT);
+
+ return socket_info + socket;
+}
+
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len)
+{
+ const struct psr_socket_info *info = get_socket_info(socket);
+ const struct feat_node *feat;
+ enum psr_feat_type feat_type;
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ if ( feat->feature != feat_type )
+ continue;
+
+ if ( feat->ops.get_feat_info(feat, data, array_len) )
+ return 0;
+ else
+ return -EINVAL;
+ }
+
+ return -ENOENT;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index b8c30d4..e340baa 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -176,15 +176,19 @@ long arch_do_sysctl(
switch ( sysctl->u.psr_cat_op.cmd )
{
case XEN_SYSCTL_PSR_CAT_get_l3_info:
- ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
- &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
- &sysctl->u.psr_cat_op.u.l3_info.cos_max,
- &sysctl->u.psr_cat_op.u.l3_info.flags);
+ {
+ uint32_t data[3];
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L3, data, 3);
+
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
ret = -EFAULT;
break;
-
+ }
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 57f47e9..0342a80 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -19,19 +19,24 @@
#include <xen/types.h>
/* CAT cpuid level */
-#define PSR_CPUID_LEVEL_CAT 0x10
+#define PSR_CPUID_LEVEL_CAT 0x10
/* Resource Type Enumeration */
-#define PSR_RESOURCE_TYPE_L3 0x2
+#define PSR_RESOURCE_TYPE_L3 0x2
/* L3 Monitoring Features */
-#define PSR_CMT_L3_OCCUPANCY 0x1
+#define PSR_CMT_L3_OCCUPANCY 0x1
/* CDP Capability */
-#define PSR_CAT_CDP_CAPABILITY (1u << 2)
+#define PSR_CAT_CDP_CAPABILITY (1u << 2)
/* L3 CDP Enable bit*/
-#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
+#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
+
+/* Used by psr_get_info() */
+#define CBM_LEN 0
+#define COS_MAX 1
+#define PSR_FLAG 2
struct psr_cmt_l3 {
unsigned int features;
@@ -63,8 +68,8 @@ int psr_alloc_rmid(struct domain *d);
void psr_free_rmid(struct domain *d);
void psr_ctxt_switch_to(struct domain *d);
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags);
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len);
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 07/24] x86: refactor psr: implement get value flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (5 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 06/24] x86: refactor psr: implement get hw info flow Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-28 12:44 ` Roger Pau Monné
2017-03-08 15:35 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 08/24] x86: refactor psr: set value: implement framework Yi Sun
` (19 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements get value flow including L3 CAT callback
function.
It also changes domctl interface to make it more general.
With this patch, 'psr-cat-show' can work for L3 CAT but not for
L3 code/data which is implemented in patch "x86: refactor psr:
implement get value flow for CDP.".
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
xen/arch/x86/domctl.c | 18 +++++++++---------
xen/arch/x86/psr.c | 43 ++++++++++++++++++++++++++++++++++++++-----
xen/include/asm-x86/psr.h | 4 ++--
3 files changed, 49 insertions(+), 16 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 8e5259f..098e399 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1440,23 +1440,23 @@ long arch_do_domctl(
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3);
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_CODE);
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_DATA);
copyback = 1;
break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 8af59d9..c1afd36 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -116,6 +116,9 @@ struct feat_ops {
/* get_feat_info is used to get feature HW info. */
bool (*get_feat_info)(const struct feat_node *feat,
uint32_t data[], unsigned int array_len);
+ /* get_val is used to get feature COS register value. */
+ bool (*get_val)(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val);
};
/*
@@ -260,9 +263,22 @@ static bool l3_cat_get_feat_info(const struct feat_node *feat,
return true;
}
+static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val)
+{
+ if ( cos > feat->info.l3_cat_info.cos_max )
+ /* Use default value. */
+ cos = 0;
+
+ *val = feat->cos_reg_val[cos];
+
+ return true;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
+ .get_val = l3_cat_get_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -482,12 +498,14 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
return socket_info + socket;
}
-int psr_get_info(unsigned int socket, enum cbm_type type,
- uint32_t data[], unsigned int array_len)
+static int psr_get(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len,
+ struct domain *d, uint64_t *val)
{
const struct psr_socket_info *info = get_socket_info(socket);
const struct feat_node *feat;
enum psr_feat_type feat_type;
+ unsigned int cos;
if ( IS_ERR(info) )
return PTR_ERR(info);
@@ -498,6 +516,15 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
if ( feat->feature != feat_type )
continue;
+ if ( d )
+ {
+ cos = d->arch.psr_cos_ids[socket];
+ if ( feat->ops.get_val(feat, cos, type, val) )
+ return 0;
+ else
+ break;
+ }
+
if ( feat->ops.get_feat_info(feat, data, array_len) )
return 0;
else
@@ -507,10 +534,16 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
return -ENOENT;
}
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type)
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len)
+{
+ return psr_get(socket, type, data, array_len, NULL, NULL);
+}
+
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint64_t *val, enum cbm_type type)
{
- return 0;
+ return psr_get(socket, type, NULL, 0, d, val);
}
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 0342a80..569e7df 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -70,8 +70,8 @@ void psr_ctxt_switch_to(struct domain *d);
int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type);
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint64_t *val, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (6 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 07/24] x86: refactor psr: implement get value flow Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:41 ` Wei Liu
` (2 more replies)
2017-02-15 8:49 ` [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
` (18 subsequent siblings)
26 siblings, 3 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
As set value flow is the most complicated one in psr, it will be
divided to some patches to make things clearer. This patch
implements the set value framework to show a whole picture firstly.
It also changes domctl interface to make it more general.
To make the set value flow be general and can support multiple features
at same time, it includes below steps:
1. Get COS ID of current domain using.
2. Assemble a value array to store all features current value
in it and replace the current value of the feature which is
being set to the new input value.
3. Find if there is already a COS ID on which all features'
values are same as the array. Then, we can reuse this COS
ID.
4. If fail to find, we need pick an available COS ID. Only COS ID which ref
is 0 or 1 can be picked.
5. Write all features MSRs according to the COS ID.
6. Update ref according to COS ID.
7. Save the COS ID into current domain's psr_cos_ids[socket] so that we
can know which COS the domain is using on the socket.
So, some functions are abstracted and the callback functions will be
implemented in next patches.
Here is an example to understand the process. The CPU supports
two featuers, e.g. L3 CAT and L2 CAT. user wants to set L3 CAT
of Dom1 to 0x1ff.
1. Get the old_cos of Dom1 which is 0. L3 CAT is the first
element of feature list. The COS registers values are below at
this time.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | ... | ... | ... |
-------------------------------
L2 CAT | 0xff | ... | ... | ... |
-------------------------------
2. Assemble The value array to be:
val[0]: 0x1ff
val[1]: 0xff
3. It cannot find a matching COS.
4. Allocate COS 1 to store the value set.
5. Write the COS 1 registers. The COS registers values are
changed to below now.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | 0x1ff | ... | ... |
-------------------------------
L2 CAT | 0xff | 0xff | ... | ... |
-------------------------------
6. The ref[1] is increased to 1 because Dom1 is using it now.
7. Save 1 to Dom1's psr_cos_ids[socket].
Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
of Dom2 is 0 too. Repeat above flow.
The val array assembled is:
val[0]: 0x1ff
val[1]: 0xff
So, it can find a matching COS, COS 1. Then, it can reuse COS 1
for Dom2.
The ref[1] is increased to 2 now because both Dom1 and Dom2 are
using this COS ID. Set 1 to Dom2's psr_cos_ids[socket].
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/domctl.c | 18 ++---
xen/arch/x86/psr.c | 202 +++++++++++++++++++++++++++++++++++++++++++++-
xen/include/asm-x86/psr.h | 4 +-
3 files changed, 210 insertions(+), 14 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 098e399..f8f5539 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1422,21 +1422,21 @@ long arch_do_domctl(
switch ( domctl->u.psr_cat_op.cmd )
{
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_CODE);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_DATA);
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index c1afd36..d414b5e 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -546,18 +546,214 @@ int psr_get_val(struct domain *d, unsigned int socket,
return psr_get(socket, type, NULL, 0, d, val);
}
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type)
+/* Set value functions */
+static unsigned int get_cos_num(const struct psr_socket_info *info)
{
return 0;
}
+static int assemble_val_array(uint64_t *val,
+ uint32_t array_len,
+ const struct psr_socket_info *info,
+ unsigned int old_cos)
+{
+ return -EINVAL;
+}
+
+static int set_new_val_to_array(uint64_t *val,
+ uint32_t array_len,
+ const struct psr_socket_info *info,
+ enum psr_feat_type feat_type,
+ enum cbm_type type,
+ uint64_t m)
+{
+ return -EINVAL;
+}
+
+static int find_cos(const uint64_t *val, uint32_t array_len,
+ enum psr_feat_type feat_type,
+ const struct psr_socket_info *info)
+{
+ return -ENOENT;
+}
+
+static int pick_avail_cos(const struct psr_socket_info *info,
+ const uint64_t *val, uint32_t array_len,
+ unsigned int old_cos,
+ enum psr_feat_type feat_type)
+{
+ return -ENOENT;
+}
+
+static int write_psr_msr(unsigned int socket, unsigned int cos,
+ const uint64_t *val)
+{
+ return -ENOENT;
+}
+
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t val, enum cbm_type type)
+{
+ unsigned int old_cos;
+ int cos, ret;
+ unsigned int *ref;
+ uint64_t *val_array;
+ struct psr_socket_info *info = get_socket_info(socket);
+ uint32_t array_len;
+ enum psr_feat_type feat_type;
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( !test_bit(feat_type, &info->feat_mask) )
+ return -ENOENT;
+
+ /*
+ * Step 0:
+ * old_cos means the COS ID current domain is using. By default, it is 0.
+ *
+ * For every COS ID, there is a reference count to record how many domains
+ * are using the COS register corresponding to this COS ID.
+ * - If ref[old_cos] is 0, that means this COS is not used by any domain.
+ * - If ref[old_cos] is 1, that means this COS is only used by current
+ * domain.
+ * - If ref[old_cos] is more than 1, that mean multiple domains are using
+ * this COS.
+ */
+ old_cos = d->arch.psr_cos_ids[socket];
+ if ( old_cos > MAX_COS_REG_CNT )
+ return -EOVERFLOW;
+
+ ref = info->cos_ref;
+
+ /*
+ * Step 1:
+ * Assemle a value array to store all featues cos_reg_val[old_cos].
+ * And, set the input val into array according to the feature's
+ * position in array.
+ */
+ array_len = get_cos_num(info);
+ val_array = xzalloc_array(uint64_t, array_len);
+ if ( !val_array )
+ return -ENOMEM;
+
+ if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 )
+ {
+ xfree(val_array);
+ return ret;
+ }
+
+ if ( (ret = set_new_val_to_array(val_array, array_len, info,
+ feat_type, type, val)) != 0 )
+ {
+ xfree(val_array);
+ return ret;
+ }
+
+ /*
+ * Lock here to make sure the ref is not changed during find and
+ * write process.
+ */
+ spin_lock(&info->ref_lock);
+
+ /*
+ * Step 2:
+ * Try to find if there is already a COS ID on which all features' values
+ * are same as the array. Then, we can reuse this COS ID.
+ */
+ cos = find_cos(val_array, array_len, feat_type, info);
+ if ( cos >= 0 )
+ {
+ if ( cos == old_cos )
+ {
+ spin_unlock(&info->ref_lock);
+ xfree(val_array);
+ return 0;
+ }
+ }
+ else
+ {
+ /*
+ * Step 3:
+ * If fail to find, we need allocate a new COS ID.
+ * If multiple domains are using same COS ID, its ref is more
+ * than 1. That means we cannot free this COS to make current domain
+ * use it. Because other domains are using the value saved in the COS.
+ * Unless the ref is changed to 1 (mean only current domain is using
+ * it), we cannot allocate the COS ID to current domain.
+ * So, only the COS ID which ref is 1 or 0 can be allocated.
+ */
+ cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
+ if ( cos < 0 )
+ {
+ spin_unlock(&info->ref_lock);
+ xfree(val_array);
+ return cos;
+ }
+
+ /*
+ * Step 4:
+ * Write all features MSRs according to the COS ID.
+ */
+ ret = write_psr_msr(socket, cos, val_array);
+ if ( ret )
+ {
+ spin_unlock(&info->ref_lock);
+ xfree(val_array);
+ return ret;
+ }
+ }
+
+ /*
+ * Step 5:
+ * Update ref according to COS ID.
+ */
+ ref[cos]++;
+ ASSERT(ref[cos] || cos == 0);
+ ref[old_cos]--;
+ spin_unlock(&info->ref_lock);
+
+ /*
+ * Step 6:
+ * Save the COS ID into current domain's psr_cos_ids[] so that we can know
+ * which COS the domain is using on the socket. One domain can only use
+ * one COS ID at same time on each socket.
+ */
+ d->arch.psr_cos_ids[socket] = cos;
+ xfree(val_array);
+
+ return 0;
+}
+
/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
static void psr_free_cos(struct domain *d)
{
- if( !d->arch.psr_cos_ids )
+ unsigned int socket, cos;
+
+ if ( !d->arch.psr_cos_ids )
return;
+ /* Domain is free so its cos_ref should be decreased. */
+ for ( socket = 0; socket < nr_sockets; socket++ )
+ {
+ struct psr_socket_info *info;
+
+ /* cos 0 is default one which does not need be handled. */
+ if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
+ continue;
+
+ /*
+ * If domain uses other cos ids, all corresponding refs must have been
+ * increased 1 for this domain. So, we need decrease them.
+ */
+ info = socket_info + socket;
+ ASSERT(info->cos_ref[cos] || cos == 0);
+ spin_lock(&info->ref_lock);
+ info->cos_ref[cos]--;
+ spin_unlock(&info->ref_lock);
+ }
+
xfree(d->arch.psr_cos_ids);
d->arch.psr_cos_ids = NULL;
}
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 569e7df..fde7882 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -72,8 +72,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
int psr_get_val(struct domain *d, unsigned int socket,
uint64_t *val, enum cbm_type type);
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type);
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t val, enum cbm_type type);
int psr_domain_init(struct domain *d);
void psr_domain_free(struct domain *d);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (7 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 08/24] x86: refactor psr: set value: implement framework Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-03-08 16:54 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
` (17 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
Only can one COS ID be used by one domain at one time. That means all enabled
features' COS registers at this COS ID are valid for this domain at that time.
When user updates a feature's value, we need make sure all other features'
values are not affected. So, we firstly need assemble an array which contains
all features current values and replace the setting feature's value in array
to new value.
Then, we can try to find if there is a COS ID on which all features' COS
registers values are same as the array. If we can find, we just use this COS
ID. If fail to find, we need allocate a new COS ID.
This patch implements value array assembling flow.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 151 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 146 insertions(+), 5 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index d414b5e..ae108b9 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -119,6 +119,32 @@ struct feat_ops {
/* get_val is used to get feature COS register value. */
bool (*get_val)(const struct feat_node *feat, unsigned int cos,
enum cbm_type type, uint64_t *val);
+ /*
+ * get_cos_num is used to get the COS registers amount used by the
+ * feature for one setting, e.g. CDP uses 2 COSs but CAT uses 1.
+ */
+ unsigned int (*get_cos_num)(const struct feat_node *feat);
+ /*
+ * get_old_val and set_new_val are a pair of functions called in order.
+ * The caller will traverse all features in the list and call both
+ * functions for every feature to do below two things:
+ * 1. get old_cos register value of all supported features and
+ * 2. set the new value for the feature.
+ *
+ * All the values are set into value array according the traversal order,
+ * meaning the same order of feature list members.
+ *
+ * The return value meaning:
+ * 0 - success.
+ * negative - error.
+ */
+ int (*get_old_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos);
+ int (*set_new_val)(uint64_t val[],
+ const struct feat_node *feat,
+ enum cbm_type type,
+ uint64_t m);
};
/*
@@ -207,6 +233,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
return feat_type;
}
+static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
+{
+ unsigned int first_bit, zero_bit;
+
+ /* Set bits should only in the range of [0, cbm_len). */
+ if ( cbm & (~0ull << cbm_len) )
+ return false;
+
+ /* At least one bit need to be set. */
+ if ( cbm == 0 )
+ return false;
+
+ first_bit = find_first_bit(&cbm, cbm_len);
+ zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
+
+ /* Set bits should be contiguous. */
+ if ( zero_bit < cbm_len &&
+ find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
+ return false;
+
+ return true;
+}
+
/* L3 CAT functions implementation. */
static void l3_cat_init_feature(struct cpuid_leaf regs,
struct feat_node *feat,
@@ -275,10 +324,45 @@ static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
return true;
}
+static unsigned int l3_cat_get_cos_num(const struct feat_node *feat)
+{
+ return 1;
+}
+
+static int l3_cat_get_old_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos)
+{
+ if ( old_cos > feat->info.l3_cat_info.cos_max )
+ /* Use default value. */
+ old_cos = 0;
+
+ /* CAT */
+ val[0] = feat->cos_reg_val[old_cos];
+
+ return 0;
+}
+
+static int l3_cat_set_new_val(uint64_t val[],
+ const struct feat_node *feat,
+ enum cbm_type type,
+ uint64_t m)
+{
+ if ( !psr_check_cbm(feat->info.l3_cat_info.cbm_len, m) )
+ return -EINVAL;
+
+ val[0] = m;
+
+ return 0;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
.get_val = l3_cat_get_val,
+ .get_cos_num = l3_cat_get_cos_num,
+ .get_old_val = l3_cat_get_old_val,
+ .set_new_val = l3_cat_set_new_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -549,15 +633,42 @@ int psr_get_val(struct domain *d, unsigned int socket,
/* Set value functions */
static unsigned int get_cos_num(const struct psr_socket_info *info)
{
- return 0;
+ const struct feat_node *feat_tmp;
+ unsigned int num = 0;
+
+ /* Get all features total amount. */
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ num += feat_tmp->ops.get_cos_num(feat_tmp);
+
+ return num;
}
-static int assemble_val_array(uint64_t *val,
+static int combine_val_array(uint64_t *val,
uint32_t array_len,
const struct psr_socket_info *info,
unsigned int old_cos)
{
- return -EINVAL;
+ const struct feat_node *feat;
+ int ret;
+ uint64_t *val_tmp = val;
+
+ if ( !val )
+ return -EINVAL;
+
+ /* Get all features current values according to old_cos. */
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ /* value getting order is same as feature list */
+ ret = feat->ops.get_old_val(val_tmp, feat, old_cos);
+ if ( ret )
+ return ret;
+
+ val_tmp += feat->ops.get_cos_num(feat);
+ if ( val_tmp - val > array_len)
+ return -ENOSPC;
+ }
+
+ return 0;
}
static int set_new_val_to_array(uint64_t *val,
@@ -567,7 +678,37 @@ static int set_new_val_to_array(uint64_t *val,
enum cbm_type type,
uint64_t m)
{
- return -EINVAL;
+ const struct feat_node *feat;
+ int ret;
+ uint64_t *val_tmp = val;
+
+ /* Set new value into array according to feature's position in array. */
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ if ( feat->feature != feat_type )
+ {
+ val_tmp += feat->ops.get_cos_num(feat);
+ if ( val_tmp - val > array_len)
+ return -ENOSPC;
+
+ continue;
+ }
+
+ /*
+ * Value setting position is same as feature list.
+ * Different features may have different setting behaviors, e.g. CDP
+ * has two values (DATA/CODE) which need us to save input value to
+ * different position in the array according to type, so we have to
+ * maintain a callback function.
+ */
+ ret = feat->ops.set_new_val(val_tmp, feat, type, m);
+ if ( ret )
+ return ret;
+ else
+ break;
+ }
+
+ return 0;
}
static int find_cos(const uint64_t *val, uint32_t array_len,
@@ -638,7 +779,7 @@ int psr_set_val(struct domain *d, unsigned int socket,
if ( !val_array )
return -ENOMEM;
- if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 )
+ if ( (ret = combine_val_array(val_array, array_len, info, old_cos)) != 0 )
{
xfree(val_array);
return ret;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (8 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-03-08 17:03 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
` (16 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
Continue with patch:
'x86: refactor psr: set value: assemble features value array'
We can try to find if there is a COS ID on which all features' COS registers
values are same as the array assembled before.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 93 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index ae108b9..7fab28b 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -145,6 +145,19 @@ struct feat_ops {
const struct feat_node *feat,
enum cbm_type type,
uint64_t m);
+ /*
+ * compare_val is used in set value process to compare if the
+ * input value array can match all the features' COS registers values
+ * according to input cos id.
+ *
+ * The return value is the amount of entries to skip in the value array
+ * or error.
+ * 1 - one entry in value array.
+ * 2 - two entries in value array, e.g. CDP uses two entries.
+ * negative - error.
+ */
+ int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
+ unsigned int cos, bool *found);
};
/*
@@ -356,6 +369,34 @@ static int l3_cat_set_new_val(uint64_t val[],
return 0;
}
+static int l3_cat_compare_val(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos, bool *found)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
+
+ /*
+ * Different features' cos_max are different. If cos id of the feature
+ * being set exceeds other feature's cos_max, the val of other feature
+ * must be default value. HW supports such case.
+ */
+ if ( cos > feat->info.l3_cat_info.cos_max )
+ {
+ if ( val[0] != l3_def_cbm )
+ {
+ *found = false;
+ return -ENOENT;
+ }
+ *found = true;
+ }
+ else
+ *found = (val[0] == feat->cos_reg_val[cos]);
+
+ return 0;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
@@ -363,6 +404,7 @@ static const struct feat_ops l3_cat_ops = {
.get_cos_num = l3_cat_get_cos_num,
.get_old_val = l3_cat_get_old_val,
.set_new_val = l3_cat_set_new_val,
+ .compare_val = l3_cat_compare_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -715,6 +757,57 @@ static int find_cos(const uint64_t *val, uint32_t array_len,
enum psr_feat_type feat_type,
const struct psr_socket_info *info)
{
+ unsigned int cos;
+ const unsigned int *ref = info->cos_ref;
+ const struct feat_node *feat;
+ const uint64_t *val_tmp = val;
+ int ret;
+ bool found = false;
+ unsigned int cos_max = 0;
+
+ /* cos_max is the one of the feature which is being set. */
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ if ( feat->feature != feat_type )
+ continue;
+
+ cos_max = feat->ops.get_cos_max(feat);
+ if ( cos_max > 0 )
+ break;
+ }
+
+ for ( cos = 0; cos <= cos_max; cos++ )
+ {
+ if ( cos && !ref[cos] )
+ continue;
+
+ /* Not found, need find again from beginning. */
+ val_tmp = val;
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ /*
+ * Compare value according to feature list order.
+ * We must follow this order because value array is assembled
+ * as this order in get_old_set_new().
+ */
+ ret = feat->ops.compare_val(val_tmp, feat, cos, &found);
+ if ( ret < 0 )
+ return ret;
+
+ /* If fail to match, go to next cos to compare. */
+ if ( !found )
+ break;
+
+ val_tmp += feat->ops.get_cos_num(feat);
+ if ( val_tmp - val > array_len )
+ return -EINVAL;
+ }
+
+ /* For this COS ID all entries in the values array did match. Use it. */
+ if ( found )
+ return cos;
+ }
+
return -ENOENT;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (9 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-03-09 14:10 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 12/24] x86: refactor psr: set value: implement write msr flow Yi Sun
` (15 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
Continue with previous patch:
'x86: refactor psr: set value: implement cos finding flow.'
If fail to find a COS ID, we need pick a new COS ID for domain. Only COS ID
that ref[COS_ID] is 1 or 0 can be picked to input a new set feature values.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 99 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 7fab28b..5de53ac 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -158,6 +158,17 @@ struct feat_ops {
*/
int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
unsigned int cos, bool *found);
+ /*
+ * fits_cos_max is used to check if the input cos id exceeds the
+ * feature's cos_max and if the input value is not the default one.
+ * Even if the associated cos exceeds the cos_max, HW can work with default
+ * value. That is the reason we need check if input value is default one.
+ * If both criteria are fulfilled, that means the input exceeds the range.
+ * If not, that means the input fits the requirements.
+ */
+ bool (*fits_cos_max)(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos);
};
/*
@@ -397,6 +408,25 @@ static int l3_cat_compare_val(const uint64_t val[],
return 0;
}
+static bool l3_cat_fits_cos_max(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
+
+ if ( cos > feat->info.l3_cat_info.cos_max &&
+ val[0] != l3_def_cbm )
+ /*
+ * Exceed cos_max and value to set is not default,
+ * return error.
+ */
+ return false;
+
+ return true;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
@@ -405,6 +435,7 @@ static const struct feat_ops l3_cat_ops = {
.get_old_val = l3_cat_get_old_val,
.set_new_val = l3_cat_set_new_val,
.compare_val = l3_cat_compare_val,
+ .fits_cos_max = l3_cat_fits_cos_max,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -811,11 +842,79 @@ static int find_cos(const uint64_t *val, uint32_t array_len,
return -ENOENT;
}
+static bool fits_cos_max(const uint64_t *val,
+ uint32_t array_len,
+ const struct psr_socket_info *info,
+ unsigned int cos)
+{
+ unsigned int ret;
+ const uint64_t *val_tmp = val;
+ const struct feat_node *feat;
+
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ ret = feat->ops.fits_cos_max(val_tmp, feat, cos);
+ if ( !ret )
+ return false;
+
+ val_tmp += feat->ops.get_cos_num(feat);
+ if ( val_tmp - val > array_len )
+ return false;
+ }
+
+ return true;
+}
+
static int pick_avail_cos(const struct psr_socket_info *info,
const uint64_t *val, uint32_t array_len,
unsigned int old_cos,
enum psr_feat_type feat_type)
{
+ unsigned int cos;
+ unsigned int cos_max = 0;
+ const struct feat_node *feat;
+ const unsigned int *ref = info->cos_ref;
+
+ /*
+ * cos_max is the one of the feature which is being set.
+ */
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ if ( feat->feature != feat_type )
+ continue;
+
+ cos_max = feat->ops.get_cos_max(feat);
+ if ( cos_max > 0 )
+ break;
+ }
+
+ if ( !cos_max )
+ return -ENOENT;
+
+ /*
+ * If old cos is referred only by the domain, then use it. And, we cannot
+ * use id 0 because it stores the default values.
+ */
+ if ( old_cos && ref[old_cos] == 1 &&
+ fits_cos_max(val, array_len, info, old_cos) )
+ return old_cos;
+
+ /* Find an unused one other than cos0. */
+ for ( cos = 1; cos <= cos_max; cos++ )
+ {
+ /*
+ * ref is 0 means this COS is not used by other domain and
+ * can be used for current setting.
+ */
+ if ( !ref[cos] )
+ {
+ if ( !fits_cos_max(val, array_len, info, cos) )
+ return -ENOENT;
+
+ return cos;
+ }
+ }
+
return -ENOENT;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 12/24] x86: refactor psr: set value: implement write msr flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (10 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-15 8:49 ` [PATCH v8 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
` (14 subsequent siblings)
26 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
Continue with previous patch:
'x86: refactor psr: set value: implement cos id picking flow.'
We have got all features values and COS ID to set. Then, we write MSRs of all
features except the setting value is same as original value.
Till now, set value process is completed.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v8:
- modify 'write_msr' callback function to 'void' because we have to set
all features' cbm. When input cos exceeds some features' cos_max, just
skip them but not break the iteration.
---
xen/arch/x86/psr.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 76 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5de53ac..82bb8fe 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -169,6 +169,9 @@ struct feat_ops {
bool (*fits_cos_max)(const uint64_t val[],
const struct feat_node *feat,
unsigned int cos);
+ /* write_msr is used to write out feature MSR register. */
+ void (*write_msr)(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat);
};
/*
@@ -427,6 +430,21 @@ static bool l3_cat_fits_cos_max(const uint64_t val[],
return true;
}
+static void l3_cat_write_msr(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat)
+{
+ if ( cos > feat->info.l3_cat_info.cos_max )
+ return;
+
+ if ( feat->cos_reg_val[cos] != val[0] )
+ {
+ feat->cos_reg_val[cos] = val[0];
+ wrmsrl(MSR_IA32_PSR_L3_MASK(cos), val[0]);
+ }
+
+ return;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
@@ -436,6 +454,7 @@ static const struct feat_ops l3_cat_ops = {
.set_new_val = l3_cat_set_new_val,
.compare_val = l3_cat_compare_val,
.fits_cos_max = l3_cat_fits_cos_max,
+ .write_msr = l3_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -918,10 +937,66 @@ static int pick_avail_cos(const struct psr_socket_info *info,
return -ENOENT;
}
+static unsigned int get_socket_cpu(unsigned int socket)
+{
+ if ( likely(socket < nr_sockets) )
+ return cpumask_any(socket_cpumask[socket]);
+
+ return nr_cpu_ids;
+}
+
+struct cos_write_info
+{
+ unsigned int cos;
+ struct list_head *feat_list;
+ const uint64_t *val;
+};
+
+static void do_write_psr_msr(void *data)
+{
+ struct cos_write_info *info = (struct cos_write_info *)data;
+ unsigned int cos = info->cos;
+ struct list_head *feat_list= info->feat_list;
+ const uint64_t *val = info->val;
+ struct feat_node *feat;
+
+ if ( !feat_list )
+ return;
+
+ /*
+ * We need set all features values into MSRs. Even the cos exceeds some
+ * features' cos_max, just skip them.
+ */
+ list_for_each_entry(feat, feat_list, list)
+ {
+ feat->ops.write_msr(cos, val, feat);
+ val += feat->ops.get_cos_num(feat);
+ }
+}
+
static int write_psr_msr(unsigned int socket, unsigned int cos,
const uint64_t *val)
{
- return -ENOENT;
+ struct psr_socket_info *info = get_socket_info(socket);
+ struct cos_write_info data =
+ {
+ .cos = cos,
+ .feat_list = &info->feat_list,
+ .val = val,
+ };
+
+ if ( socket == cpu_to_socket(smp_processor_id()) )
+ do_write_psr_msr(&data);
+ else
+ {
+ unsigned int cpu = get_socket_cpu(socket);
+
+ if ( cpu >= nr_cpu_ids )
+ return -ENOTSOCK;
+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msr, &data, 1);
+ }
+
+ return 0;
}
int psr_set_val(struct domain *d, unsigned int socket,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 13/24] x86: refactor psr: implement CPU init and free flow for CDP.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (11 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 12/24] x86: refactor psr: set value: implement write msr flow Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-28 14:52 ` Roger Pau Monné
2017-03-09 14:53 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 14/24] x86: refactor psr: implement get hw info " Yi Sun
` (13 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the CPU init and free flow for CDP including L3 CDP
initialization callback function.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 98 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 82bb8fe..4c08779 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -97,6 +97,7 @@ struct psr_cat_hw_info {
struct feat_hw_info {
union {
struct psr_cat_hw_info l3_cat_info;
+ struct psr_cat_hw_info l3_cdp_info;
};
};
@@ -195,6 +196,22 @@ struct feat_node {
struct list_head list;
};
+/*
+ * get_data - get DATA COS register value from input COS ID.
+ * @feat: the feature list entry.
+ * @cos: the COS ID.
+ */
+#define get_cdp_data(feat, cos) \
+ ( feat->cos_reg_val[cos * 2] )
+
+/*
+ * get_cdp_code - get CODE COS register value from input COS ID.
+ * @feat: the feature list entry.
+ * @cos: the COS ID.
+ */
+#define get_cdp_code(feat, cos) \
+ ( feat->cos_reg_val[cos * 2 + 1] )
+
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -217,6 +234,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
* cpu_add_remove_lock spinlock.
*/
static struct feat_node *feat_l3_cat;
+static struct feat_node *feat_l3_cdp;
/* Common functions. */
static void free_feature(struct psr_socket_info *info)
@@ -457,6 +475,63 @@ static const struct feat_ops l3_cat_ops = {
.write_msr = l3_cat_write_msr,
};
+/* L3 CDP functions implementation. */
+static void l3_cdp_init_feature(struct cpuid_leaf regs,
+ struct feat_node *feat,
+ struct psr_socket_info *info)
+{
+ struct psr_cat_hw_info l3_cdp = { };
+ unsigned int socket;
+ uint64_t val;
+
+ /* No valid value so do not enable feature. */
+ if ( !regs.a || !regs.d )
+ return;
+
+ l3_cdp.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
+ /* Cut half of cos_max when CDP is enabled. */
+ l3_cdp.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK) >> 1;
+
+ /* cos=0 is reserved as default cbm(all ones). */
+ get_cdp_code(feat, 0) =
+ (1ull << l3_cdp.cbm_len) - 1;
+ get_cdp_data(feat, 0) =
+ (1ull << l3_cdp.cbm_len) - 1;
+
+ /* We only write mask1 since mask0 is always all ones by default. */
+ wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << l3_cdp.cbm_len) - 1);
+ rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
+ wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
+
+ feat->feature = PSR_SOCKET_L3_CDP;
+ ASSERT(!test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask));
+ __set_bit(PSR_SOCKET_L3_CDP, &info->feat_mask);
+
+ feat->info.l3_cdp_info = l3_cdp;
+
+ info->nr_feat++;
+
+ /* Add this feature into list. */
+ list_add_tail(&feat->list, &info->feat_list);
+
+ socket = cpu_to_socket(smp_processor_id());
+ if ( !opt_cpu_info )
+ return;
+
+ printk(XENLOG_INFO "L3 CDP: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ socket, feat->info.l3_cdp_info.cos_max,
+ feat->info.l3_cdp_info.cbm_len);
+}
+
+static unsigned int l3_cdp_get_cos_max(const struct feat_node *feat)
+{
+ return feat->info.l3_cdp_info.cos_max;
+}
+
+struct feat_ops l3_cdp_ops = {
+ .get_cos_max = l3_cdp_get_cos_max,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1213,12 +1288,21 @@ static void cpu_init_work(void)
{
cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
- feat = feat_l3_cat;
- /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
- feat_l3_cat = NULL;
- feat->ops = l3_cat_ops;
-
- l3_cat_init_feature(regs, feat, info);
+ if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
+ !test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) )
+ {
+ feat = feat_l3_cdp;
+ /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
+ feat_l3_cdp = NULL;
+ feat->ops = l3_cdp_ops;
+ l3_cdp_init_feature(regs, feat, info);
+ } else {
+ feat = feat_l3_cat;
+ /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
+ feat_l3_cat = NULL;
+ feat->ops = l3_cat_ops;
+ l3_cat_init_feature(regs, feat, info);
+ }
}
}
@@ -1267,6 +1351,14 @@ static int psr_cpu_prepare(void)
(feat_l3_cat = xzalloc(struct feat_node)) == NULL )
return -ENOMEM;
+ if ( feat_l3_cdp == NULL &&
+ (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
+ {
+ xfree(feat_l3_cat);
+ feat_l3_cat = NULL;
+ return -ENOMEM;
+ }
+
return 0;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 14/24] x86: refactor psr: implement get hw info flow for CDP.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (12 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-02-28 14:54 ` Roger Pau Monné
2017-02-15 8:49 ` [PATCH v8 15/24] x86: refactor psr: implement get value " Yi Sun
` (12 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements get HW info flow for CDP including L3 CDP callback
function.
It also changes sysctl function to make it work for CDP.
With this patch, 'psr-hwinfo' can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 18 ++++++++++++++++++
xen/arch/x86/sysctl.c | 24 +++++++++++++++++++++---
2 files changed, 39 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 4c08779..72c9888 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -270,6 +270,10 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
case PSR_CBM_TYPE_L3:
feat_type = PSR_SOCKET_L3_CAT;
break;
+ case PSR_CBM_TYPE_L3_DATA:
+ case PSR_CBM_TYPE_L3_CODE:
+ feat_type = PSR_SOCKET_L3_CDP;
+ break;
default:
feat_type = PSR_SOCKET_UNKNOWN;
break;
@@ -528,8 +532,22 @@ static unsigned int l3_cdp_get_cos_max(const struct feat_node *feat)
return feat->info.l3_cdp_info.cos_max;
}
+static bool l3_cdp_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], uint32_t array_len)
+{
+ if ( !data || 3 > array_len )
+ return false;
+
+ data[CBM_LEN] = feat->info.l3_cdp_info.cbm_len;
+ data[COS_MAX] = feat->info.l3_cdp_info.cos_max;
+ data[PSR_FLAG] |= XEN_SYSCTL_PSR_CAT_L3_CDP;
+
+ return true;
+}
+
struct feat_ops l3_cdp_ops = {
.get_cos_max = l3_cdp_get_cos_max,
+ .get_feat_info = l3_cdp_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index e340baa..568bfe9 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -181,9 +181,27 @@ long arch_do_sysctl(
ret = psr_get_info(sysctl->u.psr_cat_op.target,
PSR_CBM_TYPE_L3, data, 3);
- sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
- sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
- sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
+ if ( !ret )
+ {
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
+ } else {
+ /*
+ * Check if CDP is enabled.
+ *
+ * Per spec, L3 CAT and CDP cannot co-exist. So, we need replace
+ * output values to CDP's if it is enabled.
+ */
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L3_CODE, data, 3);
+ if ( !ret )
+ {
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
+ }
+ }
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
ret = -EFAULT;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 15/24] x86: refactor psr: implement get value flow for CDP.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (13 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 14/24] x86: refactor psr: implement get hw info " Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-28 14:59 ` Roger Pau Monné
2017-02-15 8:49 ` [PATCH v8 16/24] x86: refactor psr: implement set value callback functions " Yi Sun
` (11 subsequent siblings)
26 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements L3 CDP get value callback function.
With this patch, 'psr-cat-show' can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 72c9888..72ed923 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -545,9 +545,25 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
return true;
}
+static bool l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val)
+{
+ if ( cos > feat->info.l3_cdp_info.cos_max )
+ /* Use default value. */
+ cos = 0;
+
+ if ( type == PSR_CBM_TYPE_L3_DATA )
+ *val = get_cdp_data(feat, cos);
+ else
+ *val = get_cdp_code(feat, cos);
+
+ return true;
+}
+
struct feat_ops l3_cdp_ops = {
.get_cos_max = l3_cdp_get_cos_max,
.get_feat_info = l3_cdp_get_feat_info,
+ .get_val = l3_cdp_get_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 16/24] x86: refactor psr: implement set value callback functions for CDP.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (14 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 15/24] x86: refactor psr: implement get value " Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-02-15 8:49 ` [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
` (10 subsequent siblings)
26 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements L3 CDP set value related callback functions.
With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v8:
- modify 'l3_cdp_write_msr' to 'void'.
---
xen/arch/x86/psr.c | 118 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 118 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 72ed923..0a0bab9 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -560,10 +560,128 @@ static bool l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
return true;
}
+static unsigned int l3_cdp_get_cos_num(const struct feat_node *feat)
+{
+ return 2;
+}
+
+static int l3_cdp_get_old_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos)
+{
+ if ( old_cos > feat->info.l3_cdp_info.cos_max )
+ /* Use default value. */
+ old_cos = 0;
+
+ /* Data */
+ val[0] = get_cdp_data(feat, old_cos);
+ /* Code */
+ val[1] = get_cdp_code(feat, old_cos);
+
+ return 0;
+}
+
+static int l3_cdp_set_new_val(uint64_t val[],
+ const struct feat_node *feat,
+ enum cbm_type type,
+ uint64_t m)
+{
+ if ( !psr_check_cbm(feat->info.l3_cdp_info.cbm_len, m) )
+ return -EINVAL;
+
+ if ( type == PSR_CBM_TYPE_L3_DATA )
+ val[0] = m;
+ else
+ val[1] = m;
+
+ return 0;
+}
+
+static int l3_cdp_compare_val(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos, bool *found)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.l3_cdp_info.cbm_len) - 1;
+
+ /*
+ * Different features' cos_max are different. If cos id of the feature
+ * being set exceeds other feature's cos_max, the val of other feature
+ * must be default value. HW supports such case.
+ */
+ if ( cos > feat->info.l3_cdp_info.cos_max )
+ {
+ if ( val[0] != l3_def_cbm ||
+ val[1] != l3_def_cbm )
+ {
+ *found = false;
+ return -ENOENT;
+ }
+ *found = true;
+ }
+ else
+ *found = (val[0] == get_cdp_data(feat, cos) &&
+ val[1] == get_cdp_code(feat, cos));
+
+ return 0;
+}
+
+static bool l3_cdp_fits_cos_max(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.l3_cdp_info.cbm_len) - 1;
+
+ if ( cos > feat->info.l3_cdp_info.cos_max &&
+ (val[0] != l3_def_cbm || val[1] != l3_def_cbm) )
+ /*
+ * Exceed cos_max and value to set is not default,
+ * return error.
+ */
+ return false;
+
+ return true;
+}
+
+static void l3_cdp_write_msr(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat)
+{
+ /*
+ * If input cos is more than the cos_max of the feature, we should
+ * not set the value.
+ */
+ if ( cos > feat->info.l3_cdp_info.cos_max )
+ return;
+
+ /* Data */
+ if ( get_cdp_data(feat, cos) != val[0] )
+ {
+ get_cdp_data(feat, cos) = val[0];
+ wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(cos), val[0]);
+ }
+ /* Code */
+ if ( get_cdp_code(feat, cos) != val[1] )
+ {
+ get_cdp_code(feat, cos) = val[1];
+ wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(cos), val[1]);
+ }
+
+ return;
+}
+
struct feat_ops l3_cdp_ops = {
.get_cos_max = l3_cdp_get_cos_max,
.get_feat_info = l3_cdp_get_feat_info,
.get_val = l3_cdp_get_val,
+ .get_cos_num = l3_cdp_get_cos_num,
+ .get_old_val = l3_cdp_get_old_val,
+ .set_new_val = l3_cdp_set_new_val,
+ .compare_val = l3_cdp_compare_val,
+ .fits_cos_max = l3_cdp_fits_cos_max,
+ .write_msr = l3_cdp_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (15 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 16/24] x86: refactor psr: implement set value callback functions " Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-28 15:15 ` Roger Pau Monné
2017-03-09 15:04 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
` (9 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the CPU init and free flow for L2 CAT including
L2 CAT initialization callback function.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++
xen/include/asm-x86/psr.h | 1 +
2 files changed, 73 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0a0bab9..4489637 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -98,6 +98,7 @@ struct feat_hw_info {
union {
struct psr_cat_hw_info l3_cat_info;
struct psr_cat_hw_info l3_cdp_info;
+ struct psr_cat_hw_info l2_cat_info;
};
};
@@ -235,6 +236,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
*/
static struct feat_node *feat_l3_cat;
static struct feat_node *feat_l3_cdp;
+static struct feat_node *feat_l2_cat;
/* Common functions. */
static void free_feature(struct psr_socket_info *info)
@@ -684,6 +686,53 @@ struct feat_ops l3_cdp_ops = {
.write_msr = l3_cdp_write_msr,
};
+/* L2 CAT callback functions implementation. */
+static void l2_cat_init_feature(struct cpuid_leaf regs,
+ struct feat_node *feat,
+ struct psr_socket_info *info)
+{
+ struct psr_cat_hw_info l2_cat = { };
+ unsigned int socket;
+
+ /* No valid values so do not enable the feature. */
+ if ( !regs.a || !regs.d )
+ return;
+
+ l2_cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
+ l2_cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
+
+ /* cos=0 is reserved as default cbm(all ones). */
+ feat->cos_reg_val[0] = (1ull << l2_cat.cbm_len) - 1;
+
+ feat->feature = PSR_SOCKET_L2_CAT;
+ ASSERT(!test_bit(PSR_SOCKET_L2_CAT, &info->feat_mask));
+ __set_bit(PSR_SOCKET_L2_CAT, &info->feat_mask);
+
+ feat->info.l2_cat_info = l2_cat;
+
+ info->nr_feat++;
+
+ /* Add this feature into list. */
+ list_add_tail(&feat->list, &info->feat_list);
+
+ socket = cpu_to_socket(smp_processor_id());
+ if ( !opt_cpu_info )
+ return;
+
+ printk(XENLOG_INFO "L2 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u.\n",
+ socket, feat->info.l2_cat_info.cos_max,
+ feat->info.l2_cat_info.cbm_len);
+}
+
+static unsigned int l2_cat_get_cos_max(const struct feat_node *feat)
+{
+ return feat->info.l2_cat_info.cos_max;
+}
+
+struct feat_ops l2_cat_ops = {
+ .get_cos_max = l2_cat_get_cos_max,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1456,6 +1505,18 @@ static void cpu_init_work(void)
l3_cat_init_feature(regs, feat, info);
}
}
+
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L2 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, ®s);
+
+ feat = feat_l2_cat;
+ /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
+ feat_l2_cat = NULL;
+ feat->ops = l2_cat_ops;
+ l2_cat_init_feature(regs, feat, info);
+ }
}
static void cpu_fini_work(unsigned int cpu)
@@ -1511,6 +1572,17 @@ static int psr_cpu_prepare(void)
return -ENOMEM;
}
+ if ( feat_l2_cat == NULL &&
+ (feat_l2_cat = xzalloc(struct feat_node)) == NULL )
+ {
+ xfree(feat_l3_cat);
+ feat_l3_cat = NULL;
+
+ xfree(feat_l3_cdp);
+ feat_l3_cdp = NULL;
+ return -ENOMEM;
+ }
+
return 0;
}
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index fde7882..d7ed012 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -23,6 +23,7 @@
/* Resource Type Enumeration */
#define PSR_RESOURCE_TYPE_L3 0x2
+#define PSR_RESOURCE_TYPE_L2 0x4
/* L3 Monitoring Features */
#define PSR_CMT_L3_OCCUPANCY 0x1
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (16 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-28 15:18 ` Roger Pau Monné
2017-03-09 15:13 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 19/24] x86: L2 CAT: implement get value flow Yi Sun
` (8 subsequent siblings)
26 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements get HW info flow for L2 CAT including L2 CAT callback
function.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 16 ++++++++++++++++
xen/arch/x86/sysctl.c | 15 +++++++++++++++
xen/include/asm-x86/psr.h | 1 +
xen/include/public/sysctl.h | 6 ++++++
4 files changed, 38 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 4489637..05100b4 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -276,6 +276,9 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
case PSR_CBM_TYPE_L3_CODE:
feat_type = PSR_SOCKET_L3_CDP;
break;
+ case PSR_CBM_TYPE_L2:
+ feat_type = PSR_SOCKET_L2_CAT;
+ break;
default:
feat_type = PSR_SOCKET_UNKNOWN;
break;
@@ -729,8 +732,21 @@ static unsigned int l2_cat_get_cos_max(const struct feat_node *feat)
return feat->info.l2_cat_info.cos_max;
}
+static bool l2_cat_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], uint32_t array_len)
+{
+ if ( !data || 2 > array_len )
+ return false;
+
+ data[CBM_LEN] = feat->info.l2_cat_info.cbm_len;
+ data[COS_MAX] = feat->info.l2_cat_info.cos_max;
+
+ return true;
+}
+
struct feat_ops l2_cat_ops = {
.get_cos_max = l2_cat_get_cos_max,
+ .get_feat_info = l2_cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 568bfe9..01cf3b7 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -207,6 +207,21 @@ long arch_do_sysctl(
ret = -EFAULT;
break;
}
+ case XEN_SYSCTL_PSR_CAT_get_l2_info:
+ {
+ uint32_t dat[2];
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L2, dat, 2);
+ if ( ret )
+ break;
+
+ sysctl->u.psr_cat_op.u.l2_info.cbm_len = dat[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l2_info.cos_max = dat[COS_MAX];
+
+ if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
+ ret = -EFAULT;
+ break;
+ }
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index d7ed012..2e1b3d0 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -56,6 +56,7 @@ enum cbm_type {
PSR_CBM_TYPE_L3,
PSR_CBM_TYPE_L3_CODE,
PSR_CBM_TYPE_L3_DATA,
+ PSR_CBM_TYPE_L2,
};
extern struct psr_cmt *psr_cmt;
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 00f5e77..cbf5372 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
#define XEN_SYSCTL_PSR_CAT_get_l3_info 0
+#define XEN_SYSCTL_PSR_CAT_get_l2_info 1
struct xen_sysctl_psr_cat_op {
uint32_t cmd; /* IN: XEN_SYSCTL_PSR_CAT_* */
uint32_t target; /* IN */
@@ -754,6 +755,11 @@ struct xen_sysctl_psr_cat_op {
#define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
uint32_t flags; /* OUT: CAT flags */
} l3_info;
+
+ struct {
+ uint32_t cbm_len; /* OUT: CBM length */
+ uint32_t cos_max; /* OUT: Maximum COS */
+ } l2_info;
} u;
};
typedef struct xen_sysctl_psr_cat_op xen_sysctl_psr_cat_op_t;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 19/24] x86: L2 CAT: implement get value flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (17 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-28 15:20 ` Roger Pau Monné
2017-02-15 8:49 ` [PATCH v8 20/24] x86: L2 CAT: implement set " Yi Sun
` (7 subsequent siblings)
26 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements L2 CAT get value callback function and
interface in domctl.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/domctl.c | 7 +++++++
xen/arch/x86/psr.c | 12 ++++++++++++
xen/include/public/domctl.h | 1 +
3 files changed, 20 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index f8f5539..68c2d60 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1460,6 +1460,13 @@ long arch_do_domctl(
copyback = 1;
break;
+ case XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM:
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L2);
+ copyback = 1;
+ break;
+
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 05100b4..2adf62c 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -744,9 +744,21 @@ static bool l2_cat_get_feat_info(const struct feat_node *feat,
return true;
}
+static bool l2_cat_get_val(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val)
+{
+ if ( cos > feat->info.l2_cat_info.cos_max )
+ cos = 0;
+
+ *val = feat->cos_reg_val[cos];
+
+ return true;
+}
+
struct feat_ops l2_cat_ops = {
.get_cos_max = l2_cat_get_cos_max,
.get_feat_info = l2_cat_get_feat_info,
+ .get_val = l2_cat_get_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 85cbb7c..8c183ba 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
uint64_t data; /* IN/OUT */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 20/24] x86: L2 CAT: implement set value flow.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (18 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 19/24] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-28 15:25 ` Roger Pau Monné
2017-02-15 8:49 ` [PATCH v8 21/24] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
` (6 subsequent siblings)
26 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements L2 CAT set value related callback functions
and domctl interface.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v8:
- modify 'l2_cat_write_msr' to 'void'.
---
xen/arch/x86/domctl.c | 6 +++
xen/arch/x86/psr.c | 95 +++++++++++++++++++++++++++++++++++++++++
xen/include/asm-x86/msr-index.h | 1 +
xen/include/public/domctl.h | 1 +
4 files changed, 103 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 68c2d60..38dc087 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1439,6 +1439,12 @@ long arch_do_domctl(
PSR_CBM_TYPE_L3_DATA);
break;
+ case XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM:
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L2);
+ break;
+
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
ret = psr_get_val(d, domctl->u.psr_cat_op.target,
&domctl->u.psr_cat_op.data,
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 2adf62c..5604e03 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -755,10 +755,105 @@ static bool l2_cat_get_val(const struct feat_node *feat, unsigned int cos,
return true;
}
+static unsigned int l2_cat_get_cos_num(const struct feat_node *feat)
+{
+ /* L2 CAT uses one COS. */
+ return 1;
+}
+
+static int l2_cat_get_old_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos)
+{
+ if ( old_cos > feat->info.l2_cat_info.cos_max )
+ /* Use default value. */
+ old_cos = 0;
+
+ val[0] = feat->cos_reg_val[old_cos];
+
+ return 0;
+}
+
+static int l2_cat_set_new_val(uint64_t val[],
+ const struct feat_node *feat,
+ enum cbm_type type,
+ uint64_t m)
+{
+ if ( !psr_check_cbm(feat->info.l2_cat_info.cbm_len, m) )
+ return -EINVAL;
+
+ val[0] = m;
+
+ return 0;
+}
+
+static int l2_cat_compare_val(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos, bool *found)
+{
+ uint64_t l2_def_cbm;
+
+ l2_def_cbm = (1ull << feat->info.l2_cat_info.cbm_len) - 1;
+
+ if ( cos > feat->info.l2_cat_info.cos_max )
+ {
+ if ( val[0] != l2_def_cbm )
+ {
+ *found = false;
+ return -ENOENT;
+ }
+ *found = true;
+ }
+ else
+ *found = (val[0] == feat->cos_reg_val[cos]);
+
+ return 0;
+}
+
+static bool l2_cat_fits_cos_max(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos)
+{
+ uint64_t l2_def_cbm;
+
+ l2_def_cbm = (1ull << feat->info.l2_cat_info.cbm_len) - 1;
+
+ if ( cos > feat->info.l2_cat_info.cos_max &&
+ val[0] != l2_def_cbm )
+ /*
+ * Exceed cos_max and value to set is not default,
+ * return error.
+ */
+ return false;
+
+ return true;
+}
+
+static void l2_cat_write_msr(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat)
+{
+ if ( cos > feat->info.l2_cat_info.cos_max )
+ return;
+
+ if ( feat->cos_reg_val[cos] != val[0] )
+ {
+ feat->cos_reg_val[cos] = val[0];
+ wrmsrl(MSR_IA32_PSR_L2_MASK(cos), val[0]);
+ }
+
+ return;
+}
+
struct feat_ops l2_cat_ops = {
.get_cos_max = l2_cat_get_cos_max,
.get_feat_info = l2_cat_get_feat_info,
.get_val = l2_cat_get_val,
+ .get_cos_num = l2_cat_get_cos_num,
+ .get_old_val = l2_cat_get_old_val,
+ .set_new_val = l2_cat_set_new_val,
+ .compare_val = l2_cat_compare_val,
+ .fits_cos_max = l2_cat_fits_cos_max,
+ .write_msr = l2_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h
index 98dbff1..a41e63a 100644
--- a/xen/include/asm-x86/msr-index.h
+++ b/xen/include/asm-x86/msr-index.h
@@ -343,6 +343,7 @@
#define MSR_IA32_PSR_L3_MASK(n) (0x00000c90 + (n))
#define MSR_IA32_PSR_L3_MASK_CODE(n) (0x00000c90 + (n) * 2 + 1)
#define MSR_IA32_PSR_L3_MASK_DATA(n) (0x00000c90 + (n) * 2)
+#define MSR_IA32_PSR_L2_MASK(n) (0x00000d10 + (n))
/* Intel Model 6 */
#define MSR_P6_PERFCTR(n) (0x000000c1 + (n))
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 8c183ba..523a2cd 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM 6
#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 21/24] tools: L2 CAT: support get HW info for L2 CAT.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (19 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 20/24] x86: L2 CAT: implement set " Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-15 8:49 ` [PATCH v8 22/24] tools: L2 CAT: support show cbm " Yi Sun
` (5 subsequent siblings)
26 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements xl/xc changes to support get HW info
for L2 CAT.
'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
info.
Example(on machine which only supports L2 CAT):
Cache Monitoring Technology (CMT):
Enabled : 0
Cache Allocation Technology (CAT): L2
Socket ID : 0
Maximum COS : 3
CBM length : 8
Default CBM : 0xff
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/include/xenctrl.h | 6 ++---
tools/libxc/xc_psr.c | 39 +++++++++++++++++++++++---------
tools/libxl/libxl.h | 9 ++++++++
tools/libxl/libxl_psr.c | 19 +++++++++++-----
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 52 +++++++++++++++++++++++++++++++++----------
6 files changed, 95 insertions(+), 31 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 85d7fe5..7308332 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2637,9 +2637,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
xc_psr_cat_type type, uint32_t target,
uint64_t *data);
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled);
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len,
+ bool *cdp_enabled);
int xc_get_cpu_levelling_caps(xc_interface *xch, uint32_t *caps);
int xc_get_cpu_featureset(xc_interface *xch, uint32_t index,
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 43b3286..de40615 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -317,24 +317,41 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
return rc;
}
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled)
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len, bool *cdp_enabled)
{
- int rc;
+ int rc = -1;
DECLARE_SYSCTL;
sysctl.cmd = XEN_SYSCTL_psr_cat_op;
- sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
sysctl.u.psr_cat_op.target = socket;
- rc = xc_sysctl(xch, &sysctl);
- if ( !rc )
+ switch ( lvl )
{
- *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
- *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
- *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
- XEN_SYSCTL_PSR_CAT_L3_CDP;
+ case 2:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l2_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.l2_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.l2_info.cbm_len;
+ *cdp_enabled = false;
+ }
+ break;
+ case 3:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
+ *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
+ XEN_SYSCTL_PSR_CAT_L3_CDP;
+ }
+ break;
+ default:
+ errno = EOPNOTSUPP;
+ break;
}
return rc;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 3924464..c75a928 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -904,6 +904,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, const libxl_mac *src);
* If this is defined, the Code and Data Prioritization feature is supported.
*/
#define LIBXL_HAVE_PSR_CDP 1
+
+/*
+ * LIBXL_HAVE_PSR_L2_CAT
+ *
+ * If this is defined, the L2 Cache Allocation Technology feature is supported.
+ */
+#define LIBXL_HAVE_PSR_L2_CAT 1
#endif
/*
@@ -2166,6 +2173,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
* On success, the function returns an array of elements in 'info',
* and the length in 'nr'.
*/
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl);
int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
int *nr);
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr);
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index ec5c79d..68c45da 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -352,8 +352,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
return rc;
}
-int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
- int *nr)
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl)
{
GC_INIT(ctx);
int rc;
@@ -380,9 +380,8 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
libxl_for_each_set_bit(socketid, socketmap) {
ptr[i].id = socketid;
- if (xc_psr_cat_get_l3_info(ctx->xch, socketid, &ptr[i].cos_max,
- &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
- libxl__psr_cat_log_err_msg(gc, errno);
+ if (xc_psr_cat_get_info(ctx->xch, socketid, lvl, &ptr[i].cos_max,
+ &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
rc = ERROR_FAIL;
free(ptr);
goto out;
@@ -398,6 +397,16 @@ out:
return rc;
}
+int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr)
+{
+ int rc;
+
+ rc = libxl_psr_cat_get_info(ctx, info, nr, 3);
+
+ return rc;
+}
+
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr)
{
int i;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a612d1f..5a401b8 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -899,6 +899,7 @@ libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
(1, "L3_CBM"),
(2, "L3_CBM_CODE"),
(3, "L3_CBM_DATA"),
+ (4, "L2_CBM"),
])
libxl_psr_cat_info = Struct("psr_cat_info", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 37ebdce..3d307fa 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9340,21 +9340,19 @@ int main_psr_cmt_show(int argc, char **argv)
}
#endif
-#ifdef LIBXL_HAVE_PSR_CAT
-static int psr_cat_hwinfo(void)
+#if defined(LIBXL_HAVE_PSR_CAT) || defined(LIBXL_HAVE_PSR_L2_CAT)
+static int psr_l3_cat_hwinfo(void)
{
- int rc;
- int i, nr;
+ int rc, nr;
+ unsigned int i;
uint32_t l3_cache_size;
libxl_psr_cat_info *info;
- printf("Cache Allocation Technology (CAT):\n");
-
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
- if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (rc)
return rc;
- }
+
+ printf("Cache Allocation Technology (CAT): L3\n");
for (i = 0; i < nr; i++) {
rc = libxl_psr_cmt_get_l3_cache_size(ctx, info[i].id, &l3_cache_size);
@@ -9463,7 +9461,7 @@ static int psr_cat_show(uint32_t domid)
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
if (rc) {
fprintf(stderr, "Failed to get cat info\n");
return rc;
@@ -9480,6 +9478,32 @@ out:
return rc;
}
+static int psr_l2_cat_hwinfo(void)
+{
+ int rc;
+ unsigned int i;
+ int nr;
+ libxl_psr_cat_info *info;
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 2);
+ if (rc)
+ return rc;
+
+ printf("Cache Allocation Technology (CAT): L2\n");
+
+ for (i = 0; i < nr; i++) {
+ /* There is no CMT on L2 cache so far. */
+ printf("%-16s: %u\n", "Socket ID", info[i].id);
+ printf("%-16s: %u\n", "Maximum COS", info[i].cos_max);
+ printf("%-16s: %u\n", "CBM length", info[i].cbm_len);
+ printf("%-16s: %#llx\n", "Default CBM",
+ (1ull << info[i].cbm_len) - 1);
+ }
+
+ libxl_psr_cat_info_list_free(info, nr);
+ return rc;
+}
+
int main_psr_cat_cbm_set(int argc, char **argv)
{
uint32_t domid;
@@ -9597,7 +9621,11 @@ int main_psr_hwinfo(int argc, char **argv)
ret = psr_cmt_hwinfo();
if (!ret && (all || cat))
- ret = psr_cat_hwinfo();
+ ret = psr_l3_cat_hwinfo();
+
+ /* L2 CAT is independent of CMT and L3 CAT */
+ if (all || cat)
+ ret = psr_l2_cat_hwinfo();
return ret;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 22/24] tools: L2 CAT: support show cbm for L2 CAT.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (20 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 21/24] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-15 8:49 ` [PATCH v8 23/24] tools: L2 CAT: support set " Yi Sun
` (4 subsequent siblings)
26 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements changes in xl/xc changes to support
showing CBM of L2 CAT.
The new level option is introduced to original CAT showing
command in order to show CBM for specified level CAT.
- 'xl psr-cat-show' is updated to show CBM of a domain
according to input cache level.
Examples:
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/include/xenctrl.h | 1 +
tools/libxc/xc_psr.c | 3 ++
tools/libxl/xl_cmdimpl.c | 85 +++++++++++++++++++++++++++++--------------
tools/libxl/xl_cmdtable.c | 3 +-
4 files changed, 63 insertions(+), 29 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 7308332..0e2c2d8 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2613,6 +2613,7 @@ enum xc_psr_cat_type {
XC_PSR_CAT_L3_CBM = 1,
XC_PSR_CAT_L3_CBM_CODE = 2,
XC_PSR_CAT_L3_CBM_DATA = 3,
+ XC_PSR_CAT_L2_CBM = 4,
};
typedef enum xc_psr_cat_type xc_psr_cat_type;
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index de40615..6f114fb 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -299,6 +299,9 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 3d307fa..9d93ddd 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9388,7 +9388,7 @@ static void psr_cat_print_one_domain_cbm_type(uint32_t domid, uint32_t socketid,
}
static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
char *domain_name;
@@ -9396,27 +9396,38 @@ static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
printf("%5d%25s", domid, domain_name);
free(domain_name);
- if (!cdp_enabled) {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM);
- } else {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ switch (lvl) {
+ case 3:
+ if (!cdp_enabled) {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM);
+ } else {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ }
+ break;
+ case 2:
psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ LIBXL_PSR_CBM_TYPE_L2_CBM);
+ break;
+ default:
+ printf("Input lvl %d is wrong!", lvl);
+ break;
}
printf("\n");
}
static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
int i, nr_domains;
libxl_dominfo *list;
if (domid != INVALID_DOMID) {
- psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled, lvl);
return 0;
}
@@ -9426,49 +9437,59 @@ static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
}
for (i = 0; i < nr_domains; i++)
- psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled, lvl);
libxl_dominfo_list_free(list, nr_domains);
return 0;
}
-static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
+static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info,
+ unsigned int lvl)
{
int rc;
uint32_t l3_cache_size;
- rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
- if (rc) {
- fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
- info->id);
- return -1;
+ printf("%-16s: %u\n", "Socket ID", info->id);
+
+ /* So far, CMT only supports L3 cache. */
+ if (lvl == 3) {
+ rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
+ if (rc) {
+ fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
+ info->id);
+ return -1;
+ }
+ printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
}
- printf("%-16s: %u\n", "Socket ID", info->id);
- printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
printf("%-16s: %#llx\n", "Default CBM", (1ull << info->cbm_len) - 1);
if (info->cdp_enabled)
printf("%5s%25s%16s%16s\n", "ID", "NAME", "CBM (code)", "CBM (data)");
else
printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
- return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled);
+ return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled, lvl);
}
-static int psr_cat_show(uint32_t domid)
+static int psr_cat_show(uint32_t domid, unsigned int lvl)
{
int i, nr;
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (lvl != 2 && lvl != 3) {
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
+ }
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, lvl);
if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ fprintf(stderr, "Failed to get %s cat info\n", (lvl == 3)?"L3":"L2");
return rc;
}
for (i = 0; i < nr; i++) {
- rc = psr_cat_print_socket(domid, info + i);
+ rc = psr_cat_print_socket(domid, info + i, lvl);
if (rc)
goto out;
}
@@ -9579,11 +9600,19 @@ int main_psr_cat_cbm_set(int argc, char **argv)
int main_psr_cat_show(int argc, char **argv)
{
- int opt;
+ int opt = 0;
uint32_t domid;
+ unsigned int lvl = 3;
- SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cat-show", 0) {
- /* No options */
+ static struct option opts[] = {
+ {"level", 1, 0, 'l'},
+ COMMON_LONG_OPTS
+ };
+
+ SWITCH_FOREACH_OPT(opt, "l:", opts, "psr-cat-show", 0) {
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
if (optind >= argc)
@@ -9595,7 +9624,7 @@ int main_psr_cat_show(int argc, char **argv)
return 2;
}
- return psr_cat_show(domid);
+ return psr_cat_show(domid, lvl);
}
int main_psr_hwinfo(int argc, char **argv)
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index 588d5d9..c5fbad4 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -556,7 +556,8 @@ struct cmd_spec cmd_table[] = {
{ "psr-cat-show",
&main_psr_cat_show, 0, 1,
"Show Cache Allocation Technology information",
- "<Domain>",
+ "[options] <Domain>",
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
},
#endif
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 23/24] tools: L2 CAT: support set cbm for L2 CAT.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (21 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 22/24] tools: L2 CAT: support show cbm " Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-15 8:49 ` [PATCH v8 24/24] docs: add L2 CAT description in docs Yi Sun
` (3 subsequent siblings)
26 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the xl/xc changes to support set CBM
for L2 CAT.
The new level option is introduced to original CAT setting
command in order to set CBM for specified level CAT.
- 'xl psr-cat-cbm-set' is updated to set cache capacity
bitmasks(CBM) for a domain according to input cache level.
root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/xc_psr.c | 3 +++
tools/libxl/xl_cmdimpl.c | 34 +++++++++++++++++++++++-----------
tools/libxl/xl_cmdtable.c | 3 ++-
3 files changed, 28 insertions(+), 12 deletions(-)
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 6f114fb..8fd6812 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -266,6 +266,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 9d93ddd..2faadf7 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9536,19 +9536,21 @@ int main_psr_cat_cbm_set(int argc, char **argv)
char *value;
libxl_string_list socket_list;
unsigned long start, end;
- int i, j, len;
+ unsigned int i, j, len;
+ unsigned int lvl = 3;
static struct option opts[] = {
{"socket", 1, 0, 's'},
{"data", 0, 0, 'd'},
{"code", 0, 0, 'c'},
+ {"level", 1, 0, 'l'},
COMMON_LONG_OPTS
};
libxl_socket_bitmap_alloc(ctx, &target_map, 0);
libxl_bitmap_set_none(&target_map);
- SWITCH_FOREACH_OPT(opt, "s:cd", opts, "psr-cat-cbm-set", 2) {
+ SWITCH_FOREACH_OPT(opt, "s:l:cd", opts, "psr-cat-set", 2) {
case 's':
trim(isspace, optarg, &value);
split_string_into_string_list(value, ",", &socket_list);
@@ -9568,24 +9570,34 @@ int main_psr_cat_cbm_set(int argc, char **argv)
case 'c':
opt_code = 1;
break;
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
- if (opt_data && opt_code) {
- fprintf(stderr, "Cannot handle -c and -d at the same time\n");
- return -1;
- } else if (opt_data) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
- } else if (opt_code) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ if (lvl == 2)
+ type = LIBXL_PSR_CBM_TYPE_L2_CBM;
+ else if (lvl == 3) {
+ if (opt_data && opt_code) {
+ fprintf(stderr, "Cannot handle -c and -d at the same time\n");
+ return EXIT_FAILURE;
+ } else if (opt_data) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
+ } else if (opt_code) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ } else {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ }
} else {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
}
if (libxl_bitmap_is_empty(&target_map))
libxl_bitmap_set_any(&target_map);
if (argc != optind + 2) {
- help("psr-cat-cbm-set");
+ help("psr-cat-set");
return 2;
}
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index c5fbad4..433ed6a 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -545,11 +545,12 @@ struct cmd_spec cmd_table[] = {
},
#endif
#ifdef LIBXL_HAVE_PSR_CAT
- { "psr-cat-cbm-set",
+ { "psr-cat-set",
&main_psr_cat_cbm_set, 0, 1,
"Set cache capacity bitmasks(CBM) for a domain",
"[options] <Domain> <CBM>",
"-s <socket> Specify the socket to process, otherwise all sockets are processed\n"
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
"-c Set code CBM if CDP is supported\n"
"-d Set data CBM if CDP is supported\n"
},
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v8 24/24] docs: add L2 CAT description in docs.
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (22 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 23/24] tools: L2 CAT: support set " Yi Sun
@ 2017-02-15 8:49 ` Yi Sun
2017-02-15 16:14 ` [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Konrad Rzeszutek Wilk
` (2 subsequent siblings)
26 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-15 8:49 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch adds L2 CAT description in related documents.
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
docs/man/xl.pod.1.in | 25 ++++++++++++++++++++++---
docs/misc/xl-psr.markdown | 10 ++++++++--
2 files changed, 30 insertions(+), 5 deletions(-)
diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 7caed08..5e7676e 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -1711,6 +1711,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
a domain is attached to the monitoring service, monitoring data can be shown
for any of these monitoring types.
+There is no cache monitoring and memory bandwidth monitoring on L2 cache so
+far.
+
=over 4
=item B<psr-cmt-attach> [I<domain-id>]
@@ -1735,7 +1738,7 @@ monitor types are:
Intel Broadwell and later server platforms offer capabilities to configure and
make use of the Cache Allocation Technology (CAT) mechanisms, which enable more
-cache resources (i.e. L3 cache) to be made available for high priority
+cache resources (i.e. L3/L2 cache) to be made available for high priority
applications. In the Xen implementation, CAT is used to control cache allocation
on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
(CBM) for the domain.
@@ -1745,7 +1748,7 @@ Intel Broadwell and later server platforms also offer Code/Data Prioritization
applications. CDP is used on a per VM basis in the Xen implementation. To
specify code or data CBM for the domain, CDP feature must be enabled and CBM
type options need to be specified when setting CBM, and the type options (code
-and data) are mutually exclusive.
+and data) are mutually exclusive. There is no CDP support on L2 so far.
=over 4
@@ -1762,6 +1765,11 @@ B<OPTIONS>
Specify the socket to process, otherwise all sockets are processed.
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
=item B<-c>, B<--code>
Set code CBM when CDP is enabled.
@@ -1772,10 +1780,21 @@ Set data CBM when CDP is enabled.
=back
-=item B<psr-cat-show> [I<domain-id>]
+=item B<psr-cat-show> [I<OPTIONS>] [I<domain-id>]
Show CAT settings for a certain domain or all domains.
+B<OPTIONS>
+
+=over 4
+
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
+=back
+
=back
=head1 IGNORED FOR COMPATIBILITY WITH XM
diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
index c3c1e8e..bd2b6bd 100644
--- a/docs/misc/xl-psr.markdown
+++ b/docs/misc/xl-psr.markdown
@@ -70,7 +70,7 @@ total-mem-bandwidth instead of cache-occupancy). E.g. after a `xl psr-cmt-attach
Cache Allocation Technology (CAT) is a new feature available on Intel
Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
-partition cache allocation (i.e. L3 cache) based on application priority or
+partition cache allocation (i.e. L3/L2 cache) based on application priority or
Class of Service (COS). Each COS is configured using capacity bitmasks (CBM)
which represent cache capacity and indicate the degree of overlap and
isolation between classes. System cache resource is divided into numbers of
@@ -119,13 +119,19 @@ A cbm is valid only when:
In a multi-socket system, the same cbm will be set on each socket by default.
Per socket cbm can be specified with the `--socket SOCKET` option.
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
+
Setting the CBM may not be successful if insufficient COS is available. In
such case unused COS(es) may be freed by setting CBM of all related domains to
its default value(all-ones).
Per domain CBM settings can be shown by:
-`xl psr-cat-show`
+`xl psr-cat-show [OPTIONS] <domid>`
+
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
## Code and Data Prioritization (CDP)
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (23 preceding siblings ...)
2017-02-15 8:49 ` [PATCH v8 24/24] docs: add L2 CAT description in docs Yi Sun
@ 2017-02-15 16:14 ` Konrad Rzeszutek Wilk
2017-02-26 18:00 ` Wei Liu
2017-02-28 11:02 ` Roger Pau Monné
26 siblings, 0 replies; 122+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-15 16:14 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:15PM +0800, Yi Sun wrote:
> Hi all,
>
> We plan to bring a new PSR (Platform Shared Resource) feature called
> Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
>
> Besides the L2 CAT implementaion, we refactor the psr.c to make it more
> flexible to add new features and fulfill the principle, open for extension
> but closed for modification. We abstract the general operations of all
> features and encapsulate them into a structure. Then, the development
> of new feature is simple to mainly implement these callback functions.
>
> The patch set can be found at:
> https://github.com/yisun-git/xen.git l2_cat_v8
>
> v7:
> - patch 1:
> - change revision info.
> - add content int 'Areas for improvement'.
> - patch 4:
> - remove unused parameter of psr_cpu_prepare.
> - add comments in cpu_fini_work.
> - coding style fix.
> - patch 12:
> - modify 'write_msr' callback function to 'void' because we have to set
> all features' cbm. When input cos exceeds some features' cos_max, just
> skip them but not break the iteration.
> - patch 16:
> - modify 'l3_cdp_write_msr' to 'void'.
> - patch 20:
> - modify 'l2_cat_write_msr' to 'void'.
While I appreciate you acting so quickly on the changes it is kind of hard to
review while you repost so quickly.
Please in the future just respond to your patch (say #4)
with an updated patch (v6.1 for example) which would have
the changes requested.
Ditto for #12, #16 and #20.
Unless of course there are some big changes to be done (those
above seem pretty small).
Also you may want to include a legend of which patch has been acked,
reviewed and which needs attention.
Something like:
a - acked
r - reviewed-by
ar docs: create Cache Allocation Technology (CAT) and Code and Data
ar x86: refactor psr: remove L3 CAT/CDP codes.
ar x86: refactor psr: implement main data structures.
..snip..
x86: refactor psr: implement get hw info flow for CDP.
And so on. That helps folks figure out which to jump
to.
Or you can say at the description:
PAtch #1->#6 have been Reviewed.
#7-blah need help, etc.
Thanks!
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-02-15 8:49 ` [PATCH v8 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
@ 2017-02-15 16:49 ` Konrad Rzeszutek Wilk
2017-02-26 17:40 ` Wei Liu
1 sibling, 0 replies; 122+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-15 16:49 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:16PM +0800, Yi Sun wrote:
> This patch creates CAT and CDP feature document in doc/features/. It describes
> key points to implement L3 CAT/CDP and L2 CAT which is described in details in
> Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES".
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> v8:
> - change revision info.
> - add content int 'Areas for improvement'.
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-02-15 8:49 ` [PATCH v8 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-02-15 16:49 ` Konrad Rzeszutek Wilk
@ 2017-02-26 17:40 ` Wei Liu
1 sibling, 0 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:40 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:16PM +0800, Yi Sun wrote:
> This patch creates CAT and CDP feature document in doc/features/. It describes
> key points to implement L3 CAT/CDP and L2 CAT which is described in details in
> Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES".
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
> + 3. `psr-hwinfo [OPTIONS]`:
> +
> + Show CMT & L2 CAT & L3 CAT/CDP HW information on every socket.
Add "CMT" to the list of acronyms as well?
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 02/24] x86: refactor psr: remove L3 CAT/CDP codes.
2017-02-15 8:49 ` [PATCH v8 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
@ 2017-02-26 17:40 ` Wei Liu
0 siblings, 0 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:40 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:17PM +0800, Yi Sun wrote:
> The current cache allocation codes in psr.c do not consider
> future features addition and are not friendly to extend.
>
> To make psr.c be more flexible to add new features and fulfill
> the program principle, open for extension but closed for
> modification, we have to refactor the psr.c:
> 1. Analyze cache allocation features and abstract general data
> structures.
> 2. Analyze the init and all other functions flow, abstract all
> steps that different features may have different implementations.
> Make these steps be callback functions and register feature
> specific fuctions. Then, the main processes will not be changed
> when introducing a new feature.
>
> Because the quantity of refactor codes is big and the logics are
> changed a lot, it will cause reviewers confused if just change
> old codes. Reviewers have to understand both old codes and new
> implementations. After review iterations from V1 to V3, Jan has
> proposed to remove all old cache allocation codes firstly, then
> implement new codes step by step. This will help to make codes
> be more easily reviewable.
>
> There is no construction without destruction. So, this patch
> removes all current L3 CAT/CDP codes in psr.c. The following
> patches will introduce the new mechanism.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> Acked-by: Jan Beulich <jbeulich@suse.com>
> Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-15 8:49 ` [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
@ 2017-02-26 17:41 ` Wei Liu
2017-02-27 6:42 ` Yi Sun
2017-02-27 8:41 ` Jan Beulich
2017-03-08 14:56 ` Jan Beulich
1 sibling, 2 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:41 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:19PM +0800, Yi Sun wrote:
> This patch implements the CPU init and free flow including L3 CAT
> initialization and feature list free.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Either you need to use a separate patch to move cpuid_count_leaf or you
should state it is moved in the commit message.
> +
> +/* Common functions. */
> +static void free_feature(struct psr_socket_info *info)
> +{
> + struct feat_node *feat, *next;
> +
> + if ( !info )
> + return;
> +
> + /*
> + * Free resources of features. But we do not free global feature list
> + * entry, like feat_l3_cat. Although it may cause a few memory leak,
> + * it is OK simplify things.
I don't think it is OK to leak memory in the hypervisor in general.
Please specify why it is OK in this particular case in the comment.
> + */
> + list_for_each_entry_safe(feat, next, &info->feat_list, list)
> + {
> + if ( !feat )
> + return;
> +
> + __clear_bit(feat->feature, &info->feat_mask);
> + list_del(&feat->list);
> + xfree(feat);
> + }
> +}
> +
> -static int psr_cpu_prepare(unsigned int cpu)
> +static void cpu_init_work(void)
> +{
> + struct psr_socket_info *info;
> + unsigned int socket;
> + unsigned int cpu = smp_processor_id();
> + struct feat_node *feat;
> + struct cpuid_leaf regs = { .a = 0, .b = 0, .c = 0, .d = 0 };
> +
> + if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
> + return;
> + else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
> + {
> + __clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
> + return;
> + }
> +
> + socket = cpu_to_socket(cpu);
> + info = socket_info + socket;
> + if ( info->feat_mask )
> + return;
> +
> + INIT_LIST_HEAD(&info->feat_list);
> + spin_lock_init(&info->ref_lock);
> +
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> + if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> + {
You can move
struct feat_node *feat
here.
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> +
> + feat = feat_l3_cat;
> + /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
> + feat_l3_cat = NULL;
> + feat->ops = l3_cat_ops;
> +
> + l3_cat_init_feature(regs, feat, info);
> + }
> +}
> +
[...]
>
> @@ -359,7 +528,7 @@ static int cpu_callback(
> switch ( action )
> {
> case CPU_UP_PREPARE:
> - rc = psr_cpu_prepare(cpu);
> + rc = psr_cpu_prepare();
> break;
> case CPU_STARTING:
> psr_cpu_init();
> @@ -388,10 +557,14 @@ static int __init psr_presmp_init(void)
> if ( (opt_psr & PSR_CMT) && opt_rmid_max )
> init_psr_cmt(opt_rmid_max);
>
> - psr_cpu_prepare(0);
> + if ( opt_psr & PSR_CAT )
> + init_psr();
> +
> + if ( psr_cpu_prepare() )
> + psr_free();
>
> psr_cpu_init();
> - if ( psr_cmt_enabled() )
> + if ( psr_cmt_enabled() || socket_info )
Why not have psr_cat_enabled() here?
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 05/24] x86: refactor psr: implement Domain init/free and schedule flows.
2017-02-15 8:49 ` [PATCH v8 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
@ 2017-02-26 17:41 ` Wei Liu
2017-03-08 15:04 ` Jan Beulich
1 sibling, 0 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:41 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:20PM +0800, Yi Sun wrote:
> +static inline unsigned int get_max_cos_max(const struct psr_socket_info *info)
> +{
> + const struct feat_node *feat;
> + unsigned int cos_max = 0;
> +
> + list_for_each_entry(feat, &info->feat_list, list)
> + cos_max = max(feat->ops.get_cos_max(feat), cos_max);
> +
> + return cos_max;
> +}
> +
> static inline void psr_assoc_init(void)
> {
> struct psr_assoc *psra = &this_cpu(psr_assoc);
>
> - if ( psr_cmt_enabled() )
> + if ( socket_info )
If you choose to follow my suggestion to introduce a psr_???_enabled
function, you can use it here.
> + {
> + unsigned int socket = cpu_to_socket(smp_processor_id());
> + const struct psr_socket_info *info = socket_info + socket;
> + unsigned int cos_max = get_max_cos_max(info);
> +
> + if ( info->feat_mask )
> + psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
> + PSR_ASSOC_REG_SHIFT;
> + }
> +
> + if ( psr_cmt_enabled() || psra->cos_mask )
> rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
> }
>
> @@ -375,6 +405,13 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
> *reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
> }
>
> +static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
> + uint64_t cos_mask)
> +{
> + *reg = (*reg & ~cos_mask) |
> + (((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
> +}
> +
> void psr_ctxt_switch_to(struct domain *d)
> {
> struct psr_assoc *psra = &this_cpu(psr_assoc);
> @@ -383,6 +420,11 @@ void psr_ctxt_switch_to(struct domain *d)
> if ( psr_cmt_enabled() )
> psr_assoc_rmid(®, d->arch.psr_rmid);
>
> + if ( psra->cos_mask )
> + psr_assoc_cos(®, d->arch.psr_cos_ids ?
> + d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
> + 0, psra->cos_mask);
> +
> if ( reg != psra->val )
> {
> wrmsrl(MSR_IA32_PSR_ASSOC, reg);
> @@ -408,14 +450,32 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> return 0;
> }
>
> +/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> +static void psr_free_cos(struct domain *d)
> +{
> + if( !d->arch.psr_cos_ids )
Coding style issue. You actually fixed it in a later patch. Please fix
it here instead.
> + return;
> +
> + xfree(d->arch.psr_cos_ids);
> + d->arch.psr_cos_ids = NULL;
> +}
> +
> int psr_domain_init(struct domain *d)
> {
> + if ( socket_info )
> + {
> + d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
> + if ( !d->arch.psr_cos_ids )
> + return -ENOMEM;
> + }
> +
I suggest you encapsulate this snippet into psr_alloc_cos to match
psr_free_cos.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 06/24] x86: refactor psr: implement get hw info flow.
2017-02-15 8:49 ` [PATCH v8 06/24] x86: refactor psr: implement get hw info flow Yi Sun
@ 2017-02-26 17:41 ` Wei Liu
2017-02-28 12:34 ` Roger Pau Monné
2017-03-08 15:15 ` Jan Beulich
2 siblings, 0 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:41 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:21PM +0800, Yi Sun wrote:
[...]
> /* L3 CAT functions implementation. */
> static void l3_cat_init_feature(struct cpuid_leaf regs,
> struct feat_node *feat,
> @@ -225,8 +247,22 @@ static unsigned int l3_cat_get_cos_max(const struct feat_node *feat)
> return feat->info.l3_cat_info.cos_max;
> }
>
> +static bool l3_cat_get_feat_info(const struct feat_node *feat,
> + uint32_t data[], unsigned int array_len)
> +{
> + if ( !data || 3 > array_len )
I don't think this is documented, but this is the first time I see the
constant on the left. Maybe considering swapping it to array_len < 3?
Other than this:
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-02-15 8:49 ` [PATCH v8 08/24] x86: refactor psr: set value: implement framework Yi Sun
@ 2017-02-26 17:41 ` Wei Liu
2017-02-27 7:06 ` Yi Sun
2017-02-28 13:58 ` Roger Pau Monné
2017-03-08 16:07 ` Jan Beulich
2 siblings, 1 reply; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:41 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:23PM +0800, Yi Sun wrote:
[...]
> +int psr_set_val(struct domain *d, unsigned int socket,
> + uint64_t val, enum cbm_type type)
IMHO it would be far better to use goto style error handling in such a
complex function. You can avoid missing one of the exit paths when
refactoring this function later.
> +{
> + unsigned int old_cos;
> + int cos, ret;
> + unsigned int *ref;
> + uint64_t *val_array;
> + struct psr_socket_info *info = get_socket_info(socket);
> + uint32_t array_len;
> + enum psr_feat_type feat_type;
> +
> + if ( IS_ERR(info) )
> + return PTR_ERR(info);
> +
> + feat_type = psr_cbm_type_to_feat_type(type);
> + if ( !test_bit(feat_type, &info->feat_mask) )
> + return -ENOENT;
> +
> + /*
> + * Step 0:
> + * old_cos means the COS ID current domain is using. By default, it is 0.
> + *
> + * For every COS ID, there is a reference count to record how many domains
> + * are using the COS register corresponding to this COS ID.
> + * - If ref[old_cos] is 0, that means this COS is not used by any domain.
> + * - If ref[old_cos] is 1, that means this COS is only used by current
> + * domain.
> + * - If ref[old_cos] is more than 1, that mean multiple domains are using
> + * this COS.
> + */
> + old_cos = d->arch.psr_cos_ids[socket];
> + if ( old_cos > MAX_COS_REG_CNT )
How could this happen? This function is the setter of cos, it is a bug
if it ever sets a value larger than MAX_COS_REG_CNT.
> + return -EOVERFLOW;
> +
> + ref = info->cos_ref;
> +
> + /*
> + * Step 1:
> + * Assemle a value array to store all featues cos_reg_val[old_cos].
> + * And, set the input val into array according to the feature's
> + * position in array.
> + */
> + array_len = get_cos_num(info);
> + val_array = xzalloc_array(uint64_t, array_len);
> + if ( !val_array )
> + return -ENOMEM;
> +
> + if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 )
> + {
> + xfree(val_array);
> + return ret;
> + }
> +
> + if ( (ret = set_new_val_to_array(val_array, array_len, info,
> + feat_type, type, val)) != 0 )
> + {
> + xfree(val_array);
> + return ret;
> + }
> +
> + /*
> + * Lock here to make sure the ref is not changed during find and
> + * write process.
> + */
> + spin_lock(&info->ref_lock);
> +
> + /*
> + * Step 2:
> + * Try to find if there is already a COS ID on which all features' values
> + * are same as the array. Then, we can reuse this COS ID.
> + */
> + cos = find_cos(val_array, array_len, feat_type, info);
> + if ( cos >= 0 )
> + {
> + if ( cos == old_cos )
> + {
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return 0;
> + }
> + }
> + else
> + {
> + /*
> + * Step 3:
> + * If fail to find, we need allocate a new COS ID.
> + * If multiple domains are using same COS ID, its ref is more
> + * than 1. That means we cannot free this COS to make current domain
> + * use it. Because other domains are using the value saved in the COS.
> + * Unless the ref is changed to 1 (mean only current domain is using
> + * it), we cannot allocate the COS ID to current domain.
> + * So, only the COS ID which ref is 1 or 0 can be allocated.
> + */
> + cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
> + if ( cos < 0 )
> + {
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return cos;
> + }
> +
> + /*
> + * Step 4:
> + * Write all features MSRs according to the COS ID.
> + */
> + ret = write_psr_msr(socket, cos, val_array);
> + if ( ret )
> + {
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return ret;
> + }
> + }
> +
> + /*
> + * Step 5:
> + * Update ref according to COS ID.
> + */
> + ref[cos]++;
> + ASSERT(ref[cos] || cos == 0);
> + ref[old_cos]--;
> + spin_unlock(&info->ref_lock);
> +
> + /*
> + * Step 6:
> + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> + * which COS the domain is using on the socket. One domain can only use
> + * one COS ID at same time on each socket.
> + */
> + d->arch.psr_cos_ids[socket] = cos;
> + xfree(val_array);
> +
> + return 0;
> +}
> +
> /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> static void psr_free_cos(struct domain *d)
> {
> - if( !d->arch.psr_cos_ids )
> + unsigned int socket, cos;
> +
> + if ( !d->arch.psr_cos_ids )
> return;
>
> + /* Domain is free so its cos_ref should be decreased. */
> + for ( socket = 0; socket < nr_sockets; socket++ )
> + {
> + struct psr_socket_info *info;
> +
> + /* cos 0 is default one which does not need be handled. */
> + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
Break this into two lines. The assignment doesn't have to happen within
if().
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-02-15 8:49 ` [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
@ 2017-02-26 17:43 ` Wei Liu
2017-02-27 7:11 ` Yi Sun
2017-03-08 16:54 ` Jan Beulich
1 sibling, 1 reply; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:43 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:24PM +0800, Yi Sun wrote:
[...]
>
> +static unsigned int l3_cat_get_cos_num(const struct feat_node *feat)
> +{
> + return 1;
> +}
> +
> +static int l3_cat_get_old_val(uint64_t val[],
And the length of val is? How can you bound-check the access?
But I *think* this is just a pointer to uint64_t, you can just use
uint64_t *val here and *val = x; in code?
Same comment applies to the set_new_val handler as well.
> + const struct feat_node *feat,
> + unsigned int old_cos)
> +{
> + if ( old_cos > feat->info.l3_cat_info.cos_max )
> + /* Use default value. */
> + old_cos = 0;
> +
> + /* CAT */
> + val[0] = feat->cos_reg_val[old_cos];
> +
> + return 0;
> +}
> +
> +static int l3_cat_set_new_val(uint64_t val[],
> + const struct feat_node *feat,
> + enum cbm_type type,
> + uint64_t m)
> +{
> + if ( !psr_check_cbm(feat->info.l3_cat_info.cbm_len, m) )
> + return -EINVAL;
> +
> + val[0] = m;
> +
> + return 0;
> +}
> +
> static const struct feat_ops l3_cat_ops = {
> .get_cos_max = l3_cat_get_cos_max,
> .get_feat_info = l3_cat_get_feat_info,
> .get_val = l3_cat_get_val,
> + .get_cos_num = l3_cat_get_cos_num,
> + .get_old_val = l3_cat_get_old_val,
> + .set_new_val = l3_cat_set_new_val,
> };
>
> static void __init parse_psr_bool(char *s, char *value, char *feature,
> @@ -549,15 +633,42 @@ int psr_get_val(struct domain *d, unsigned int socket,
> /* Set value functions */
> static unsigned int get_cos_num(const struct psr_socket_info *info)
> {
> - return 0;
> + const struct feat_node *feat_tmp;
> + unsigned int num = 0;
> +
> + /* Get all features total amount. */
> + list_for_each_entry(feat_tmp, &info->feat_list, list)
> + num += feat_tmp->ops.get_cos_num(feat_tmp);
> +
> + return num;
> }
>
> -static int assemble_val_array(uint64_t *val,
> +static int combine_val_array(uint64_t *val,
> uint32_t array_len,
> const struct psr_socket_info *info,
> unsigned int old_cos)
Please just name this function combine_val_array in your previous patch
instead of trying to rename it here. Or just don't change the name at
all -- I don't see why changing name is necessary.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow.
2017-02-15 8:49 ` [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
@ 2017-02-26 17:43 ` Wei Liu
2017-02-27 7:16 ` Yi Sun
2017-03-08 17:03 ` Jan Beulich
1 sibling, 1 reply; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:43 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:25PM +0800, Yi Sun wrote:
> Continue with patch:
> 'x86: refactor psr: set value: assemble features value array'
>
> We can try to find if there is a COS ID on which all features' COS registers
> values are same as the array assembled before.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> xen/arch/x86/psr.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 93 insertions(+)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index ae108b9..7fab28b 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -145,6 +145,19 @@ struct feat_ops {
> const struct feat_node *feat,
> enum cbm_type type,
> uint64_t m);
> + /*
> + * compare_val is used in set value process to compare if the
> + * input value array can match all the features' COS registers values
> + * according to input cos id.
> + *
> + * The return value is the amount of entries to skip in the value array
> + * or error.
> + * 1 - one entry in value array.
> + * 2 - two entries in value array, e.g. CDP uses two entries.
> + * negative - error.
This doesn't match the code (yet?).
What about return value 0?
> + */
> + int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
> + unsigned int cos, bool *found);
Indentation.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow.
2017-02-15 8:49 ` [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
@ 2017-02-26 17:43 ` Wei Liu
2017-03-09 14:10 ` Jan Beulich
1 sibling, 0 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:43 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:26PM +0800, Yi Sun wrote:
> static int pick_avail_cos(const struct psr_socket_info *info,
> const uint64_t *val, uint32_t array_len,
> unsigned int old_cos,
> enum psr_feat_type feat_type)
> {
> + unsigned int cos;
> + unsigned int cos_max = 0;
> + const struct feat_node *feat;
> + const unsigned int *ref = info->cos_ref;
> +
> + /*
> + * cos_max is the one of the feature which is being set.
> + */
Single line comment should be like:
/* xxxx */
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 14/24] x86: refactor psr: implement get hw info flow for CDP.
2017-02-15 8:49 ` [PATCH v8 14/24] x86: refactor psr: implement get hw info " Yi Sun
@ 2017-02-26 17:43 ` Wei Liu
2017-02-28 14:54 ` Roger Pau Monné
1 sibling, 0 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:43 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:29PM +0800, Yi Sun wrote:
[...]
>
> static void __init parse_psr_bool(char *s, char *value, char *feature,
> diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
> index e340baa..568bfe9 100644
> --- a/xen/arch/x86/sysctl.c
> +++ b/xen/arch/x86/sysctl.c
> @@ -181,9 +181,27 @@ long arch_do_sysctl(
> ret = psr_get_info(sysctl->u.psr_cat_op.target,
> PSR_CBM_TYPE_L3, data, 3);
>
> - sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
> - sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
> - sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
> + if ( !ret )
> + {
> + sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
> + sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
> + sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
> + } else {
> + /*
> + * Check if CDP is enabled.
> + *
> + * Per spec, L3 CAT and CDP cannot co-exist. So, we need replace
> + * output values to CDP's if it is enabled.
> + */
> + ret = psr_get_info(sysctl->u.psr_cat_op.target,
> + PSR_CBM_TYPE_L3_CODE, data, 3);
Indentation.
> + if ( !ret )
> + {
> + sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
> + sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
> + sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
> + }
> + }
>
> if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
> ret = -EFAULT;
> --
> 1.9.1
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 16/24] x86: refactor psr: implement set value callback functions for CDP.
2017-02-15 8:49 ` [PATCH v8 16/24] x86: refactor psr: implement set value callback functions " Yi Sun
@ 2017-02-26 17:43 ` Wei Liu
2017-02-27 7:19 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Wei Liu @ 2017-02-26 17:43 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:31PM +0800, Yi Sun wrote:
> This patch implements L3 CDP set value related callback functions.
>
> With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> v8:
> - modify 'l3_cdp_write_msr' to 'void'.
> ---
> xen/arch/x86/psr.c | 118 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 118 insertions(+)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 72ed923..0a0bab9 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -560,10 +560,128 @@ static bool l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> return true;
> }
>
> +static unsigned int l3_cdp_get_cos_num(const struct feat_node *feat)
> +{
> + return 2;
Please avoid using magic number, or document where 2 comes from.
> +}
> +
> +static int l3_cdp_get_old_val(uint64_t val[],
Hmm... so val is indeed an array. I think we need to pass the length as
well.
> + const struct feat_node *feat,
> + unsigned int old_cos)
> +{
> + if ( old_cos > feat->info.l3_cdp_info.cos_max )
> + /* Use default value. */
> + old_cos = 0;
> +
> + /* Data */
> + val[0] = get_cdp_data(feat, old_cos);
> + /* Code */
> + val[1] = get_cdp_code(feat, old_cos);
> +
> + return 0;
> +}
> +
> +static int l3_cdp_set_new_val(uint64_t val[],
> + const struct feat_node *feat,
> + enum cbm_type type,
> + uint64_t m)
> +{
> + if ( !psr_check_cbm(feat->info.l3_cdp_info.cbm_len, m) )
> + return -EINVAL;
> +
> + if ( type == PSR_CBM_TYPE_L3_DATA )
> + val[0] = m;
> + else
> + val[1] = m;
> +
> + return 0;
> +}
> +
> +static int l3_cdp_compare_val(const uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int cos, bool *found)
> +{
> + uint64_t l3_def_cbm;
> +
> + l3_def_cbm = (1ull << feat->info.l3_cdp_info.cbm_len) - 1;
This is getting repetitive. Please consider providing a macro to
calculate this value.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (24 preceding siblings ...)
2017-02-15 16:14 ` [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Konrad Rzeszutek Wilk
@ 2017-02-26 18:00 ` Wei Liu
2017-02-28 11:02 ` Roger Pau Monné
26 siblings, 0 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-26 18:00 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
Hi Yi
I went through this series and made some comments -- since I needed to
answer your questions about behaviour of toolstack.
I'm not a x86 maintainer so my suggestions could be wrong. Please wait
until x86 maintainers review your series before resending.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-26 17:41 ` Wei Liu
@ 2017-02-27 6:42 ` Yi Sun
2017-02-27 11:45 ` Wei Liu
2017-02-27 8:41 ` Jan Beulich
1 sibling, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-02-27 6:42 UTC (permalink / raw)
To: Wei Liu
Cc: kevin.tian, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-26 17:41:08, Wei Liu wrote:
> On Wed, Feb 15, 2017 at 04:49:19PM +0800, Yi Sun wrote:
> > This patch implements the CPU init and free flow including L3 CAT
> > initialization and feature list free.
> >
> > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
>
> Either you need to use a separate patch to move cpuid_count_leaf or you
> should state it is moved in the commit message.
>
Thanks! I will add description in commit message.
> > +
> > +/* Common functions. */
> > +static void free_feature(struct psr_socket_info *info)
> > +{
> > + struct feat_node *feat, *next;
> > +
> > + if ( !info )
> > + return;
> > +
> > + /*
> > + * Free resources of features. But we do not free global feature list
> > + * entry, like feat_l3_cat. Although it may cause a few memory leak,
> > + * it is OK simplify things.
>
> I don't think it is OK to leak memory in the hypervisor in general.
> Please specify why it is OK in this particular case in the comment.
>
In most cases, such global feature list entry will be added into feature list
so that it can be freed here.
In extreme case, e.g. socket 1 does not support L3 CAT. The feat_l3_cat
allocated in psr_cpu_prepare will not be released. But this is rare case.
Jan, Konrad and me disucssed this before. Per Jan's suggestion, we do not free
it.
> > + */
> > + list_for_each_entry_safe(feat, next, &info->feat_list, list)
> > + {
> > + if ( !feat )
> > + return;
> > +
> > + __clear_bit(feat->feature, &info->feat_mask);
> > + list_del(&feat->list);
> > + xfree(feat);
> > + }
> > +}
> > +
> > -static int psr_cpu_prepare(unsigned int cpu)
> > +static void cpu_init_work(void)
> > +{
> > + struct psr_socket_info *info;
> > + unsigned int socket;
> > + unsigned int cpu = smp_processor_id();
> > + struct feat_node *feat;
> > + struct cpuid_leaf regs = { .a = 0, .b = 0, .c = 0, .d = 0 };
> > +
> > + if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
> > + return;
> > + else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
> > + {
> > + __clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
> > + return;
> > + }
> > +
> > + socket = cpu_to_socket(cpu);
> > + info = socket_info + socket;
> > + if ( info->feat_mask )
> > + return;
> > +
> > + INIT_LIST_HEAD(&info->feat_list);
> > + spin_lock_init(&info->ref_lock);
> > +
> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> > + if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> > + {
>
> You can move
>
> struct feat_node *feat
>
> here.
>
This variable will also be used by L2 CAT which codes exist in a different
branch. So, I declare it at the top of the function. Please refer below
patch:
[PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow.
> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> > +
> > + feat = feat_l3_cat;
> > + /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
> > + feat_l3_cat = NULL;
> > + feat->ops = l3_cat_ops;
> > +
> > + l3_cat_init_feature(regs, feat, info);
> > + }
> > +}
> > +
> [...]
> >
> > @@ -359,7 +528,7 @@ static int cpu_callback(
> > switch ( action )
> > {
> > case CPU_UP_PREPARE:
> > - rc = psr_cpu_prepare(cpu);
> > + rc = psr_cpu_prepare();
> > break;
> > case CPU_STARTING:
> > psr_cpu_init();
> > @@ -388,10 +557,14 @@ static int __init psr_presmp_init(void)
> > if ( (opt_psr & PSR_CMT) && opt_rmid_max )
> > init_psr_cmt(opt_rmid_max);
> >
> > - psr_cpu_prepare(0);
> > + if ( opt_psr & PSR_CAT )
> > + init_psr();
> > +
> > + if ( psr_cpu_prepare() )
> > + psr_free();
> >
> > psr_cpu_init();
> > - if ( psr_cmt_enabled() )
> > + if ( psr_cmt_enabled() || socket_info )
>
> Why not have psr_cat_enabled() here?
>
psr_cmt_enabled() returns true of false by checking if the global pointer
'psr_cmt' has been allocated or not. The 'psr_cmt' is also used in sysctl.c.
For allocation features, we have a similar global pointer 'socket_info'. But
it is only used in psr.c and all allocation features(CAT/CDP/MBA) use it. So
we directly use it in psr.c to check if related initialization has been done.
> Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-02-26 17:41 ` Wei Liu
@ 2017-02-27 7:06 ` Yi Sun
2017-02-27 10:55 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-02-27 7:06 UTC (permalink / raw)
To: Wei Liu
Cc: kevin.tian, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-26 17:41:43, Wei Liu wrote:
> On Wed, Feb 15, 2017 at 04:49:23PM +0800, Yi Sun wrote:
> [...]
> > +int psr_set_val(struct domain *d, unsigned int socket,
> > + uint64_t val, enum cbm_type type)
>
> IMHO it would be far better to use goto style error handling in such a
> complex function. You can avoid missing one of the exit paths when
> refactoring this function later.
>
Ok, I will consider to use goto for error handling.
> > +{
> > + unsigned int old_cos;
> > + int cos, ret;
> > + unsigned int *ref;
> > + uint64_t *val_array;
> > + struct psr_socket_info *info = get_socket_info(socket);
> > + uint32_t array_len;
> > + enum psr_feat_type feat_type;
> > +
> > + if ( IS_ERR(info) )
> > + return PTR_ERR(info);
> > +
> > + feat_type = psr_cbm_type_to_feat_type(type);
> > + if ( !test_bit(feat_type, &info->feat_mask) )
> > + return -ENOENT;
> > +
> > + /*
> > + * Step 0:
> > + * old_cos means the COS ID current domain is using. By default, it is 0.
> > + *
> > + * For every COS ID, there is a reference count to record how many domains
> > + * are using the COS register corresponding to this COS ID.
> > + * - If ref[old_cos] is 0, that means this COS is not used by any domain.
> > + * - If ref[old_cos] is 1, that means this COS is only used by current
> > + * domain.
> > + * - If ref[old_cos] is more than 1, that mean multiple domains are using
> > + * this COS.
> > + */
> > + old_cos = d->arch.psr_cos_ids[socket];
> > + if ( old_cos > MAX_COS_REG_CNT )
>
> How could this happen? This function is the setter of cos, it is a bug
> if it ever sets a value larger than MAX_COS_REG_CNT.
>
You are right. This should not happen. This check is just a protection. If you
think it is not necessary, I will remove it.
> > + return -EOVERFLOW;
> > +
> > + ref = info->cos_ref;
> > +
> > + /*
> > + * Step 1:
> > + * Assemle a value array to store all featues cos_reg_val[old_cos].
> > + * And, set the input val into array according to the feature's
> > + * position in array.
> > + */
> > + array_len = get_cos_num(info);
> > + val_array = xzalloc_array(uint64_t, array_len);
> > + if ( !val_array )
> > + return -ENOMEM;
> > +
> > + if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 )
> > + {
> > + xfree(val_array);
> > + return ret;
> > + }
> > +
> > + if ( (ret = set_new_val_to_array(val_array, array_len, info,
> > + feat_type, type, val)) != 0 )
> > + {
> > + xfree(val_array);
> > + return ret;
> > + }
> > +
> > + /*
> > + * Lock here to make sure the ref is not changed during find and
> > + * write process.
> > + */
> > + spin_lock(&info->ref_lock);
> > +
> > + /*
> > + * Step 2:
> > + * Try to find if there is already a COS ID on which all features' values
> > + * are same as the array. Then, we can reuse this COS ID.
> > + */
> > + cos = find_cos(val_array, array_len, feat_type, info);
> > + if ( cos >= 0 )
> > + {
> > + if ( cos == old_cos )
> > + {
> > + spin_unlock(&info->ref_lock);
> > + xfree(val_array);
> > + return 0;
> > + }
> > + }
> > + else
> > + {
> > + /*
> > + * Step 3:
> > + * If fail to find, we need allocate a new COS ID.
> > + * If multiple domains are using same COS ID, its ref is more
> > + * than 1. That means we cannot free this COS to make current domain
> > + * use it. Because other domains are using the value saved in the COS.
> > + * Unless the ref is changed to 1 (mean only current domain is using
> > + * it), we cannot allocate the COS ID to current domain.
> > + * So, only the COS ID which ref is 1 or 0 can be allocated.
> > + */
> > + cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
> > + if ( cos < 0 )
> > + {
> > + spin_unlock(&info->ref_lock);
> > + xfree(val_array);
> > + return cos;
> > + }
> > +
> > + /*
> > + * Step 4:
> > + * Write all features MSRs according to the COS ID.
> > + */
> > + ret = write_psr_msr(socket, cos, val_array);
> > + if ( ret )
> > + {
> > + spin_unlock(&info->ref_lock);
> > + xfree(val_array);
> > + return ret;
> > + }
> > + }
> > +
> > + /*
> > + * Step 5:
> > + * Update ref according to COS ID.
> > + */
> > + ref[cos]++;
> > + ASSERT(ref[cos] || cos == 0);
> > + ref[old_cos]--;
> > + spin_unlock(&info->ref_lock);
> > +
> > + /*
> > + * Step 6:
> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> > + * which COS the domain is using on the socket. One domain can only use
> > + * one COS ID at same time on each socket.
> > + */
> > + d->arch.psr_cos_ids[socket] = cos;
> > + xfree(val_array);
> > +
> > + return 0;
> > +}
> > +
> > /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> > static void psr_free_cos(struct domain *d)
> > {
> > - if( !d->arch.psr_cos_ids )
> > + unsigned int socket, cos;
> > +
> > + if ( !d->arch.psr_cos_ids )
> > return;
> >
> > + /* Domain is free so its cos_ref should be decreased. */
> > + for ( socket = 0; socket < nr_sockets; socket++ )
> > + {
> > + struct psr_socket_info *info;
> > +
> > + /* cos 0 is default one which does not need be handled. */
> > + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
>
> Break this into two lines. The assignment doesn't have to happen within
> if().
>
Ok, thanks!
> Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-02-26 17:43 ` Wei Liu
@ 2017-02-27 7:11 ` Yi Sun
2017-02-27 11:45 ` Wei Liu
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-02-27 7:11 UTC (permalink / raw)
To: Wei Liu
Cc: kevin.tian, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-26 17:43:04, Wei Liu wrote:
> On Wed, Feb 15, 2017 at 04:49:24PM +0800, Yi Sun wrote:
> [...]
> >
> > +static unsigned int l3_cat_get_cos_num(const struct feat_node *feat)
> > +{
> > + return 1;
> > +}
> > +
> > +static int l3_cat_get_old_val(uint64_t val[],
>
> And the length of val is? How can you bound-check the access?
>
> But I *think* this is just a pointer to uint64_t, you can just use
> uint64_t *val here and *val = x; in code?
>
> Same comment applies to the set_new_val handler as well.
>
Such implementation is to simplify these functions. The bound-check will be
done in caller functions, such as combine_val_array/set_new_val_to_array/etc.
> > + const struct feat_node *feat,
> > + unsigned int old_cos)
> > +{
> > + if ( old_cos > feat->info.l3_cat_info.cos_max )
> > + /* Use default value. */
> > + old_cos = 0;
> > +
> > + /* CAT */
> > + val[0] = feat->cos_reg_val[old_cos];
> > +
> > + return 0;
> > +}
> > +
[...]
> > -static int assemble_val_array(uint64_t *val,
> > +static int combine_val_array(uint64_t *val,
> > uint32_t array_len,
> > const struct psr_socket_info *info,
> > unsigned int old_cos)
>
> Please just name this function combine_val_array in your previous patch
> instead of trying to rename it here. Or just don't change the name at
> all -- I don't see why changing name is necessary.
>
Per Konrad's suggestion to change the name. Because he thought the 'assemble'
is not accurate here.
> Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow.
2017-02-26 17:43 ` Wei Liu
@ 2017-02-27 7:16 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-27 7:16 UTC (permalink / raw)
To: Wei Liu
Cc: kevin.tian, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-26 17:43:20, Wei Liu wrote:
> On Wed, Feb 15, 2017 at 04:49:25PM +0800, Yi Sun wrote:
> > Continue with patch:
> > 'x86: refactor psr: set value: assemble features value array'
> >
> > We can try to find if there is a COS ID on which all features' COS registers
> > values are same as the array assembled before.
> >
> > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> > ---
> > xen/arch/x86/psr.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 93 insertions(+)
> >
> > diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> > index ae108b9..7fab28b 100644
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -145,6 +145,19 @@ struct feat_ops {
> > const struct feat_node *feat,
> > enum cbm_type type,
> > uint64_t m);
> > + /*
> > + * compare_val is used in set value process to compare if the
> > + * input value array can match all the features' COS registers values
> > + * according to input cos id.
> > + *
> > + * The return value is the amount of entries to skip in the value array
> > + * or error.
> > + * 1 - one entry in value array.
> > + * 2 - two entries in value array, e.g. CDP uses two entries.
> > + * negative - error.
>
> This doesn't match the code (yet?).
>
> What about return value 0?
>
Oh, sorry, per Jan's suggestion, I have change the type of the function. But I
forgot the modify the comments. Will fix it.
> > + */
> > + int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
> > + unsigned int cos, bool *found);
>
> Indentation.
>
Thanks!
>
> Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 16/24] x86: refactor psr: implement set value callback functions for CDP.
2017-02-26 17:43 ` Wei Liu
@ 2017-02-27 7:19 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-02-27 7:19 UTC (permalink / raw)
To: Wei Liu
Cc: kevin.tian, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-26 17:43:55, Wei Liu wrote:
> On Wed, Feb 15, 2017 at 04:49:31PM +0800, Yi Sun wrote:
> > This patch implements L3 CDP set value related callback functions.
> >
> > With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.
> >
> > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> > ---
> > v8:
> > - modify 'l3_cdp_write_msr' to 'void'.
> > ---
> > xen/arch/x86/psr.c | 118 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 118 insertions(+)
> >
> > diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> > index 72ed923..0a0bab9 100644
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -560,10 +560,128 @@ static bool l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> > return true;
> > }
> >
> > +static unsigned int l3_cdp_get_cos_num(const struct feat_node *feat)
> > +{
> > + return 2;
>
> Please avoid using magic number, or document where 2 comes from.
>
Will add comment to explain this.
> > +}
> > +
> > +static int l3_cdp_get_old_val(uint64_t val[],
>
> Hmm... so val is indeed an array. I think we need to pass the length as
> well.
>
I explained it in previous match to address your comment. The array length
is checked in caller function. Is that acceptable to you?
> > + const struct feat_node *feat,
> > + unsigned int old_cos)
> > +{
> > + if ( old_cos > feat->info.l3_cdp_info.cos_max )
> > + /* Use default value. */
> > + old_cos = 0;
> > +
> > + /* Data */
> > + val[0] = get_cdp_data(feat, old_cos);
> > + /* Code */
> > + val[1] = get_cdp_code(feat, old_cos);
> > +
> > + return 0;
> > +}
> > +
> > +static int l3_cdp_set_new_val(uint64_t val[],
> > + const struct feat_node *feat,
> > + enum cbm_type type,
> > + uint64_t m)
> > +{
> > + if ( !psr_check_cbm(feat->info.l3_cdp_info.cbm_len, m) )
> > + return -EINVAL;
> > +
> > + if ( type == PSR_CBM_TYPE_L3_DATA )
> > + val[0] = m;
> > + else
> > + val[1] = m;
> > +
> > + return 0;
> > +}
> > +
> > +static int l3_cdp_compare_val(const uint64_t val[],
> > + const struct feat_node *feat,
> > + unsigned int cos, bool *found)
> > +{
> > + uint64_t l3_def_cbm;
> > +
> > + l3_def_cbm = (1ull << feat->info.l3_cdp_info.cbm_len) - 1;
>
> This is getting repetitive. Please consider providing a macro to
> calculate this value.
>
Ok, thanks!
> Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-26 17:41 ` Wei Liu
2017-02-27 6:42 ` Yi Sun
@ 2017-02-27 8:41 ` Jan Beulich
1 sibling, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-02-27 8:41 UTC (permalink / raw)
To: wei.liu2, yi.y.sun
Cc: kevin.tian, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
mengxu, xen-devel, chao.p.peng
>>> Wei Liu <wei.liu2@citrix.com> 02/26/17 6:41 PM >>>
>On Wed, Feb 15, 2017 at 04:49:19PM +0800, Yi Sun wrote:
>> +static void free_feature(struct psr_socket_info *info)
>> +{
>> + struct feat_node *feat, *next;
>> +
>> + if ( !info )
>> + return;
>> +
>> + /*
>> + * Free resources of features. But we do not free global feature list
>> + * entry, like feat_l3_cat. Although it may cause a few memory leak,
>> + * it is OK simplify things.
>
>I don't think it is OK to leak memory in the hypervisor in general.
>Please specify why it is OK in this particular case in the comment.
The problem is to call this a leak in the comment. There's no leak here,
the allocation is simply being kept until the next CPU online attempt.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-02-27 7:06 ` Yi Sun
@ 2017-02-27 10:55 ` Jan Beulich
0 siblings, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-02-27 10:55 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> Yi Sun <yi.y.sun@linux.intel.com> 02/27/17 8:06 AM >>>
>On 17-02-26 17:41:43, Wei Liu wrote:
>> On Wed, Feb 15, 2017 at 04:49:23PM +0800, Yi Sun wrote:
>> > + /*
>> > + * Step 0:
>> > + * old_cos means the COS ID current domain is using. By default, it is 0.
>> > + *
>> > + * For every COS ID, there is a reference count to record how many domains
>> > + * are using the COS register corresponding to this COS ID.
>> > + * - If ref[old_cos] is 0, that means this COS is not used by any domain.
>> > + * - If ref[old_cos] is 1, that means this COS is only used by current
>> > + * domain.
>> > + * - If ref[old_cos] is more than 1, that mean multiple domains are using
>> > + * this COS.
>> > + */
>> > + old_cos = d->arch.psr_cos_ids[socket];
>> > + if ( old_cos > MAX_COS_REG_CNT )
>>
>> How could this happen? This function is the setter of cos, it is a bug
>> if it ever sets a value larger than MAX_COS_REG_CNT.
>>
>You are right. This should not happen. This check is just a protection. If you
>think it is not necessary, I will remove it.
In which case - make in an ASSERT() instead of an if().
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-27 6:42 ` Yi Sun
@ 2017-02-27 11:45 ` Wei Liu
0 siblings, 0 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-27 11:45 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, Wei Liu, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Mon, Feb 27, 2017 at 02:42:31PM +0800, Yi Sun wrote:
> On 17-02-26 17:41:08, Wei Liu wrote:
> > On Wed, Feb 15, 2017 at 04:49:19PM +0800, Yi Sun wrote:
> > > This patch implements the CPU init and free flow including L3 CAT
> > > initialization and feature list free.
> > >
> > > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> >
> > Either you need to use a separate patch to move cpuid_count_leaf or you
> > should state it is moved in the commit message.
> >
> Thanks! I will add description in commit message.
>
> > > +
> > > +/* Common functions. */
> > > +static void free_feature(struct psr_socket_info *info)
> > > +{
> > > + struct feat_node *feat, *next;
> > > +
> > > + if ( !info )
> > > + return;
> > > +
> > > + /*
> > > + * Free resources of features. But we do not free global feature list
> > > + * entry, like feat_l3_cat. Although it may cause a few memory leak,
> > > + * it is OK simplify things.
> >
> > I don't think it is OK to leak memory in the hypervisor in general.
> > Please specify why it is OK in this particular case in the comment.
> >
> In most cases, such global feature list entry will be added into feature list
> so that it can be freed here.
>
> In extreme case, e.g. socket 1 does not support L3 CAT. The feat_l3_cat
> allocated in psr_cpu_prepare will not be released. But this is rare case.
>
> Jan, Konrad and me disucssed this before. Per Jan's suggestion, we do not free
> it.
Then I would suggest you to not use "leak" in the comment. And put the
relevant bits from the discussion in the comment. Otherwise a drive-by
reviewer like me will call this out again. :-)
>
> > > + */
> > > + list_for_each_entry_safe(feat, next, &info->feat_list, list)
> > > + {
> > > + if ( !feat )
> > > + return;
> > > +
> > > + __clear_bit(feat->feature, &info->feat_mask);
> > > + list_del(&feat->list);
> > > + xfree(feat);
> > > + }
> > > +}
> > > +
> > > -static int psr_cpu_prepare(unsigned int cpu)
> > > +static void cpu_init_work(void)
> > > +{
> > > + struct psr_socket_info *info;
> > > + unsigned int socket;
> > > + unsigned int cpu = smp_processor_id();
> > > + struct feat_node *feat;
> > > + struct cpuid_leaf regs = { .a = 0, .b = 0, .c = 0, .d = 0 };
> > > +
> > > + if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
> > > + return;
> > > + else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
> > > + {
> > > + __clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
> > > + return;
> > > + }
> > > +
> > > + socket = cpu_to_socket(cpu);
> > > + info = socket_info + socket;
> > > + if ( info->feat_mask )
> > > + return;
> > > +
> > > + INIT_LIST_HEAD(&info->feat_list);
> > > + spin_lock_init(&info->ref_lock);
> > > +
> > > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> > > + if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> > > + {
> >
> > You can move
> >
> > struct feat_node *feat
> >
> > here.
> >
> This variable will also be used by L2 CAT which codes exist in a different
> branch. So, I declare it at the top of the function. Please refer below
> patch:
> [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow.
OK.
>
> > > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> > > +
> > > + feat = feat_l3_cat;
> > > + /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
> > > + feat_l3_cat = NULL;
> > > + feat->ops = l3_cat_ops;
> > > +
> > > + l3_cat_init_feature(regs, feat, info);
> > > + }
> > > +}
> > > +
> > [...]
> > >
> > > @@ -359,7 +528,7 @@ static int cpu_callback(
> > > switch ( action )
> > > {
> > > case CPU_UP_PREPARE:
> > > - rc = psr_cpu_prepare(cpu);
> > > + rc = psr_cpu_prepare();
> > > break;
> > > case CPU_STARTING:
> > > psr_cpu_init();
> > > @@ -388,10 +557,14 @@ static int __init psr_presmp_init(void)
> > > if ( (opt_psr & PSR_CMT) && opt_rmid_max )
> > > init_psr_cmt(opt_rmid_max);
> > >
> > > - psr_cpu_prepare(0);
> > > + if ( opt_psr & PSR_CAT )
> > > + init_psr();
> > > +
> > > + if ( psr_cpu_prepare() )
> > > + psr_free();
> > >
> > > psr_cpu_init();
> > > - if ( psr_cmt_enabled() )
> > > + if ( psr_cmt_enabled() || socket_info )
> >
> > Why not have psr_cat_enabled() here?
> >
> psr_cmt_enabled() returns true of false by checking if the global pointer
> 'psr_cmt' has been allocated or not. The 'psr_cmt' is also used in sysctl.c.
> For allocation features, we have a similar global pointer 'socket_info'. But
> it is only used in psr.c and all allocation features(CAT/CDP/MBA) use it. So
> we directly use it in psr.c to check if related initialization has been done.
The problem with using socket_info directly is that the name doesn't
tell you much. It doesn't carry specific semantics by itself. Wrapping
it inside an inline function with a proper name is much nicer. Also
there is the possibility that in the future you change the code to use
socket_info for a slightly different purpose or you want to expose it
outside of psr.c, you then need to retrospectively inspect all sites to
make sure you don't screw things up. IMHO using a psr_XXX_enabled like
psr_cmt_enabled is a small change with big benefit.
This is just a general suggestion. I don't feel too strongly about this.
If the maintainers are happy with the code as-is, you don't need to
change.
Wei.
>
> > Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-02-27 7:11 ` Yi Sun
@ 2017-02-27 11:45 ` Wei Liu
0 siblings, 0 replies; 122+ messages in thread
From: Wei Liu @ 2017-02-27 11:45 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, Wei Liu, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Mon, Feb 27, 2017 at 03:11:35PM +0800, Yi Sun wrote:
> On 17-02-26 17:43:04, Wei Liu wrote:
> > On Wed, Feb 15, 2017 at 04:49:24PM +0800, Yi Sun wrote:
> > [...]
> > >
> > > +static unsigned int l3_cat_get_cos_num(const struct feat_node *feat)
> > > +{
> > > + return 1;
> > > +}
> > > +
> > > +static int l3_cat_get_old_val(uint64_t val[],
> >
> > And the length of val is? How can you bound-check the access?
> >
> > But I *think* this is just a pointer to uint64_t, you can just use
> > uint64_t *val here and *val = x; in code?
> >
> > Same comment applies to the set_new_val handler as well.
> >
> Such implementation is to simplify these functions. The bound-check will be
> done in caller functions, such as combine_val_array/set_new_val_to_array/etc.
>
Please consider adding a comment to say bound-check is don't by the
caller.
> > > + const struct feat_node *feat,
> > > + unsigned int old_cos)
> > > +{
> > > + if ( old_cos > feat->info.l3_cat_info.cos_max )
> > > + /* Use default value. */
> > > + old_cos = 0;
> > > +
> > > + /* CAT */
> > > + val[0] = feat->cos_reg_val[old_cos];
> > > +
> > > + return 0;
> > > +}
> > > +
> [...]
> > > -static int assemble_val_array(uint64_t *val,
> > > +static int combine_val_array(uint64_t *val,
> > > uint32_t array_len,
> > > const struct psr_socket_info *info,
> > > unsigned int old_cos)
> >
> > Please just name this function combine_val_array in your previous patch
> > instead of trying to rename it here. Or just don't change the name at
> > all -- I don't see why changing name is necessary.
> >
> Per Konrad's suggestion to change the name. Because he thought the 'assemble'
> is not accurate here.
>
The such change should be inside the patch to introduce the framework,
not in this patch.
Wei.
> > Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (25 preceding siblings ...)
2017-02-26 18:00 ` Wei Liu
@ 2017-02-28 11:02 ` Roger Pau Monné
2017-03-01 4:54 ` Yi Sun
26 siblings, 1 reply; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 11:02 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:15PM +0800, Yi Sun wrote:
> Hi all,
>
> We plan to bring a new PSR (Platform Shared Resource) feature called
> Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
>
> Besides the L2 CAT implementaion, we refactor the psr.c to make it more
> flexible to add new features and fulfill the principle, open for extension
> but closed for modification. We abstract the general operations of all
Hello,
I see that you use the term "open for extension but closed for modification"
here and in order patches. Could you please clarify what this means? "closed
for modification" doesn't seem to make much sense for an open source project,
where all the code is always open for modification.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 03/24] x86: refactor psr: implement main data structures.
2017-02-15 8:49 ` [PATCH v8 03/24] x86: refactor psr: implement main data structures Yi Sun
@ 2017-02-28 11:58 ` Roger Pau Monné
2017-03-01 5:10 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 11:58 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:18PM +0800, Yi Sun wrote:
> To construct an extendible framework, we need analyze PSR features
> and abstract the common things and feature specific things. Then,
> encapsulate them into different data structures.
>
> By analyzing PSR features, we can get below map.
> +------+------+------+
> --------->| Dom0 | Dom1 | ... |
> | +------+------+------+
> | |
> |Dom ID | cos_id of domain
> | V
> | +-----------------------------------------------------------------------------+
> User --------->| PSR |
> Socket ID | +--------------+---------------+---------------+ |
> | | Socket0 Info | Socket 1 Info | ... | |
> | +--------------+---------------+---------------+ |
> | | cos_id=0 cos_id=1 ... |
> | | +-----------------------+-----------------------+-----------+ |
> | |->Ref : | ref 0 | ref 1 | ... | |
> | | +-----------------------+-----------------------+-----------+ |
> | | +-----------------------+-----------------------+-----------+ |
> | |->L3 CAT: | cos 0 | cos 1 | ... | |
> | | +-----------------------+-----------------------+-----------+ |
> | | +-----------------------+-----------------------+-----------+ |
> | |->L2 CAT: | cos 0 | cos 1 | ... | |
> | | +-----------------------+-----------------------+-----------+ |
> | | +-----------+-----------+-----------+-----------+-----------+ |
> | |->CDP : | cos0 code | cos0 data | cos1 code | cos1 data | ... | |
> | +-----------+-----------+-----------+-----------+-----------+ |
> +-----------------------------------------------------------------------------+
>
> So, we need define a socket info data structure, 'struct
> psr_socket_info' to manage information per socket. It contains a
> reference count array according to COS ID and a feature list to
> manage all features enabled. Every entry of the reference count
> array is used to record how many domains are using the COS registers
> according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
> Dom1 uses COS_ID=1 registers of both features to save CBM values, like
> below.
> +-------+-------+-------+-----+
> | COS 0 | COS 1 | COS 2 | ... |
> +-------+-------+-------+-----+
> L3 CAT | 0x7ff | 0x1ff | ... | ... |
> +-------+-------+-------+-----+
> L2 CAT | 0xff | 0xff | ... | ... |
> +-------+-------+-------+-----+
>
> If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
> That means, both Dom1 and Dom2 use same COS registers(ID=1) to save same
> L3/L2 values. So, the value ref[1] is 2 which means 2 domains are using
> COS_ID 1.
>
> To manage a feature, we need define a feature node data structure,
> 'struct feat_node', to manage feature's specific HW info, its callback
> functions (all feature's specific behaviors are encapsulated into these
> callback functions), and an array of all COS registers values of this
> feature.
>
> CDP is a special feature which uses two entries of the array
> for one COS ID. So, the number of CDP COS registers is the half of L3
> CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
> it is enabled. CDP uses the COS registers array as below.
>
> +-----------+-----------+-----------+-----------+-----------+
> CDP cos_reg_val[] index: | 0 | 1 | 2 | 3 | ... |
> +-----------+-----------+-----------+-----------+-----------+
> value: | cos0 code | cos0 data | cos1 code | cos1 data | ... |
> +-----------+-----------+-----------+-----------+-----------+
>
> For more details, please refer SDM and patches to implement 'get value' and
> 'set value'.
I would recommend that you merge this with a patch that actually makes use of
this structures, or else it's hard to review it's usage IMHO.
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
> xen/arch/x86/psr.c | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 107 insertions(+), 1 deletion(-)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 96a8589..5acd9ca 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -13,16 +13,122 @@
> * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
> * more details.
> */
> -#include <xen/init.h>
> #include <xen/cpu.h>
> #include <xen/err.h>
> +#include <xen/init.h>
> +#include <xen/list.h>
> #include <xen/sched.h>
> #include <asm/psr.h>
>
> +/*
> + * Terminology:
> + * - CAT Cache Allocation Technology
> + * - CBM Capacity BitMasks
> + * - CDP Code and Data Prioritization
> + * - COS/CLOS Class of Service. Also mean COS registers.
> + * - COS_MAX Max number of COS for the feature (minus 1)
> + * - MSRs Machine Specific Registers
> + * - PSR Intel Platform Shared Resource
> + */
> +
> #define PSR_CMT (1<<0)
> #define PSR_CAT (1<<1)
> #define PSR_CDP (1<<2)
>
> +/*
> + * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
> + * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
> + * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
> + *
> + * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for
> + * up to 64 L2 CAT COS. The COS_ID=[0,63].
> + *
> + * So, the maximum COS register count of one feature is 128.
> + */
> +#define MAX_COS_REG_CNT 128
> +
> +/*
> + * PSR features are managed per socket. Below structure defines the members
> + * used to manage these features.
> + * feat_mask - Mask used to record features enabled on socket. There may be
> + * some features enabled at same time.
> + * nr_feat - Record how many features enabled.
> + * feat_list - A list used to manage all features enabled.
> + * cos_ref - A reference count array to record how many domains are using the
> + * COS_ID.
> + * Every entry of cos_ref corresponds to one COS ID.
> + * ref_lock - A lock to protect cos_ref.
> + */
> +struct psr_socket_info {
> + /*
> + * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
> + * psr_feat_type' means the bit position.
> + * bit 0: L3 CAT
> + * bit 1: L3 CDP
> + * bit 2: L2 CAT
> + */
> + unsigned int feat_mask;
> + unsigned int nr_feat;
> + struct list_head feat_list;
Isn't it a little bit overkill to use a list when there can only be a maximum
of 3 features? (and the feat_mask is currently 32bits, so I guess you don't
really foresee having more than 32 features).
I would suggest using:
struct feat_node *features[PSR_SOCKET_MAX_FEAT];
(PSR_SOCKET_MAX_FEAT comes from the expansion of the enum below). Then you can
simply use the enum value of each feature as the position of it's corresponding
feat_node into the array.
> + unsigned int cos_ref[MAX_COS_REG_CNT];
> + spinlock_t ref_lock;
> +};
> +
> +enum psr_feat_type {
> + PSR_SOCKET_L3_CAT = 0,
> + PSR_SOCKET_L3_CDP,
> + PSR_SOCKET_L2_CAT,
PSR_SOCKET_MAX_FEAT,
> +};
> +
> +/* CAT/CDP HW info data structure. */
> +struct psr_cat_hw_info {
> + unsigned int cbm_len;
> + unsigned int cos_max;
> +};
> +
> +/* Encapsulate feature specific HW info here. */
> +struct feat_hw_info {
> + union {
> + struct psr_cat_hw_info l3_cat_info;
> + };
> +};
Why don't you use an union here directly, instead of encapsulating an union
inside of a struct?
union feat_hw_info {
struct psr_cat_hw_info l3_cat_info;
};
> +
> +struct feat_node;
> +
> +/*
> + * This structure defines feature operation callback functions. Every feature
> + * enabled MUST implement such callback functions and register them to ops.
> + *
> + * Feature specific behaviors will be encapsulated into these callback
> + * functions. Then, the main flows will not be changed when introducing a new
> + * feature.
> + */
> +struct feat_ops {
> + /* get_cos_max is used to get feature's cos_max. */
> + unsigned int (*get_cos_max)(const struct feat_node *feat);
> +};
> +
> +/*
> + * This structure represents one feature.
> + * feature - Which feature it is.
> + * feat_ops - Feature operation callback functions.
> + * info - Feature HW info.
> + * cos_reg_val - Array to store the values of COS registers. One entry stores
> + * the value of one COS register.
> + * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
> + * For CDP, two entries correspond to one COS_ID. E.g.
> + * COS_ID=0 corresponds to cos_reg_val[0] (Data) and
> + * cos_reg_val[1] (Code).
> + * list - Feature list.
> + */
> +struct feat_node {
> + enum psr_feat_type feature;
If you index them in an array with the key being psr_feat_type I don't think
you need that field.
> + struct feat_ops ops;
I would place the function hooks in this struct directly, instead of nesting
them inside of another struct. The hooks AFAICT are shared between all the
different PSR features.
> + struct feat_hw_info info;
Same with this, you can place the actual union for storage here directly,
instead of nesting it inside of feat_hw_info, so:
union {
struct psr_cat_hw_info l3_cat_info;
} hw_info;
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 06/24] x86: refactor psr: implement get hw info flow.
2017-02-15 8:49 ` [PATCH v8 06/24] x86: refactor psr: implement get hw info flow Yi Sun
2017-02-26 17:41 ` Wei Liu
@ 2017-02-28 12:34 ` Roger Pau Monné
2017-03-08 15:15 ` Jan Beulich
2 siblings, 0 replies; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 12:34 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:21PM +0800, Yi Sun wrote:
> This patch implements get HW info flow including L3 CAT callback
> function.
>
> It also changes sysctl interface to make it more general.
>
> With this patch, 'psr-hwinfo' can work for L3 CAT.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
> xen/arch/x86/psr.c | 75 +++++++++++++++++++++++++++++++++++++++++++++--
> xen/arch/x86/sysctl.c | 14 +++++----
> xen/include/asm-x86/psr.h | 19 +++++++-----
> 3 files changed, 93 insertions(+), 15 deletions(-)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 798c614..8af59d9 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -84,6 +84,7 @@ enum psr_feat_type {
> PSR_SOCKET_L3_CAT = 0,
> PSR_SOCKET_L3_CDP,
> PSR_SOCKET_L2_CAT,
> + PSR_SOCKET_UNKNOWN = 0xFFFF,
> };
>
> /* CAT/CDP HW info data structure. */
> @@ -112,6 +113,9 @@ struct feat_node;
> struct feat_ops {
> /* get_cos_max is used to get feature's cos_max. */
> unsigned int (*get_cos_max)(const struct feat_node *feat);
> + /* get_feat_info is used to get feature HW info. */
> + bool (*get_feat_info)(const struct feat_node *feat,
> + uint32_t data[], unsigned int array_len);
> };
>
> /*
> @@ -182,6 +186,24 @@ static void free_feature(struct psr_socket_info *info)
> }
> }
>
> +static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> +{
> + enum psr_feat_type feat_type;
> +
> + /* Judge if feature is enabled. */
> + switch ( type )
> + {
> + case PSR_CBM_TYPE_L3:
> + feat_type = PSR_SOCKET_L3_CAT;
> + break;
> + default:
> + feat_type = PSR_SOCKET_UNKNOWN;
> + break;
> + }
> +
> + return feat_type;
> +}
> +
> /* L3 CAT functions implementation. */
> static void l3_cat_init_feature(struct cpuid_leaf regs,
> struct feat_node *feat,
> @@ -225,8 +247,22 @@ static unsigned int l3_cat_get_cos_max(const struct feat_node *feat)
> return feat->info.l3_cat_info.cos_max;
> }
>
> +static bool l3_cat_get_feat_info(const struct feat_node *feat,
> + uint32_t data[], unsigned int array_len)
> +{
> + if ( !data || 3 > array_len )
Shouldn't this be array_len != 3 and then I would rather prefer this to be set
in a define, it's used here and in XEN_SYSCTL_PSR_CAT_get_l3_info, maybe you
should add it's define below of the PSR_FLAG define?
#define CAT_L3_FEAT_SIZE 3
Or some better worded name.
[...]
> int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
> index b8c30d4..e340baa 100644
> --- a/xen/arch/x86/sysctl.c
> +++ b/xen/arch/x86/sysctl.c
> @@ -176,15 +176,19 @@ long arch_do_sysctl(
> switch ( sysctl->u.psr_cat_op.cmd )
> {
> case XEN_SYSCTL_PSR_CAT_get_l3_info:
> - ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
> - &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
> - &sysctl->u.psr_cat_op.u.l3_info.cos_max,
> - &sysctl->u.psr_cat_op.u.l3_info.flags);
> + {
> + uint32_t data[3];
Space between variable declaration and code.
> + ret = psr_get_info(sysctl->u.psr_cat_op.target,
> + PSR_CBM_TYPE_L3, data, 3);
Last parameter should be ARRAY_SIZE(data) IMHO.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 07/24] x86: refactor psr: implement get value flow.
2017-02-15 8:49 ` [PATCH v8 07/24] x86: refactor psr: implement get value flow Yi Sun
@ 2017-02-28 12:44 ` Roger Pau Monné
2017-03-01 5:21 ` Yi Sun
2017-03-08 15:35 ` Jan Beulich
1 sibling, 1 reply; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 12:44 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:22PM +0800, Yi Sun wrote:
> This patch implements get value flow including L3 CAT callback
> function.
>
> It also changes domctl interface to make it more general.
>
> With this patch, 'psr-cat-show' can work for L3 CAT but not for
> L3 code/data which is implemented in patch "x86: refactor psr:
> implement get value flow for CDP.".
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
> xen/arch/x86/domctl.c | 18 +++++++++---------
> xen/arch/x86/psr.c | 43 ++++++++++++++++++++++++++++++++++++++-----
> xen/include/asm-x86/psr.h | 4 ++--
> 3 files changed, 49 insertions(+), 16 deletions(-)
[...]
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 8af59d9..c1afd36 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -116,6 +116,9 @@ struct feat_ops {
> /* get_feat_info is used to get feature HW info. */
> bool (*get_feat_info)(const struct feat_node *feat,
> uint32_t data[], unsigned int array_len);
> + /* get_val is used to get feature COS register value. */
> + bool (*get_val)(const struct feat_node *feat, unsigned int cos,
> + enum cbm_type type, uint64_t *val);
> };
>
> /*
> @@ -260,9 +263,22 @@ static bool l3_cat_get_feat_info(const struct feat_node *feat,
> return true;
> }
>
> +static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
> + enum cbm_type type, uint64_t *val)
> +{
> + if ( cos > feat->info.l3_cat_info.cos_max )
> + /* Use default value. */
> + cos = 0;
I don't know much, but shouldn't this return false instead of silently
defaulting to 0? This doesn't seem to be what the caller expects.
> +
> + *val = feat->cos_reg_val[cos];
> +
> + return true;
> +}
> +
> static const struct feat_ops l3_cat_ops = {
> .get_cos_max = l3_cat_get_cos_max,
> .get_feat_info = l3_cat_get_feat_info,
> + .get_val = l3_cat_get_val,
> };
>
> static void __init parse_psr_bool(char *s, char *value, char *feature,
> @@ -482,12 +498,14 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
> return socket_info + socket;
> }
>
> -int psr_get_info(unsigned int socket, enum cbm_type type,
> - uint32_t data[], unsigned int array_len)
> +static int psr_get(unsigned int socket, enum cbm_type type,
> + uint32_t data[], unsigned int array_len,
> + struct domain *d, uint64_t *val)
> {
> const struct psr_socket_info *info = get_socket_info(socket);
> const struct feat_node *feat;
> enum psr_feat_type feat_type;
> + unsigned int cos;
>
> if ( IS_ERR(info) )
> return PTR_ERR(info);
> @@ -498,6 +516,15 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> if ( feat->feature != feat_type )
> continue;
>
> + if ( d )
> + {
> + cos = d->arch.psr_cos_ids[socket];
> + if ( feat->ops.get_val(feat, cos, type, val) )
> + return 0;
> + else
No need for the "else" branch here.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-02-15 8:49 ` [PATCH v8 08/24] x86: refactor psr: set value: implement framework Yi Sun
2017-02-26 17:41 ` Wei Liu
@ 2017-02-28 13:58 ` Roger Pau Monné
2017-03-01 6:23 ` Yi Sun
2017-03-08 16:07 ` Jan Beulich
2 siblings, 1 reply; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 13:58 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:23PM +0800, Yi Sun wrote:
> As set value flow is the most complicated one in psr, it will be
> divided to some patches to make things clearer. This patch
> implements the set value framework to show a whole picture firstly.
>
> It also changes domctl interface to make it more general.
>
> To make the set value flow be general and can support multiple features
> at same time, it includes below steps:
> 1. Get COS ID of current domain using.
> 2. Assemble a value array to store all features current value
> in it and replace the current value of the feature which is
> being set to the new input value.
> 3. Find if there is already a COS ID on which all features'
> values are same as the array. Then, we can reuse this COS
> ID.
> 4. If fail to find, we need pick an available COS ID. Only COS ID which ref
> is 0 or 1 can be picked.
> 5. Write all features MSRs according to the COS ID.
> 6. Update ref according to COS ID.
> 7. Save the COS ID into current domain's psr_cos_ids[socket] so that we
> can know which COS the domain is using on the socket.
>
> So, some functions are abstracted and the callback functions will be
> implemented in next patches.
>
> Here is an example to understand the process. The CPU supports
> two featuers, e.g. L3 CAT and L2 CAT. user wants to set L3 CAT
> of Dom1 to 0x1ff.
> 1. Get the old_cos of Dom1 which is 0. L3 CAT is the first
> element of feature list. The COS registers values are below at
> this time.
> -------------------------------
> | COS 0 | COS 1 | COS 2 | ... |
> -------------------------------
> L3 CAT | 0x7ff | ... | ... | ... |
> -------------------------------
> L2 CAT | 0xff | ... | ... | ... |
> -------------------------------
>
> 2. Assemble The value array to be:
> val[0]: 0x1ff
> val[1]: 0xff
>
> 3. It cannot find a matching COS.
>
> 4. Allocate COS 1 to store the value set.
>
> 5. Write the COS 1 registers. The COS registers values are
> changed to below now.
> -------------------------------
> | COS 0 | COS 1 | COS 2 | ... |
> -------------------------------
> L3 CAT | 0x7ff | 0x1ff | ... | ... |
> -------------------------------
> L2 CAT | 0xff | 0xff | ... | ... |
> -------------------------------
>
> 6. The ref[1] is increased to 1 because Dom1 is using it now.
>
> 7. Save 1 to Dom1's psr_cos_ids[socket].
>
> Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
> of Dom2 is 0 too. Repeat above flow.
>
> The val array assembled is:
> val[0]: 0x1ff
> val[1]: 0xff
>
> So, it can find a matching COS, COS 1. Then, it can reuse COS 1
> for Dom2.
>
> The ref[1] is increased to 2 now because both Dom1 and Dom2 are
> using this COS ID. Set 1 to Dom2's psr_cos_ids[socket].
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> xen/arch/x86/domctl.c | 18 ++---
> xen/arch/x86/psr.c | 202 +++++++++++++++++++++++++++++++++++++++++++++-
> xen/include/asm-x86/psr.h | 4 +-
> 3 files changed, 210 insertions(+), 14 deletions(-)
[...]
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index c1afd36..d414b5e 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -546,18 +546,214 @@ int psr_get_val(struct domain *d, unsigned int socket,
> return psr_get(socket, type, NULL, 0, d, val);
> }
>
> -int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> - uint64_t cbm, enum cbm_type type)
> +/* Set value functions */
> +static unsigned int get_cos_num(const struct psr_socket_info *info)
> {
> return 0;
> }
>
> +static int assemble_val_array(uint64_t *val,
> + uint32_t array_len,
> + const struct psr_socket_info *info,
> + unsigned int old_cos)
> +{
> + return -EINVAL;
> +}
> +
> +static int set_new_val_to_array(uint64_t *val,
> + uint32_t array_len,
> + const struct psr_socket_info *info,
> + enum psr_feat_type feat_type,
> + enum cbm_type type,
> + uint64_t m)
> +{
> + return -EINVAL;
> +}
> +
> +static int find_cos(const uint64_t *val, uint32_t array_len,
> + enum psr_feat_type feat_type,
> + const struct psr_socket_info *info)
> +{
ASSERT(spin_is_locked(info->ref_lock));
> + return -ENOENT;
> +}
> +
> +static int pick_avail_cos(const struct psr_socket_info *info,
> + const uint64_t *val, uint32_t array_len,
> + unsigned int old_cos,
> + enum psr_feat_type feat_type)
> +{
ASSERT(spin_is_locked(info->ref_lock));
> + return -ENOENT;
> +}
> +
> +static int write_psr_msr(unsigned int socket, unsigned int cos,
> + const uint64_t *val)
> +{
ASSERT(spin_is_locked(info->ref_lock));
> + return -ENOENT;
> +}
> +
> +int psr_set_val(struct domain *d, unsigned int socket,
> + uint64_t val, enum cbm_type type)
> +{
> + unsigned int old_cos;
> + int cos, ret;
> + unsigned int *ref;
> + uint64_t *val_array;
> + struct psr_socket_info *info = get_socket_info(socket);
> + uint32_t array_len;
> + enum psr_feat_type feat_type;
> +
> + if ( IS_ERR(info) )
> + return PTR_ERR(info);
> +
> + feat_type = psr_cbm_type_to_feat_type(type);
> + if ( !test_bit(feat_type, &info->feat_mask) )
> + return -ENOENT;
> +
> + /*
> + * Step 0:
> + * old_cos means the COS ID current domain is using. By default, it is 0.
> + *
> + * For every COS ID, there is a reference count to record how many domains
> + * are using the COS register corresponding to this COS ID.
> + * - If ref[old_cos] is 0, that means this COS is not used by any domain.
> + * - If ref[old_cos] is 1, that means this COS is only used by current
> + * domain.
> + * - If ref[old_cos] is more than 1, that mean multiple domains are using
> + * this COS.
> + */
> + old_cos = d->arch.psr_cos_ids[socket];
> + if ( old_cos > MAX_COS_REG_CNT )
> + return -EOVERFLOW;
> +
> + ref = info->cos_ref;
> +
> + /*
> + * Step 1:
> + * Assemle a value array to store all featues cos_reg_val[old_cos].
> + * And, set the input val into array according to the feature's
> + * position in array.
> + */
> + array_len = get_cos_num(info);
> + val_array = xzalloc_array(uint64_t, array_len);
> + if ( !val_array )
> + return -ENOMEM;
> +
> + if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 )
> + {
> + xfree(val_array);
> + return ret;
> + }
> +
> + if ( (ret = set_new_val_to_array(val_array, array_len, info,
> + feat_type, type, val)) != 0 )
> + {
> + xfree(val_array);
> + return ret;
> + }
> +
> + /*
> + * Lock here to make sure the ref is not changed during find and
> + * write process.
> + */
> + spin_lock(&info->ref_lock);
> +
> + /*
> + * Step 2:
> + * Try to find if there is already a COS ID on which all features' values
> + * are same as the array. Then, we can reuse this COS ID.
> + */
> + cos = find_cos(val_array, array_len, feat_type, info);
> + if ( cos >= 0 )
> + {
> + if ( cos == old_cos )
> + {
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return 0;
> + }
> + }
> + else
> + {
> + /*
> + * Step 3:
> + * If fail to find, we need allocate a new COS ID.
> + * If multiple domains are using same COS ID, its ref is more
> + * than 1. That means we cannot free this COS to make current domain
> + * use it. Because other domains are using the value saved in the COS.
> + * Unless the ref is changed to 1 (mean only current domain is using
> + * it), we cannot allocate the COS ID to current domain.
> + * So, only the COS ID which ref is 1 or 0 can be allocated.
> + */
> + cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
> + if ( cos < 0 )
> + {
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return cos;
> + }
> +
> + /*
> + * Step 4:
> + * Write all features MSRs according to the COS ID.
> + */
> + ret = write_psr_msr(socket, cos, val_array);
> + if ( ret )
> + {
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return ret;
> + }
> + }
> +
> + /*
> + * Step 5:
> + * Update ref according to COS ID.
> + */
> + ref[cos]++;
> + ASSERT(ref[cos] || cos == 0);
> + ref[old_cos]--;
If cos == 0 you can overflow ref[0].
> + spin_unlock(&info->ref_lock);
> +
> + /*
> + * Step 6:
> + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> + * which COS the domain is using on the socket. One domain can only use
> + * one COS ID at same time on each socket.
> + */
> + d->arch.psr_cos_ids[socket] = cos;
> + xfree(val_array);
> +
> + return 0;
> +}
> +
> /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> static void psr_free_cos(struct domain *d)
> {
> - if( !d->arch.psr_cos_ids )
> + unsigned int socket, cos;
An ASSERT that the domain is locked would be better than a comment.
> + if ( !d->arch.psr_cos_ids )
> return;
>
> + /* Domain is free so its cos_ref should be decreased. */
> + for ( socket = 0; socket < nr_sockets; socket++ )
> + {
> + struct psr_socket_info *info;
> +
> + /* cos 0 is default one which does not need be handled. */
> + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
> + continue;
> +
> + /*
> + * If domain uses other cos ids, all corresponding refs must have been
> + * increased 1 for this domain. So, we need decrease them.
> + */
> + info = socket_info + socket;
> + ASSERT(info->cos_ref[cos] || cos == 0);
> + spin_lock(&info->ref_lock);
> + info->cos_ref[cos]--;
If cos == 0, it is possible to reach this with info->cos_ref[cos] == 0, in
which case you are overflowing it?
> + spin_unlock(&info->ref_lock);
> + }
> +
> xfree(d->arch.psr_cos_ids);
> d->arch.psr_cos_ids = NULL;
> }
> diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
> index 569e7df..fde7882 100644
> --- a/xen/include/asm-x86/psr.h
> +++ b/xen/include/asm-x86/psr.h
> @@ -72,8 +72,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> uint32_t data[], unsigned int array_len);
> int psr_get_val(struct domain *d, unsigned int socket,
> uint64_t *val, enum cbm_type type);
> -int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> - uint64_t cbm, enum cbm_type type);
> +int psr_set_val(struct domain *d, unsigned int socket,
> + uint64_t val, enum cbm_type type);
>
> int psr_domain_init(struct domain *d);
> void psr_domain_free(struct domain *d);
> --
> 1.9.1
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 13/24] x86: refactor psr: implement CPU init and free flow for CDP.
2017-02-15 8:49 ` [PATCH v8 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
@ 2017-02-28 14:52 ` Roger Pau Monné
2017-03-09 14:53 ` Jan Beulich
1 sibling, 0 replies; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 14:52 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:28PM +0800, Yi Sun wrote:
> This patch implements the CPU init and free flow for CDP including L3 CDP
> initialization callback function.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> xen/arch/x86/psr.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++----
> 1 file changed, 98 insertions(+), 6 deletions(-)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 82bb8fe..4c08779 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -97,6 +97,7 @@ struct psr_cat_hw_info {
> struct feat_hw_info {
> union {
> struct psr_cat_hw_info l3_cat_info;
> + struct psr_cat_hw_info l3_cdp_info;
> };
> };
>
> @@ -195,6 +196,22 @@ struct feat_node {
> struct list_head list;
> };
>
> +/*
> + * get_data - get DATA COS register value from input COS ID.
> + * @feat: the feature list entry.
> + * @cos: the COS ID.
> + */
> +#define get_cdp_data(feat, cos) \
> + ( feat->cos_reg_val[cos * 2] )
This should be:
((feat)->cos_reg_val[(cos) * 2])
And the same treatment should be applied to the macro below.
> +
> +/*
> + * get_cdp_code - get CODE COS register value from input COS ID.
> + * @feat: the feature list entry.
> + * @cos: the COS ID.
> + */
> +#define get_cdp_code(feat, cos) \
> + ( feat->cos_reg_val[cos * 2 + 1] )
> +
> struct psr_assoc {
> uint64_t val;
> uint64_t cos_mask;
> @@ -217,6 +234,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
> * cpu_add_remove_lock spinlock.
> */
> static struct feat_node *feat_l3_cat;
> +static struct feat_node *feat_l3_cdp;
>
> /* Common functions. */
> static void free_feature(struct psr_socket_info *info)
> @@ -457,6 +475,63 @@ static const struct feat_ops l3_cat_ops = {
> .write_msr = l3_cat_write_msr,
> };
>
> +/* L3 CDP functions implementation. */
> +static void l3_cdp_init_feature(struct cpuid_leaf regs,
> + struct feat_node *feat,
> + struct psr_socket_info *info)
> +{
> + struct psr_cat_hw_info l3_cdp = { };
> + unsigned int socket;
> + uint64_t val;
> +
> + /* No valid value so do not enable feature. */
> + if ( !regs.a || !regs.d )
> + return;
> +
> + l3_cdp.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
> + /* Cut half of cos_max when CDP is enabled. */
> + l3_cdp.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK) >> 1;
> +
> + /* cos=0 is reserved as default cbm(all ones). */
> + get_cdp_code(feat, 0) =
> + (1ull << l3_cdp.cbm_len) - 1;
I think that all those ull sufixes should be turned into uint64_t casts,
because that's the type that you are actually using. Or else just use ul, which
is the same and shorter.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 14/24] x86: refactor psr: implement get hw info flow for CDP.
2017-02-15 8:49 ` [PATCH v8 14/24] x86: refactor psr: implement get hw info " Yi Sun
2017-02-26 17:43 ` Wei Liu
@ 2017-02-28 14:54 ` Roger Pau Monné
1 sibling, 0 replies; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 14:54 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:29PM +0800, Yi Sun wrote:
> This patch implements get HW info flow for CDP including L3 CDP callback
> function.
>
> It also changes sysctl function to make it work for CDP.
>
> With this patch, 'psr-hwinfo' can work for L3 CDP.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> xen/arch/x86/psr.c | 18 ++++++++++++++++++
> xen/arch/x86/sysctl.c | 24 +++++++++++++++++++++---
> 2 files changed, 39 insertions(+), 3 deletions(-)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 4c08779..72c9888 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -270,6 +270,10 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> case PSR_CBM_TYPE_L3:
> feat_type = PSR_SOCKET_L3_CAT;
> break;
> + case PSR_CBM_TYPE_L3_DATA:
> + case PSR_CBM_TYPE_L3_CODE:
> + feat_type = PSR_SOCKET_L3_CDP;
> + break;
> default:
> feat_type = PSR_SOCKET_UNKNOWN;
> break;
> @@ -528,8 +532,22 @@ static unsigned int l3_cdp_get_cos_max(const struct feat_node *feat)
> return feat->info.l3_cdp_info.cos_max;
> }
>
> +static bool l3_cdp_get_feat_info(const struct feat_node *feat,
> + uint32_t data[], uint32_t array_len)
> +{
> + if ( !data || 3 > array_len )
array_len != 3?
> + return false;
> +
> + data[CBM_LEN] = feat->info.l3_cdp_info.cbm_len;
> + data[COS_MAX] = feat->info.l3_cdp_info.cos_max;
> + data[PSR_FLAG] |= XEN_SYSCTL_PSR_CAT_L3_CDP;
> +
> + return true;
> +}
> +
> struct feat_ops l3_cdp_ops = {
> .get_cos_max = l3_cdp_get_cos_max,
> + .get_feat_info = l3_cdp_get_feat_info,
> };
>
> static void __init parse_psr_bool(char *s, char *value, char *feature,
> diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
> index e340baa..568bfe9 100644
> --- a/xen/arch/x86/sysctl.c
> +++ b/xen/arch/x86/sysctl.c
> @@ -181,9 +181,27 @@ long arch_do_sysctl(
> ret = psr_get_info(sysctl->u.psr_cat_op.target,
> PSR_CBM_TYPE_L3, data, 3);
>
> - sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
> - sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
> - sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
> + if ( !ret )
> + {
> + sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
> + sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
> + sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
> + } else {
Coding style, should be:
}
else
{
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 15/24] x86: refactor psr: implement get value flow for CDP.
2017-02-15 8:49 ` [PATCH v8 15/24] x86: refactor psr: implement get value " Yi Sun
@ 2017-02-28 14:59 ` Roger Pau Monné
0 siblings, 0 replies; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 14:59 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:30PM +0800, Yi Sun wrote:
> This patch implements L3 CDP get value callback function.
>
> With this patch, 'psr-cat-show' can work for L3 CDP.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> xen/arch/x86/psr.c | 16 ++++++++++++++++
> 1 file changed, 16 insertions(+)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 72c9888..72ed923 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -545,9 +545,25 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
> return true;
> }
>
> +static bool l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> + enum cbm_type type, uint64_t *val)
> +{
> + if ( cos > feat->info.l3_cdp_info.cos_max )
> + /* Use default value. */
> + cos = 0;
As with the other get_val, I think that this should return false.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow.
2017-02-15 8:49 ` [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
@ 2017-02-28 15:15 ` Roger Pau Monné
2017-03-01 6:35 ` Yi Sun
2017-03-09 15:04 ` Jan Beulich
1 sibling, 1 reply; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 15:15 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:32PM +0800, Yi Sun wrote:
> This patch implements the CPU init and free flow for L2 CAT including
> L2 CAT initialization callback function.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> xen/arch/x86/psr.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++
> xen/include/asm-x86/psr.h | 1 +
> 2 files changed, 73 insertions(+)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 0a0bab9..4489637 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -98,6 +98,7 @@ struct feat_hw_info {
> union {
> struct psr_cat_hw_info l3_cat_info;
> struct psr_cat_hw_info l3_cdp_info;
> + struct psr_cat_hw_info l2_cat_info;
> };
> };
>
> @@ -235,6 +236,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
> */
> static struct feat_node *feat_l3_cat;
> static struct feat_node *feat_l3_cdp;
> +static struct feat_node *feat_l2_cat;
>
> /* Common functions. */
> static void free_feature(struct psr_socket_info *info)
> @@ -684,6 +686,53 @@ struct feat_ops l3_cdp_ops = {
> .write_msr = l3_cdp_write_msr,
> };
>
> +/* L2 CAT callback functions implementation. */
> +static void l2_cat_init_feature(struct cpuid_leaf regs,
> + struct feat_node *feat,
> + struct psr_socket_info *info)
This is exactly the same code as l3_cat_init_feature with s/l3/l2/. I think
those two functions can somehow be merged, and I'm sure there's room for other
merges, IMHO L2/L3 CAT implementations should share a bunch of code, there's
too much duplication.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow.
2017-02-15 8:49 ` [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-02-28 15:18 ` Roger Pau Monné
2017-03-09 15:13 ` Jan Beulich
1 sibling, 0 replies; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 15:18 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:33PM +0800, Yi Sun wrote:
> This patch implements get HW info flow for L2 CAT including L2 CAT callback
> function.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> xen/arch/x86/psr.c | 16 ++++++++++++++++
> xen/arch/x86/sysctl.c | 15 +++++++++++++++
> xen/include/asm-x86/psr.h | 1 +
> xen/include/public/sysctl.h | 6 ++++++
> 4 files changed, 38 insertions(+)
>
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 4489637..05100b4 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -276,6 +276,9 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> case PSR_CBM_TYPE_L3_CODE:
> feat_type = PSR_SOCKET_L3_CDP;
> break;
> + case PSR_CBM_TYPE_L2:
> + feat_type = PSR_SOCKET_L2_CAT;
> + break;
> default:
> feat_type = PSR_SOCKET_UNKNOWN;
> break;
> @@ -729,8 +732,21 @@ static unsigned int l2_cat_get_cos_max(const struct feat_node *feat)
> return feat->info.l2_cat_info.cos_max;
> }
>
> +static bool l2_cat_get_feat_info(const struct feat_node *feat,
> + uint32_t data[], uint32_t array_len)
> +{
> + if ( !data || 2 > array_len )
> + return false;
> +
> + data[CBM_LEN] = feat->info.l2_cat_info.cbm_len;
> + data[COS_MAX] = feat->info.l2_cat_info.cos_max;
> +
> + return true;
> +}
> +
> struct feat_ops l2_cat_ops = {
> .get_cos_max = l2_cat_get_cos_max,
> + .get_feat_info = l2_cat_get_feat_info,
> };
>
> static void __init parse_psr_bool(char *s, char *value, char *feature,
> diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
> index 568bfe9..01cf3b7 100644
> --- a/xen/arch/x86/sysctl.c
> +++ b/xen/arch/x86/sysctl.c
> @@ -207,6 +207,21 @@ long arch_do_sysctl(
> ret = -EFAULT;
> break;
> }
> + case XEN_SYSCTL_PSR_CAT_get_l2_info:
> + {
> + uint32_t dat[2];
Missing newline, and probalby you want "data" here to match with
XEN_SYSCTL_PSR_CAT_get_l3_info?
Also I think this 2 should be a constant, and it should be used both here and
in l2_cat_get_feat_info.
> + ret = psr_get_info(sysctl->u.psr_cat_op.target,
> + PSR_CBM_TYPE_L2, dat, 2);
ARRAY_SIZE(data)
> + if ( ret )
> + break;
> +
> + sysctl->u.psr_cat_op.u.l2_info.cbm_len = dat[CBM_LEN];
> + sysctl->u.psr_cat_op.u.l2_info.cos_max = dat[COS_MAX];
> +
> + if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
> + ret = -EFAULT;
> + break;
> + }
> default:
> ret = -EOPNOTSUPP;
> break;
> diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
> index d7ed012..2e1b3d0 100644
> --- a/xen/include/asm-x86/psr.h
> +++ b/xen/include/asm-x86/psr.h
> @@ -56,6 +56,7 @@ enum cbm_type {
> PSR_CBM_TYPE_L3,
> PSR_CBM_TYPE_L3_CODE,
> PSR_CBM_TYPE_L3_DATA,
> + PSR_CBM_TYPE_L2,
> };
>
> extern struct psr_cmt *psr_cmt;
> diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
> index 00f5e77..cbf5372 100644
> --- a/xen/include/public/sysctl.h
> +++ b/xen/include/public/sysctl.h
> @@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
> DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
>
> #define XEN_SYSCTL_PSR_CAT_get_l3_info 0
> +#define XEN_SYSCTL_PSR_CAT_get_l2_info 1
> struct xen_sysctl_psr_cat_op {
> uint32_t cmd; /* IN: XEN_SYSCTL_PSR_CAT_* */
> uint32_t target; /* IN */
> @@ -754,6 +755,11 @@ struct xen_sysctl_psr_cat_op {
> #define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
> uint32_t flags; /* OUT: CAT flags */
> } l3_info;
> +
> + struct {
> + uint32_t cbm_len; /* OUT: CBM length */
> + uint32_t cos_max; /* OUT: Maximum COS */
> + } l2_info;
> } u;
> };
> typedef struct xen_sysctl_psr_cat_op xen_sysctl_psr_cat_op_t;
> --
> 1.9.1
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 19/24] x86: L2 CAT: implement get value flow.
2017-02-15 8:49 ` [PATCH v8 19/24] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-02-28 15:20 ` Roger Pau Monné
0 siblings, 0 replies; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 15:20 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:34PM +0800, Yi Sun wrote:
> This patch implements L2 CAT get value callback function and
> interface in domctl.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 05100b4..2adf62c 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -744,9 +744,21 @@ static bool l2_cat_get_feat_info(const struct feat_node *feat,
> return true;
> }
>
> +static bool l2_cat_get_val(const struct feat_node *feat, unsigned int cos,
> + enum cbm_type type, uint64_t *val)
> +{
> + if ( cos > feat->info.l2_cat_info.cos_max )
> + cos = 0;
return false;?
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 20/24] x86: L2 CAT: implement set value flow.
2017-02-15 8:49 ` [PATCH v8 20/24] x86: L2 CAT: implement set " Yi Sun
@ 2017-02-28 15:25 ` Roger Pau Monné
2017-03-01 6:59 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Roger Pau Monné @ 2017-02-28 15:25 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 15, 2017 at 04:49:35PM +0800, Yi Sun wrote:
> This patch implements L2 CAT set value related callback functions
> and domctl interface.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> v8:
> - modify 'l2_cat_write_msr' to 'void'.
> ---
> xen/arch/x86/domctl.c | 6 +++
> xen/arch/x86/psr.c | 95 +++++++++++++++++++++++++++++++++++++++++
> xen/include/asm-x86/msr-index.h | 1 +
> xen/include/public/domctl.h | 1 +
> 4 files changed, 103 insertions(+)
>
> diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
> index 68c2d60..38dc087 100644
> --- a/xen/arch/x86/domctl.c
> +++ b/xen/arch/x86/domctl.c
> @@ -1439,6 +1439,12 @@ long arch_do_domctl(
> PSR_CBM_TYPE_L3_DATA);
> break;
>
> + case XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM:
> + ret = psr_set_val(d, domctl->u.psr_cat_op.target,
> + domctl->u.psr_cat_op.data,
> + PSR_CBM_TYPE_L2);
> + break;
> +
> case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
> ret = psr_get_val(d, domctl->u.psr_cat_op.target,
> &domctl->u.psr_cat_op.data,
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 2adf62c..5604e03 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -755,10 +755,105 @@ static bool l2_cat_get_val(const struct feat_node *feat, unsigned int cos,
> return true;
> }
>
> +static unsigned int l2_cat_get_cos_num(const struct feat_node *feat)
> +{
> + /* L2 CAT uses one COS. */
> + return 1;
> +}
> +
> +static int l2_cat_get_old_val(uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int old_cos)
> +{
> + if ( old_cos > feat->info.l2_cat_info.cos_max )
> + /* Use default value. */
> + old_cos = 0;
> +
> + val[0] = feat->cos_reg_val[old_cos];
> +
> + return 0;
> +}
> +
> +static int l2_cat_set_new_val(uint64_t val[],
> + const struct feat_node *feat,
> + enum cbm_type type,
> + uint64_t m)
> +{
> + if ( !psr_check_cbm(feat->info.l2_cat_info.cbm_len, m) )
> + return -EINVAL;
> +
> + val[0] = m;
> +
> + return 0;
> +}
> +
> +static int l2_cat_compare_val(const uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int cos, bool *found)
> +{
> + uint64_t l2_def_cbm;
> +
> + l2_def_cbm = (1ull << feat->info.l2_cat_info.cbm_len) - 1;
> +
> + if ( cos > feat->info.l2_cat_info.cos_max )
> + {
> + if ( val[0] != l2_def_cbm )
> + {
> + *found = false;
> + return -ENOENT;
> + }
> + *found = true;
> + }
> + else
> + *found = (val[0] == feat->cos_reg_val[cos]);
> +
> + return 0;
The logic of this function is kind of weird IMHO, you seem to be able to return
an error, and also a parameter that indicates success ("found"). Can't this be
simplified to simply return an error code, and the found parameter removed?
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c
2017-02-28 11:02 ` Roger Pau Monné
@ 2017-03-01 4:54 ` Yi Sun
2017-03-01 8:35 ` Roger Pau Monn�
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-01 4:54 UTC (permalink / raw)
To: Roger Pau Monn�
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-28 11:02:42, Roger Pau Monn� wrote:
> On Wed, Feb 15, 2017 at 04:49:15PM +0800, Yi Sun wrote:
> > Hi all,
> >
> > We plan to bring a new PSR (Platform Shared Resource) feature called
> > Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
> >
> > Besides the L2 CAT implementaion, we refactor the psr.c to make it more
> > flexible to add new features and fulfill the principle, open for extension
> > but closed for modification. We abstract the general operations of all
>
> Hello,
>
> I see that you use the term "open for extension but closed for modification"
> here and in order patches. Could you please clarify what this means? "closed
> for modification" doesn't seem to make much sense for an open source project,
> where all the code is always open for modification.
>
> Thanks, Roger.
Thanks for your review! "closed for modification" means we do not need modify
the most codes, especially the main flows, when we add a new feature into
codes.
BRs,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 03/24] x86: refactor psr: implement main data structures.
2017-02-28 11:58 ` Roger Pau Monné
@ 2017-03-01 5:10 ` Yi Sun
2017-03-01 8:17 ` Jan Beulich
2017-03-01 8:49 ` Roger Pau Monn�
0 siblings, 2 replies; 122+ messages in thread
From: Yi Sun @ 2017-03-01 5:10 UTC (permalink / raw)
To: Roger Pau Monn�
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-28 11:58:59, Roger Pau Monn� wrote:
> On Wed, Feb 15, 2017 at 04:49:18PM +0800, Yi Sun wrote:
> > To construct an extendible framework, we need analyze PSR features
> > and abstract the common things and feature specific things. Then,
> > encapsulate them into different data structures.
> >
> > By analyzing PSR features, we can get below map.
> > +------+------+------+
> > --------->| Dom0 | Dom1 | ... |
> > | +------+------+------+
> > | |
> > |Dom ID | cos_id of domain
> > | V
> > | +-----------------------------------------------------------------------------+
> > User --------->| PSR |
> > Socket ID | +--------------+---------------+---------------+ |
> > | | Socket0 Info | Socket 1 Info | ... | |
> > | +--------------+---------------+---------------+ |
> > | | cos_id=0 cos_id=1 ... |
> > | | +-----------------------+-----------------------+-----------+ |
> > | |->Ref : | ref 0 | ref 1 | ... | |
> > | | +-----------------------+-----------------------+-----------+ |
> > | | +-----------------------+-----------------------+-----------+ |
> > | |->L3 CAT: | cos 0 | cos 1 | ... | |
> > | | +-----------------------+-----------------------+-----------+ |
> > | | +-----------------------+-----------------------+-----------+ |
> > | |->L2 CAT: | cos 0 | cos 1 | ... | |
> > | | +-----------------------+-----------------------+-----------+ |
> > | | +-----------+-----------+-----------+-----------+-----------+ |
> > | |->CDP : | cos0 code | cos0 data | cos1 code | cos1 data | ... | |
> > | +-----------+-----------+-----------+-----------+-----------+ |
> > +-----------------------------------------------------------------------------+
> >
> > So, we need define a socket info data structure, 'struct
> > psr_socket_info' to manage information per socket. It contains a
> > reference count array according to COS ID and a feature list to
> > manage all features enabled. Every entry of the reference count
> > array is used to record how many domains are using the COS registers
> > according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
> > Dom1 uses COS_ID=1 registers of both features to save CBM values, like
> > below.
> > +-------+-------+-------+-----+
> > | COS 0 | COS 1 | COS 2 | ... |
> > +-------+-------+-------+-----+
> > L3 CAT | 0x7ff | 0x1ff | ... | ... |
> > +-------+-------+-------+-----+
> > L2 CAT | 0xff | 0xff | ... | ... |
> > +-------+-------+-------+-----+
> >
> > If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
> > That means, both Dom1 and Dom2 use same COS registers(ID=1) to save same
> > L3/L2 values. So, the value ref[1] is 2 which means 2 domains are using
> > COS_ID 1.
> >
> > To manage a feature, we need define a feature node data structure,
> > 'struct feat_node', to manage feature's specific HW info, its callback
> > functions (all feature's specific behaviors are encapsulated into these
> > callback functions), and an array of all COS registers values of this
> > feature.
> >
> > CDP is a special feature which uses two entries of the array
> > for one COS ID. So, the number of CDP COS registers is the half of L3
> > CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
> > it is enabled. CDP uses the COS registers array as below.
> >
> > +-----------+-----------+-----------+-----------+-----------+
> > CDP cos_reg_val[] index: | 0 | 1 | 2 | 3 | ... |
> > +-----------+-----------+-----------+-----------+-----------+
> > value: | cos0 code | cos0 data | cos1 code | cos1 data | ... |
> > +-----------+-----------+-----------+-----------+-----------+
> >
> > For more details, please refer SDM and patches to implement 'get value' and
> > 'set value'.
>
> I would recommend that you merge this with a patch that actually makes use of
> this structures, or else it's hard to review it's usage IMHO.
>
Sorry for this. To make codes less and simpler in a patch, I split this patch
out to only show the data structures. I think I can merge it with next patch:
[PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
How do you think?
> > +struct psr_socket_info {
> > + /*
> > + * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
> > + * psr_feat_type' means the bit position.
> > + * bit 0: L3 CAT
> > + * bit 1: L3 CDP
> > + * bit 2: L2 CAT
> > + */
> > + unsigned int feat_mask;
> > + unsigned int nr_feat;
> > + struct list_head feat_list;
>
> Isn't it a little bit overkill to use a list when there can only be a maximum
> of 3 features? (and the feat_mask is currently 32bits, so I guess you don't
> really foresee having more than 32 features).
>
> I would suggest using:
>
> struct feat_node *features[PSR_SOCKET_MAX_FEAT];
>
> (PSR_SOCKET_MAX_FEAT comes from the expansion of the enum below). Then you can
> simply use the enum value of each feature as the position of it's corresponding
> feat_node into the array.
>
I really thought this before. But there may be different features enabled on
different sockets. For example, socket 0 enables L3 CAT and L2 CAT but socket 1
only supports L3 CAT. So, the feature array may be different for different
sockets. I think it is more complex to use array to handle all things than list.
> > + unsigned int cos_ref[MAX_COS_REG_CNT];
> > + spinlock_t ref_lock;
> > +};
> > +
> > +enum psr_feat_type {
> > + PSR_SOCKET_L3_CAT = 0,
> > + PSR_SOCKET_L3_CDP,
> > + PSR_SOCKET_L2_CAT,
> PSR_SOCKET_MAX_FEAT,
> > +};
> > +
> > +/* CAT/CDP HW info data structure. */
> > +struct psr_cat_hw_info {
> > + unsigned int cbm_len;
> > + unsigned int cos_max;
> > +};
> > +
> > +/* Encapsulate feature specific HW info here. */
> > +struct feat_hw_info {
> > + union {
> > + struct psr_cat_hw_info l3_cat_info;
> > + };
> > +};
>
> Why don't you use an union here directly, instead of encapsulating an union
> inside of a struct?
>
> union feat_hw_info {
> struct psr_cat_hw_info l3_cat_info;
> };
>
> > +
> > +struct feat_node;
> > +
> > +/*
> > + * This structure defines feature operation callback functions. Every feature
> > + * enabled MUST implement such callback functions and register them to ops.
> > + *
> > + * Feature specific behaviors will be encapsulated into these callback
> > + * functions. Then, the main flows will not be changed when introducing a new
> > + * feature.
> > + */
> > +struct feat_ops {
> > + /* get_cos_max is used to get feature's cos_max. */
> > + unsigned int (*get_cos_max)(const struct feat_node *feat);
> > +};
> > +
> > +/*
> > + * This structure represents one feature.
> > + * feature - Which feature it is.
> > + * feat_ops - Feature operation callback functions.
> > + * info - Feature HW info.
> > + * cos_reg_val - Array to store the values of COS registers. One entry stores
> > + * the value of one COS register.
> > + * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
> > + * For CDP, two entries correspond to one COS_ID. E.g.
> > + * COS_ID=0 corresponds to cos_reg_val[0] (Data) and
> > + * cos_reg_val[1] (Code).
> > + * list - Feature list.
> > + */
> > +struct feat_node {
> > + enum psr_feat_type feature;
>
> If you index them in an array with the key being psr_feat_type I don't think
> you need that field.
>
I need this to check if input type can match this feature, you can refer get
val or set val flow. Thanks!
> > + struct feat_ops ops;
>
> I would place the function hooks in this struct directly, instead of nesting
> them inside of another struct. The hooks AFAICT are shared between all the
> different PSR features.
>
Jan suggested this before in v4 patch. We have discussed this and Jan accepts
current implementation. The reason is below:
"To facilitate the callback functions assignment for a feature, I defined
feature specific callback function ops like below.
struct feat_ops l3_cat_ops = {
.init_feature = l3_cat_init_feature,
.get_max_cos_max = l3_cat_get_max_cos_max,
......
};
And directly assign it to global feature node in initialization process like
below.
static void cpu_init_work(void)
{
......
feat_tmp = feat_l3_cat;
feat_l3_cat = NULL;
feat_tmp->ops = l3_cat_ops;
......
}
I think this can make codes be clear."
> > + struct feat_hw_info info;
>
> Same with this, you can place the actual union for storage here directly,
> instead of nesting it inside of feat_hw_info, so:
>
> union {
> struct psr_cat_hw_info l3_cat_info;
> } hw_info;
>
Thanks for the suggestion! Will do this.
> Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 07/24] x86: refactor psr: implement get value flow.
2017-02-28 12:44 ` Roger Pau Monné
@ 2017-03-01 5:21 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-03-01 5:21 UTC (permalink / raw)
To: Roger Pau Monn�
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-28 12:44:34, Roger Pau Monn� wrote:
> On Wed, Feb 15, 2017 at 04:49:22PM +0800, Yi Sun wrote:
> > +static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
> > + enum cbm_type type, uint64_t *val)
> > +{
> > + if ( cos > feat->info.l3_cat_info.cos_max )
> > + /* Use default value. */
> > + cos = 0;
>
> I don't know much, but shouldn't this return false instead of silently
> defaulting to 0? This doesn't seem to be what the caller expects.
>
If cos exceeds the cos_max, we should return default value saved in
cos_reg_val[0]. Let me explain more, different features have different
cos_max, e.g. L3 CAT cos_max=16, L2 CAT cos_max=8, user can set L3 CAT
COS[9] for a domain. When COS ID 9 is set to ASSOC register, it works
for all features. HW automatically works as default value for L2 CAT
because the COS ID exceeds the max.
> > @@ -498,6 +516,15 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> > if ( feat->feature != feat_type )
> > continue;
> >
> > + if ( d )
> > + {
> > + cos = d->arch.psr_cos_ids[socket];
> > + if ( feat->ops.get_val(feat, cos, type, val) )
> > + return 0;
> > + else
>
> No need for the "else" branch here.
>
Thanks!
> Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-02-28 13:58 ` Roger Pau Monné
@ 2017-03-01 6:23 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-03-01 6:23 UTC (permalink / raw)
To: Roger Pau Monn�
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-28 13:58:55, Roger Pau Monn� wrote:
> > +static int find_cos(const uint64_t *val, uint32_t array_len,
> > + enum psr_feat_type feat_type,
> > + const struct psr_socket_info *info)
> > +{
> ASSERT(spin_is_locked(info->ref_lock));
> > + return -ENOENT;
> > +}
> > +
> > +static int pick_avail_cos(const struct psr_socket_info *info,
> > + const uint64_t *val, uint32_t array_len,
> > + unsigned int old_cos,
> > + enum psr_feat_type feat_type)
> > +{
> ASSERT(spin_is_locked(info->ref_lock));
> > + return -ENOENT;
> > +}
> > +
> > +static int write_psr_msr(unsigned int socket, unsigned int cos,
> > + const uint64_t *val)
> > +{
> ASSERT(spin_is_locked(info->ref_lock));
> > + return -ENOENT;
> > +}
> > +
Thank you!
> > +int psr_set_val(struct domain *d, unsigned int socket,
> > + uint64_t val, enum cbm_type type)
> > +{
[...]
> > +
> > + /*
> > + * Step 5:
> > + * Update ref according to COS ID.
> > + */
> > + ref[cos]++;
> > + ASSERT(ref[cos] || cos == 0);
> > + ref[old_cos]--;
>
> If cos == 0 you can overflow ref[0].
>
COS[0] stores the default value only. The value in it cannot be changed. So we
do not check its ref.
> > + spin_unlock(&info->ref_lock);
> > +
> > + /*
> > + * Step 6:
> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> > + * which COS the domain is using on the socket. One domain can only use
> > + * one COS ID at same time on each socket.
> > + */
> > + d->arch.psr_cos_ids[socket] = cos;
> > + xfree(val_array);
> > +
> > + return 0;
> > +}
> > +
> > /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> > static void psr_free_cos(struct domain *d)
> > {
> > - if( !d->arch.psr_cos_ids )
> > + unsigned int socket, cos;
>
> An ASSERT that the domain is locked would be better than a comment.
>
psr_free_cos is called by psr_domain_free which is finally called by
put_domain().
#define put_domain(_d) \
if ( atomic_dec_and_test(&(_d)->refcnt) ) domain_destroy(_d)
So, it is protected by refcnt.
Do you think it is necessary to check refcnt here? If codes outside have
something error to breake this protection, we cannot assure if below codes
go wrong even by ASSERT check here.
> > + if ( !d->arch.psr_cos_ids )
> > return;
> >
> > + /* Domain is free so its cos_ref should be decreased. */
> > + for ( socket = 0; socket < nr_sockets; socket++ )
> > + {
> > + struct psr_socket_info *info;
> > +
> > + /* cos 0 is default one which does not need be handled. */
> > + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
> > + continue;
> > +
> > + /*
> > + * If domain uses other cos ids, all corresponding refs must have been
> > + * increased 1 for this domain. So, we need decrease them.
> > + */
> > + info = socket_info + socket;
> > + ASSERT(info->cos_ref[cos] || cos == 0);
> > + spin_lock(&info->ref_lock);
> > + info->cos_ref[cos]--;
>
> If cos == 0, it is possible to reach this with info->cos_ref[cos] == 0, in
> which case you are overflowing it?
>
As above explanation, we do not care the ref count of COS[0].
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow.
2017-02-28 15:15 ` Roger Pau Monné
@ 2017-03-01 6:35 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-03-01 6:35 UTC (permalink / raw)
To: Roger Pau Monn�
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-28 15:15:36, Roger Pau Monn� wrote:
> On Wed, Feb 15, 2017 at 04:49:32PM +0800, Yi Sun wrote:
> > This patch implements the CPU init and free flow for L2 CAT including
> > L2 CAT initialization callback function.
> >
> > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> > ---
> > xen/arch/x86/psr.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++
> > xen/include/asm-x86/psr.h | 1 +
> > 2 files changed, 73 insertions(+)
> >
> > diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> > index 0a0bab9..4489637 100644
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -98,6 +98,7 @@ struct feat_hw_info {
> > union {
> > struct psr_cat_hw_info l3_cat_info;
> > struct psr_cat_hw_info l3_cdp_info;
> > + struct psr_cat_hw_info l2_cat_info;
> > };
> > };
> >
> > @@ -235,6 +236,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
> > */
> > static struct feat_node *feat_l3_cat;
> > static struct feat_node *feat_l3_cdp;
> > +static struct feat_node *feat_l2_cat;
> >
> > /* Common functions. */
> > static void free_feature(struct psr_socket_info *info)
> > @@ -684,6 +686,53 @@ struct feat_ops l3_cdp_ops = {
> > .write_msr = l3_cdp_write_msr,
> > };
> >
> > +/* L2 CAT callback functions implementation. */
> > +static void l2_cat_init_feature(struct cpuid_leaf regs,
> > + struct feat_node *feat,
> > + struct psr_socket_info *info)
>
> This is exactly the same code as l3_cat_init_feature with s/l3/l2/. I think
> those two functions can somehow be merged, and I'm sure there's room for other
> merges, IMHO L2/L3 CAT implementations should share a bunch of code, there's
> too much duplication.
>
Thanks! Will consider to split common functions out.
> Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 20/24] x86: L2 CAT: implement set value flow.
2017-02-28 15:25 ` Roger Pau Monné
@ 2017-03-01 6:59 ` Yi Sun
2017-03-01 11:31 ` Dario Faggioli
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-01 6:59 UTC (permalink / raw)
To: Roger Pau Monn�
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-28 15:25:39, Roger Pau Monn� wrote:
> On Wed, Feb 15, 2017 at 04:49:35PM +0800, Yi Sun wrote:
> > +static int l2_cat_compare_val(const uint64_t val[],
> > + const struct feat_node *feat,
> > + unsigned int cos, bool *found)
> > +{
> > + uint64_t l2_def_cbm;
> > +
> > + l2_def_cbm = (1ull << feat->info.l2_cat_info.cbm_len) - 1;
> > +
> > + if ( cos > feat->info.l2_cat_info.cos_max )
> > + {
> > + if ( val[0] != l2_def_cbm )
> > + {
> > + *found = false;
> > + return -ENOENT;
> > + }
> > + *found = true;
> > + }
> > + else
> > + *found = (val[0] == feat->cos_reg_val[cos]);
> > +
> > + return 0;
>
> The logic of this function is kind of weird IMHO, you seem to be able to return
> an error, and also a parameter that indicates success ("found"). Can't this be
> simplified to simply return an error code, and the found parameter removed?
>
> Roger.
As I explained in previous patch, the value must be default value if the COS ID
exceeds the max ID. If not, we have to return error to exit the whole flow. That
is the reason we return '-ENOENT'.
In find_cos(), compare_val() is called to check if there is already a COS ID
that all features values are same as input. All features in list should be
checked. The 'found' is used record the final result, if all features values are
same as input value array. You can see, after traversal of feature list, we
return the cos if found is true.
Of course, I can change the function to remove 'found' from the parameter.
But is it so necessary? Thanks!
static int find_cos(const uint64_t *val, uint32_t array_len,
...
{
...
for ( cos = 0; cos <= cos_max; cos++ )
{
if ( cos && !ref[cos] )
continue;
/* Not found, need find again from beginning. */
val_tmp = val;
list_for_each_entry(feat, &info->feat_list, list)
{
/*
* Compare value according to feature list order.
* We must follow this order because value array is assembled
* as this order in get_old_set_new().
*/
ret = feat->ops.compare_val(val_tmp, feat, cos, &found);
if ( ret < 0 )
return ret;
/* If fail to match, go to next cos to compare. */
if ( !found )
break;
val_tmp += feat->ops.get_cos_num(feat);
if ( val_tmp - val > array_len )
return -EINVAL;
}
/* For this COS ID all entries in the values array did match. Use it. */
if ( found )
return cos;
}
...
}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 03/24] x86: refactor psr: implement main data structures.
2017-03-01 5:10 ` Yi Sun
@ 2017-03-01 8:17 ` Jan Beulich
2017-03-01 8:28 ` Yi Sun
2017-03-01 8:49 ` Roger Pau Monn�
1 sibling, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-01 8:17 UTC (permalink / raw)
To: roger.pau, Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 01.03.17 at 06:10, <yi.y.sun@linux.intel.com> wrote:
> On 17-02-28 11:58:59, Roger Pau Monn wrote:
>> On Wed, Feb 15, 2017 at 04:49:18PM +0800, Yi Sun wrote:
>> > + struct feat_ops ops;
>>
>> I would place the function hooks in this struct directly, instead of nesting
>> them inside of another struct. The hooks AFAICT are shared between all the
>> different PSR features.
>>
> Jan suggested this before in v4 patch. We have discussed this and Jan
> accepts
> current implementation. The reason is below:
I'm pretty sure I didn't (in this specific form). Instead you want this
to be a pointer (to a const struct instance), i.e. ...
> "To facilitate the callback functions assignment for a feature, I defined
> feature specific callback function ops like below.
>
> struct feat_ops l3_cat_ops = {
> .init_feature = l3_cat_init_feature,
> .get_max_cos_max = l3_cat_get_max_cos_max,
> ......
> };
>
> And directly assign it to global feature node in initialization process like
> below.
>
> static void cpu_init_work(void)
> {
> ......
> feat_tmp = feat_l3_cat;
> feat_l3_cat = NULL;
> feat_tmp->ops = l3_cat_ops;
feat_tmp->ops = &l3_cat_ops;
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 03/24] x86: refactor psr: implement main data structures.
2017-03-01 8:17 ` Jan Beulich
@ 2017-03-01 8:28 ` Yi Sun
2017-03-01 8:39 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-01 8:28 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-03-01 01:17:22, Jan Beulich wrote:
> >>> On 01.03.17 at 06:10, <yi.y.sun@linux.intel.com> wrote:
> > On 17-02-28 11:58:59, Roger Pau Monn wrote:
> >> On Wed, Feb 15, 2017 at 04:49:18PM +0800, Yi Sun wrote:
> >> > + struct feat_ops ops;
> >>
> >> I would place the function hooks in this struct directly, instead of nesting
> >> them inside of another struct. The hooks AFAICT are shared between all the
> >> different PSR features.
> >>
> > Jan suggested this before in v4 patch. We have discussed this and Jan
> > accepts
> > current implementation. The reason is below:
>
> I'm pretty sure I didn't (in this specific form). Instead you want this
> to be a pointer (to a const struct instance), i.e. ...
>
Sorry for confusion. I think you suggested same thing (maybe I misunderstood?).
Quoted from v4:
"What is the reason to have a separate structure for this, when you
don't store a pointer in struct feat_node? If this was inlined there,
the odd forward declaration of struct feat_node wouldn't be needed
either. (The same question may apply to struct feat_hw_info.)"
So I explained as below and asked if you can accept such implementation. And
got your positive ack. Of course, I should declare l3_cat_ops to const.
> > "To facilitate the callback functions assignment for a feature, I defined
> > feature specific callback function ops like below.
> >
> > struct feat_ops l3_cat_ops = {
> > .init_feature = l3_cat_init_feature,
> > .get_max_cos_max = l3_cat_get_max_cos_max,
> > ......
> > };
> >
> > And directly assign it to global feature node in initialization process like
> > below.
> >
> > static void cpu_init_work(void)
> > {
> > ......
> > feat_tmp = feat_l3_cat;
> > feat_l3_cat = NULL;
> > feat_tmp->ops = l3_cat_ops;
>
> feat_tmp->ops = &l3_cat_ops;
>
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c
2017-03-01 4:54 ` Yi Sun
@ 2017-03-01 8:35 ` Roger Pau Monn�
2017-03-01 8:40 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Roger Pau Monn� @ 2017-03-01 8:35 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Mar 01, 2017 at 12:54:54PM +0800, Yi Sun wrote:
> On 17-02-28 11:02:42, Roger Pau Monn� wrote:
> > On Wed, Feb 15, 2017 at 04:49:15PM +0800, Yi Sun wrote:
> > > Hi all,
> > >
> > > We plan to bring a new PSR (Platform Shared Resource) feature called
> > > Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
> > >
> > > Besides the L2 CAT implementaion, we refactor the psr.c to make it more
> > > flexible to add new features and fulfill the principle, open for extension
> > > but closed for modification. We abstract the general operations of all
> >
> > Hello,
> >
> > I see that you use the term "open for extension but closed for modification"
> > here and in order patches. Could you please clarify what this means? "closed
> > for modification" doesn't seem to make much sense for an open source project,
> > where all the code is always open for modification.
> >
> > Thanks, Roger.
>
> Thanks for your review! "closed for modification" means we do not need modify
> the most codes, especially the main flows, when we add a new feature into
> codes.
IMHO, just saying that the current code is easily to extend in order to add new
features sounds better.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 03/24] x86: refactor psr: implement main data structures.
2017-03-01 8:28 ` Yi Sun
@ 2017-03-01 8:39 ` Jan Beulich
0 siblings, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-01 8:39 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> On 01.03.17 at 09:28, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-01 01:17:22, Jan Beulich wrote:
>> >>> On 01.03.17 at 06:10, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-02-28 11:58:59, Roger Pau Monn wrote:
>> >> On Wed, Feb 15, 2017 at 04:49:18PM +0800, Yi Sun wrote:
>> >> > + struct feat_ops ops;
>> >>
>> >> I would place the function hooks in this struct directly, instead of
> nesting
>> >> them inside of another struct. The hooks AFAICT are shared between all the
>> >> different PSR features.
>> >>
>> > Jan suggested this before in v4 patch. We have discussed this and Jan
>> > accepts
>> > current implementation. The reason is below:
>>
>> I'm pretty sure I didn't (in this specific form). Instead you want this
>> to be a pointer (to a const struct instance), i.e. ...
>>
> Sorry for confusion. I think you suggested same thing (maybe I
> misunderstood?).
>
> Quoted from v4:
> "What is the reason to have a separate structure for this, when you
> don't store a pointer in struct feat_node? If this was inlined there,
> the odd forward declaration of struct feat_node wouldn't be needed
> either. (The same question may apply to struct feat_hw_info.)"
>
> So I explained as below and asked if you can accept such implementation. And
> got your positive ack. Of course, I should declare l3_cat_ops to const.
In which case an earlier question to answer is: Why can it not be a
pointer that gets stored?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c
2017-03-01 8:35 ` Roger Pau Monn�
@ 2017-03-01 8:40 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-03-01 8:40 UTC (permalink / raw)
To: Roger Pau Monn�
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-03-01 08:35:04, Roger Pau Monn� wrote:
> On Wed, Mar 01, 2017 at 12:54:54PM +0800, Yi Sun wrote:
> > On 17-02-28 11:02:42, Roger Pau Monn� wrote:
> > > On Wed, Feb 15, 2017 at 04:49:15PM +0800, Yi Sun wrote:
> > > > Hi all,
> > > >
> > > > We plan to bring a new PSR (Platform Shared Resource) feature called
> > > > Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
> > > >
> > > > Besides the L2 CAT implementaion, we refactor the psr.c to make it more
> > > > flexible to add new features and fulfill the principle, open for extension
> > > > but closed for modification. We abstract the general operations of all
> > >
> > > Hello,
> > >
> > > I see that you use the term "open for extension but closed for modification"
> > > here and in order patches. Could you please clarify what this means? "closed
> > > for modification" doesn't seem to make much sense for an open source project,
> > > where all the code is always open for modification.
> > >
> > > Thanks, Roger.
> >
> > Thanks for your review! "closed for modification" means we do not need modify
> > the most codes, especially the main flows, when we add a new feature into
> > codes.
>
> IMHO, just saying that the current code is easily to extend in order to add new
> features sounds better.
>
Sure, thank you!
> Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 03/24] x86: refactor psr: implement main data structures.
2017-03-01 5:10 ` Yi Sun
2017-03-01 8:17 ` Jan Beulich
@ 2017-03-01 8:49 ` Roger Pau Monn�
2017-03-01 8:54 ` Jan Beulich
1 sibling, 1 reply; 122+ messages in thread
From: Roger Pau Monn� @ 2017-03-01 8:49 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Mar 01, 2017 at 01:10:02PM +0800, Yi Sun wrote:
> On 17-02-28 11:58:59, Roger Pau Monn� wrote:
> > On Wed, Feb 15, 2017 at 04:49:18PM +0800, Yi Sun wrote:
> > > To construct an extendible framework, we need analyze PSR features
> > > and abstract the common things and feature specific things. Then,
> > > encapsulate them into different data structures.
> > >
> > > By analyzing PSR features, we can get below map.
> > > +------+------+------+
> > > --------->| Dom0 | Dom1 | ... |
> > > | +------+------+------+
> > > | |
> > > |Dom ID | cos_id of domain
> > > | V
> > > | +-----------------------------------------------------------------------------+
> > > User --------->| PSR |
> > > Socket ID | +--------------+---------------+---------------+ |
> > > | | Socket0 Info | Socket 1 Info | ... | |
> > > | +--------------+---------------+---------------+ |
> > > | | cos_id=0 cos_id=1 ... |
> > > | | +-----------------------+-----------------------+-----------+ |
> > > | |->Ref : | ref 0 | ref 1 | ... | |
> > > | | +-----------------------+-----------------------+-----------+ |
> > > | | +-----------------------+-----------------------+-----------+ |
> > > | |->L3 CAT: | cos 0 | cos 1 | ... | |
> > > | | +-----------------------+-----------------------+-----------+ |
> > > | | +-----------------------+-----------------------+-----------+ |
> > > | |->L2 CAT: | cos 0 | cos 1 | ... | |
> > > | | +-----------------------+-----------------------+-----------+ |
> > > | | +-----------+-----------+-----------+-----------+-----------+ |
> > > | |->CDP : | cos0 code | cos0 data | cos1 code | cos1 data | ... | |
> > > | +-----------+-----------+-----------+-----------+-----------+ |
> > > +-----------------------------------------------------------------------------+
> > >
> > > So, we need define a socket info data structure, 'struct
> > > psr_socket_info' to manage information per socket. It contains a
> > > reference count array according to COS ID and a feature list to
> > > manage all features enabled. Every entry of the reference count
> > > array is used to record how many domains are using the COS registers
> > > according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
> > > Dom1 uses COS_ID=1 registers of both features to save CBM values, like
> > > below.
> > > +-------+-------+-------+-----+
> > > | COS 0 | COS 1 | COS 2 | ... |
> > > +-------+-------+-------+-----+
> > > L3 CAT | 0x7ff | 0x1ff | ... | ... |
> > > +-------+-------+-------+-----+
> > > L2 CAT | 0xff | 0xff | ... | ... |
> > > +-------+-------+-------+-----+
> > >
> > > If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
> > > That means, both Dom1 and Dom2 use same COS registers(ID=1) to save same
> > > L3/L2 values. So, the value ref[1] is 2 which means 2 domains are using
> > > COS_ID 1.
> > >
> > > To manage a feature, we need define a feature node data structure,
> > > 'struct feat_node', to manage feature's specific HW info, its callback
> > > functions (all feature's specific behaviors are encapsulated into these
> > > callback functions), and an array of all COS registers values of this
> > > feature.
> > >
> > > CDP is a special feature which uses two entries of the array
> > > for one COS ID. So, the number of CDP COS registers is the half of L3
> > > CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
> > > it is enabled. CDP uses the COS registers array as below.
> > >
> > > +-----------+-----------+-----------+-----------+-----------+
> > > CDP cos_reg_val[] index: | 0 | 1 | 2 | 3 | ... |
> > > +-----------+-----------+-----------+-----------+-----------+
> > > value: | cos0 code | cos0 data | cos1 code | cos1 data | ... |
> > > +-----------+-----------+-----------+-----------+-----------+
> > >
> > > For more details, please refer SDM and patches to implement 'get value' and
> > > 'set value'.
> >
> > I would recommend that you merge this with a patch that actually makes use of
> > this structures, or else it's hard to review it's usage IMHO.
> >
> Sorry for this. To make codes less and simpler in a patch, I split this patch
> out to only show the data structures. I think I can merge it with next patch:
> [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
>
> How do you think?
Now that I've looked at the other patches this doesn't seem so abstract to me,
I will leave that to the x86 maintainers, and whether they want to join it with
some actual code or not.
> > > +struct psr_socket_info {
> > > + /*
> > > + * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
> > > + * psr_feat_type' means the bit position.
> > > + * bit 0: L3 CAT
> > > + * bit 1: L3 CDP
> > > + * bit 2: L2 CAT
> > > + */
> > > + unsigned int feat_mask;
> > > + unsigned int nr_feat;
> > > + struct list_head feat_list;
> >
> > Isn't it a little bit overkill to use a list when there can only be a maximum
> > of 3 features? (and the feat_mask is currently 32bits, so I guess you don't
> > really foresee having more than 32 features).
> >
> > I would suggest using:
> >
> > struct feat_node *features[PSR_SOCKET_MAX_FEAT];
> >
> > (PSR_SOCKET_MAX_FEAT comes from the expansion of the enum below). Then you can
> > simply use the enum value of each feature as the position of it's corresponding
> > feat_node into the array.
> >
> I really thought this before. But there may be different features enabled on
> different sockets. For example, socket 0 enables L3 CAT and L2 CAT but socket 1
> only supports L3 CAT. So, the feature array may be different for different
> sockets. I think it is more complex to use array to handle all things than list.
Different sockets with different features enabled should still be fine. Each
socket has a feat_mask, and you can use that to know whether a certain feature
is enabled or not.
What I was proposing is to make feat_node an array of pointers, so if a feature
is not supported that specific pointer would be NULL (ie: if feat_mask &
PSR_SOCKET_L3_CDP == 0, features[PSR_SOCKET_L3_CDP] == NULL). And then you can
avoid all the list traversing code.
> > > + unsigned int cos_ref[MAX_COS_REG_CNT];
> > > + spinlock_t ref_lock;
> > > +};
> > > +
> > > +enum psr_feat_type {
> > > + PSR_SOCKET_L3_CAT = 0,
> > > + PSR_SOCKET_L3_CDP,
> > > + PSR_SOCKET_L2_CAT,
> > PSR_SOCKET_MAX_FEAT,
> > > +};
> > > +
> > > +/* CAT/CDP HW info data structure. */
> > > +struct psr_cat_hw_info {
> > > + unsigned int cbm_len;
> > > + unsigned int cos_max;
> > > +};
> > > +
> > > +/* Encapsulate feature specific HW info here. */
> > > +struct feat_hw_info {
> > > + union {
> > > + struct psr_cat_hw_info l3_cat_info;
> > > + };
> > > +};
> >
> > Why don't you use an union here directly, instead of encapsulating an union
> > inside of a struct?
> >
> > union feat_hw_info {
> > struct psr_cat_hw_info l3_cat_info;
> > };
> >
> > > +
> > > +struct feat_node;
> > > +
> > > +/*
> > > + * This structure defines feature operation callback functions. Every feature
> > > + * enabled MUST implement such callback functions and register them to ops.
> > > + *
> > > + * Feature specific behaviors will be encapsulated into these callback
> > > + * functions. Then, the main flows will not be changed when introducing a new
> > > + * feature.
> > > + */
> > > +struct feat_ops {
> > > + /* get_cos_max is used to get feature's cos_max. */
> > > + unsigned int (*get_cos_max)(const struct feat_node *feat);
> > > +};
> > > +
> > > +/*
> > > + * This structure represents one feature.
> > > + * feature - Which feature it is.
> > > + * feat_ops - Feature operation callback functions.
> > > + * info - Feature HW info.
> > > + * cos_reg_val - Array to store the values of COS registers. One entry stores
> > > + * the value of one COS register.
> > > + * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
> > > + * For CDP, two entries correspond to one COS_ID. E.g.
> > > + * COS_ID=0 corresponds to cos_reg_val[0] (Data) and
> > > + * cos_reg_val[1] (Code).
> > > + * list - Feature list.
> > > + */
> > > +struct feat_node {
> > > + enum psr_feat_type feature;
> >
> > If you index them in an array with the key being psr_feat_type I don't think
> > you need that field.
> >
> I need this to check if input type can match this feature, you can refer get
> val or set val flow. Thanks!
If you move to the array of pointers proposed above you no longer need this
since you can just do features[PSR_SOCKET_L3_CAT] and get the pointer to the
feat_node struct, the index into the array is going to be the feature type
already.
> > > + struct feat_ops ops;
> >
> > I would place the function hooks in this struct directly, instead of nesting
> > them inside of another struct. The hooks AFAICT are shared between all the
> > different PSR features.
> >
> Jan suggested this before in v4 patch. We have discussed this and Jan accepts
> current implementation. The reason is below:
>
> "To facilitate the callback functions assignment for a feature, I defined
> feature specific callback function ops like below.
>
> struct feat_ops l3_cat_ops = {
> .init_feature = l3_cat_init_feature,
> .get_max_cos_max = l3_cat_get_max_cos_max,
> ......
> };
This should be constified as Jan already pointed out.
I see that what I said before doesn't make sense, since you have both constant
functions pointers (that can be shared across nodes), plus local node storage
in this structure.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 03/24] x86: refactor psr: implement main data structures.
2017-03-01 8:49 ` Roger Pau Monn�
@ 2017-03-01 8:54 ` Jan Beulich
2017-03-01 9:00 ` Roger Pau Monn�
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-01 8:54 UTC (permalink / raw)
To: Roger Pau Monn�, Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 01.03.17 at 09:49, <roger.pau@citrix.com> wrote:
> What I was proposing is to make feat_node an array of pointers, so if a feature
> is not supported that specific pointer would be NULL (ie: if feat_mask &
> PSR_SOCKET_L3_CDP == 0, features[PSR_SOCKET_L3_CDP] == NULL). And then you can
> avoid all the list traversing code.
In which case the feature flags would look to become redundant too.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 03/24] x86: refactor psr: implement main data structures.
2017-03-01 8:54 ` Jan Beulich
@ 2017-03-01 9:00 ` Roger Pau Monn�
0 siblings, 0 replies; 122+ messages in thread
From: Roger Pau Monn� @ 2017-03-01 9:00 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, Yi Sun, xen-devel, he.chen, andrew.cooper3,
dario.faggioli, ian.jackson, mengxu, chao.p.peng, wei.liu2
On Wed, Mar 01, 2017 at 01:54:00AM -0700, Jan Beulich wrote:
> >>> On 01.03.17 at 09:49, <roger.pau@citrix.com> wrote:
> > What I was proposing is to make feat_node an array of pointers, so if a feature
> > is not supported that specific pointer would be NULL (ie: if feat_mask &
> > PSR_SOCKET_L3_CDP == 0, features[PSR_SOCKET_L3_CDP] == NULL). And then you can
> > avoid all the list traversing code.
>
> In which case the feature flags would look to become redundant too.
I guess so, I'm just not sure if you can have a feature available (so
feat_mask bit set) but not enabled (so features[...] == NULL). Not sure if that
makes sense or is desirable at all.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 20/24] x86: L2 CAT: implement set value flow.
2017-03-01 6:59 ` Yi Sun
@ 2017-03-01 11:31 ` Dario Faggioli
0 siblings, 0 replies; 122+ messages in thread
From: Dario Faggioli @ 2017-03-01 11:31 UTC (permalink / raw)
To: Yi Sun, Roger Pau Monn�
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, ian.jackson,
mengxu, jbeulich, chao.p.peng, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 1491 bytes --]
On Wed, 2017-03-01 at 14:59 +0800, Yi Sun wrote:
> On 17-02-28 15:25:39, Roger Pau Monn� wrote:
> > The logic of this function is kind of weird IMHO, you seem to be
> > able to return
> > an error, and also a parameter that indicates success ("found").
> > Can't this be
> > simplified to simply return an error code, and the found parameter
> > removed?
> >
> In find_cos(), compare_val() is called to check if there is already a
> COS ID
> that all features values are same as input. All features in list
> should be
> checked. The 'found' is used record the final result, if all features
> values are
> same as input value array. You can see, after traversal of feature
> list, we
> return the cos if found is true.
>
So, you can, for instance, use different error values, and check for
which one has been returned in the caller. Like, use ENOENT for
'invalid ID' as you're doing right now, and, say, ESRCH, for 'feature
not found'.
Is that right?
> Of course, I can change the function to remove 'found' from the
> parameter.
> But is it so necessary? Thanks!
>
It's hard to tell whether it is "so necessary", but I agree with Roger
that it would improve the code. :-)
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-15 8:49 ` [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
2017-02-26 17:41 ` Wei Liu
@ 2017-03-08 14:56 ` Jan Beulich
2017-03-10 1:32 ` Yi Sun
1 sibling, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-08 14:56 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> This patch implements the CPU init and free flow including L3 CAT
> initialization and feature list free.
I think from the final-effect-of-the-patch point of view the L3 CAT
aspect is more relevant, so that's what should appear in the title.
> @@ -136,11 +140,87 @@ struct psr_assoc {
>
> struct psr_cmt *__read_mostly psr_cmt;
>
> +static struct psr_socket_info *__read_mostly socket_info;
> +
> static unsigned int opt_psr;
> static unsigned int __initdata opt_rmid_max = 255;
> +static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
> static uint64_t rmid_mask;
> static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
>
> +/*
> + * Declare global feature list entry for every feature to facilitate the
> + * feature list creation. It will be allocated in psr_cpu_prepare() and
> + * inserted into feature list in cpu_init_work(). It is protected by
> + * cpu_add_remove_lock spinlock.
> + */
> +static struct feat_node *feat_l3_cat;
The comment is misleading imo: The only relevant aspect here is that
of when the allocation would need to happen vs. when it actually
happens (because putting it where we'd like it to be is not easily
possible). There's nothing list related here. I'd recommend simply
saying that we need a place to transiently store a spare node.
> +/* Common functions. */
> +static void free_feature(struct psr_socket_info *info)
> +{
> + struct feat_node *feat, *next;
> +
> + if ( !info )
> + return;
> +
> + /*
> + * Free resources of features. But we do not free global feature list
> + * entry, like feat_l3_cat. Although it may cause a few memory leak,
> + * it is OK simplify things.
> + */
> + list_for_each_entry_safe(feat, next, &info->feat_list, list)
> + {
> + if ( !feat )
> + return;
How would that happen? But well, this is going to go away anyway
with the move from list to array.
> @@ -335,18 +418,104 @@ void psr_domain_free(struct domain *d)
> psr_free_rmid(d);
> }
>
> -static int psr_cpu_prepare(unsigned int cpu)
> +static void cpu_init_work(void)
> +{
> + struct psr_socket_info *info;
> + unsigned int socket;
> + unsigned int cpu = smp_processor_id();
> + struct feat_node *feat;
> + struct cpuid_leaf regs = { .a = 0, .b = 0, .c = 0, .d = 0 };
I don't see you needing an initializer here at all, but if you really
want one for some reason, the same effect can be had with just
{}.
> + if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
Do you really mean to not universally check the global (boot CPU)
flag? I.e. possibly differing behavior on different CPUs?
> + return;
> + else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
Pointless "else".
> + {
> + __clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
setup_clear_cpu_cap() if you use boot_cpu_has() above.
> + return;
> + }
> +
> + socket = cpu_to_socket(cpu);
> + info = socket_info + socket;
> + if ( info->feat_mask )
> + return;
> +
> + INIT_LIST_HEAD(&info->feat_list);
> + spin_lock_init(&info->ref_lock);
> +
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> + if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> + {
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> +
> + feat = feat_l3_cat;
> + /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
> + feat_l3_cat = NULL;
I don't think the comment is very useful: You've consumed the object,
so you simply should not leave a dangling pointer (or else you'd risk
multiple use).
> static void psr_cpu_init(void)
> {
> + if ( socket_info )
> + cpu_init_work();
> +
> psr_assoc_init();
> }
>
> static void psr_cpu_fini(unsigned int cpu)
> {
> + if ( socket_info )
> + cpu_fini_work(cpu);
> return;
> }
Is it really useful to use another layer of helper functions here?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 05/24] x86: refactor psr: implement Domain init/free and schedule flows.
2017-02-15 8:49 ` [PATCH v8 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
2017-02-26 17:41 ` Wei Liu
@ 2017-03-08 15:04 ` Jan Beulich
1 sibling, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-08 15:04 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> @@ -362,11 +370,33 @@ void psr_free_rmid(struct domain *d)
> d->arch.psr_rmid = 0;
> }
>
> +static inline unsigned int get_max_cos_max(const struct psr_socket_info *info)
While I see that the immediately following function (in context) is
inline too, please don't add such outside of headers unless the
function really wants to be inlined. Let the compiler chose what's
best in the common case. Since you also modify the following
function, dropping the inline there seems warranted too.
> @@ -408,14 +450,32 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> return 0;
> }
>
> +/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> +static void psr_free_cos(struct domain *d)
> +{
> + if( !d->arch.psr_cos_ids )
> + return;
Pointless conditional.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 06/24] x86: refactor psr: implement get hw info flow.
2017-02-15 8:49 ` [PATCH v8 06/24] x86: refactor psr: implement get hw info flow Yi Sun
2017-02-26 17:41 ` Wei Liu
2017-02-28 12:34 ` Roger Pau Monné
@ 2017-03-08 15:15 ` Jan Beulich
2017-03-10 1:43 ` Yi Sun
2 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-08 15:15 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -84,6 +84,7 @@ enum psr_feat_type {
> PSR_SOCKET_L3_CAT = 0,
> PSR_SOCKET_L3_CDP,
> PSR_SOCKET_L2_CAT,
> + PSR_SOCKET_UNKNOWN = 0xFFFF,
Any reason to use this value, instead of just the next sequential one?
> @@ -182,6 +186,24 @@ static void free_feature(struct psr_socket_info *info)
> }
> }
>
> +static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> +{
> + enum psr_feat_type feat_type;
> +
> + /* Judge if feature is enabled. */
> + switch ( type )
> + {
> + case PSR_CBM_TYPE_L3:
> + feat_type = PSR_SOCKET_L3_CAT;
> + break;
> + default:
> + feat_type = PSR_SOCKET_UNKNOWN;
> + break;
> + }
> +
> + return feat_type;
> +}
The comment ahead of the switch() doesn't seem to describe what's
being done - there certainly is no check here whether anything is
enabled or disabled.
> @@ -225,8 +247,22 @@ static unsigned int l3_cat_get_cos_max(const struct
> feat_node *feat)
> return feat->info.l3_cat_info.cos_max;
> }
>
> +static bool l3_cat_get_feat_info(const struct feat_node *feat,
> + uint32_t data[], unsigned int array_len)
> +{
> + if ( !data || 3 > array_len )
I think the 3 here was picked upon already: This check does not
guarantee there's no array overrun ...
> + return false;
> +
> + data[CBM_LEN] = feat->info.l3_cat_info.cbm_len;
> + data[COS_MAX] = feat->info.l3_cat_info.cos_max;
> + data[PSR_FLAG] = 0;
... anywhere here. For that you'd need a *_MAX- or *_NUM-type
constant (defined next to the array index constants).
> --- a/xen/arch/x86/sysctl.c
> +++ b/xen/arch/x86/sysctl.c
> @@ -176,15 +176,19 @@ long arch_do_sysctl(
> switch ( sysctl->u.psr_cat_op.cmd )
> {
> case XEN_SYSCTL_PSR_CAT_get_l3_info:
> - ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
> - &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
> - &sysctl->u.psr_cat_op.u.l3_info.cos_max,
> - &sysctl->u.psr_cat_op.u.l3_info.flags);
> + {
> + uint32_t data[3];
> + ret = psr_get_info(sysctl->u.psr_cat_op.target,
> + PSR_CBM_TYPE_L3, data, 3);
> +
> + sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
> + sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
> + sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
>
> if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
> ret = -EFAULT;
> break;
> -
> + }
Please retain the blank line (after the } ).
> --- a/xen/include/asm-x86/psr.h
> +++ b/xen/include/asm-x86/psr.h
> @@ -19,19 +19,24 @@
> #include <xen/types.h>
>
> /* CAT cpuid level */
> -#define PSR_CPUID_LEVEL_CAT 0x10
> +#define PSR_CPUID_LEVEL_CAT 0x10
>
> /* Resource Type Enumeration */
> -#define PSR_RESOURCE_TYPE_L3 0x2
> +#define PSR_RESOURCE_TYPE_L3 0x2
>
> /* L3 Monitoring Features */
> -#define PSR_CMT_L3_OCCUPANCY 0x1
> +#define PSR_CMT_L3_OCCUPANCY 0x1
>
> /* CDP Capability */
> -#define PSR_CAT_CDP_CAPABILITY (1u << 2)
> +#define PSR_CAT_CDP_CAPABILITY (1u << 2)
>
> /* L3 CDP Enable bit*/
> -#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
> +#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
Are all these adjustments really needed here?
> +/* Used by psr_get_info() */
> +#define CBM_LEN 0
> +#define COS_MAX 1
> +#define PSR_FLAG 2
Neither comment nor names are helpful to understand the purpose of
these constants. How about PSR_INFO_IDX_* or some such?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 07/24] x86: refactor psr: implement get value flow.
2017-02-15 8:49 ` [PATCH v8 07/24] x86: refactor psr: implement get value flow Yi Sun
2017-02-28 12:44 ` Roger Pau Monné
@ 2017-03-08 15:35 ` Jan Beulich
2017-03-10 1:50 ` Yi Sun
1 sibling, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-08 15:35 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> @@ -260,9 +263,22 @@ static bool l3_cat_get_feat_info(const struct feat_node *feat,
> return true;
> }
>
> +static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
> + enum cbm_type type, uint64_t *val)
> +{
> + if ( cos > feat->info.l3_cat_info.cos_max )
> + /* Use default value. */
> + cos = 0;
> +
> + *val = feat->cos_reg_val[cos];
> +
> + return true;
This one never failing I wonder whether the same will apply to the
later ones. If so, there's little point in returning a boolean here, but
instead you could return the value instead of using indirection.
> static void __init parse_psr_bool(char *s, char *value, char *feature,
> @@ -482,12 +498,14 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
> return socket_info + socket;
> }
>
> -int psr_get_info(unsigned int socket, enum cbm_type type,
> - uint32_t data[], unsigned int array_len)
> +static int psr_get(unsigned int socket, enum cbm_type type,
The immediately preceding patch introduced thus function, and
now you're changing its name. Please give it the intended final
name right away.
> + uint32_t data[], unsigned int array_len,
> + struct domain *d, uint64_t *val)
const struct domain *, but I'm not even sure that's an appropriate
parameter here:
> @@ -498,6 +516,15 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> if ( feat->feature != feat_type )
> continue;
>
> + if ( d )
> + {
> + cos = d->arch.psr_cos_ids[socket];
You could equally well pass a more constrained pointer, like
psr_cos_ids[] on its own. But of course much depends on whether
you'll need d for other things in this function in later patches.
> + if ( feat->ops.get_val(feat, cos, type, val) )
> + return 0;
> + else
> + break;
> + }
> +
> if ( feat->ops.get_feat_info(feat, data, array_len) )
> return 0;
> else
Looking at the context here - is it really a good idea to overload the
function in this way, rather than creating a second one? Your
only complicating the live of the callers, as can be seen e.g. ...
> @@ -507,10 +534,16 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> return -ENOENT;
> }
>
> -int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> - uint64_t *cbm, enum cbm_type type)
> +int psr_get_info(unsigned int socket, enum cbm_type type,
> + uint32_t data[], unsigned int array_len)
> +{
> + return psr_get(socket, type, data, array_len, NULL, NULL);
... here and ...
> +}
> +
> +int psr_get_val(struct domain *d, unsigned int socket,
> + uint64_t *val, enum cbm_type type)
> {
> - return 0;
> + return psr_get(socket, type, NULL, 0, d, val);
> }
... here (it is a bad sign that both pass NULL on either side).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-02-15 8:49 ` [PATCH v8 08/24] x86: refactor psr: set value: implement framework Yi Sun
2017-02-26 17:41 ` Wei Liu
2017-02-28 13:58 ` Roger Pau Monné
@ 2017-03-08 16:07 ` Jan Beulich
2017-03-10 2:54 ` Yi Sun
2017-03-10 7:46 ` Yi Sun
2 siblings, 2 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-08 16:07 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> As set value flow is the most complicated one in psr, it will be
> divided to some patches to make things clearer. This patch
> implements the set value framework to show a whole picture firstly.
>
> It also changes domctl interface to make it more general.
>
> To make the set value flow be general and can support multiple features
> at same time, it includes below steps:
> 1. Get COS ID of current domain using.
What is the "using" here supposed to mean?
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -546,18 +546,214 @@ int psr_get_val(struct domain *d, unsigned int socket,
> return psr_get(socket, type, NULL, 0, d, val);
> }
>
> -int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> - uint64_t cbm, enum cbm_type type)
> +/* Set value functions */
> +static unsigned int get_cos_num(const struct psr_socket_info *info)
> {
> return 0;
> }
>
> +static int assemble_val_array(uint64_t *val,
> + uint32_t array_len,
> + const struct psr_socket_info *info,
> + unsigned int old_cos)
> +{
> + return -EINVAL;
> +}
> +
> +static int set_new_val_to_array(uint64_t *val,
insert_new_val() ? And when talking about arrays, as indicated
before, please use [] notation instead of pointers. This is
particularly relevant when the function name suggests that it
would be "val" which gets inserted, but aiui it is really ...
> + uint32_t array_len,
> + const struct psr_socket_info *info,
> + enum psr_feat_type feat_type,
> + enum cbm_type type,
> + uint64_t m)
... "m". Therefore please also consider better naming of parameters.
> +static int write_psr_msr(unsigned int socket, unsigned int cos,
> + const uint64_t *val)
> +{
> + return -ENOENT;
> +}
Is this function intended you write just one MSR, or potentially many?
In the latter case the name would perhaps better be "write_psr_msrs()".
> +int psr_set_val(struct domain *d, unsigned int socket,
> + uint64_t val, enum cbm_type type)
> +{
> + unsigned int old_cos;
> + int cos, ret;
> + unsigned int *ref;
> + uint64_t *val_array;
> + struct psr_socket_info *info = get_socket_info(socket);
> + uint32_t array_len;
> + enum psr_feat_type feat_type;
> +
> + if ( IS_ERR(info) )
> + return PTR_ERR(info);
> +
> + feat_type = psr_cbm_type_to_feat_type(type);
> + if ( !test_bit(feat_type, &info->feat_mask) )
> + return -ENOENT;
> +
> + /*
> + * Step 0:
> + * old_cos means the COS ID current domain is using. By default, it is 0.
> + *
> + * For every COS ID, there is a reference count to record how many domains
> + * are using the COS register corresponding to this COS ID.
> + * - If ref[old_cos] is 0, that means this COS is not used by any domain.
> + * - If ref[old_cos] is 1, that means this COS is only used by current
> + * domain.
> + * - If ref[old_cos] is more than 1, that mean multiple domains are using
> + * this COS.
> + */
> + old_cos = d->arch.psr_cos_ids[socket];
> + if ( old_cos > MAX_COS_REG_CNT )
> + return -EOVERFLOW;
Doesn't this need to be >= ? And isn't this happening an indication
of a bug, i.e. shouldn't there be an ASSERT_UNREACHABLE() ahead
of the return?
> + ref = info->cos_ref;
> +
> + /*
> + * Step 1:
> + * Assemle a value array to store all featues cos_reg_val[old_cos].
Assemble ... features ...
> + * And, set the input val into array according to the feature's
> + * position in array.
> + */
> + array_len = get_cos_num(info);
> + val_array = xzalloc_array(uint64_t, array_len);
> + if ( !val_array )
> + return -ENOMEM;
> +
> + if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 )
> + {
> + xfree(val_array);
> + return ret;
> + }
> +
> + if ( (ret = set_new_val_to_array(val_array, array_len, info,
> + feat_type, type, val)) != 0 )
> + {
> + xfree(val_array);
> + return ret;
> + }
> +
> + /*
> + * Lock here to make sure the ref is not changed during find and
> + * write process.
> + */
> + spin_lock(&info->ref_lock);
Once again I don't think the comment is very useful - what you say
is the ordinary purpose of acquiring a lock.
> + /*
> + * Step 2:
> + * Try to find if there is already a COS ID on which all features' values
> + * are same as the array. Then, we can reuse this COS ID.
> + */
> + cos = find_cos(val_array, array_len, feat_type, info);
> + if ( cos >= 0 )
> + {
> + if ( cos == old_cos )
> + {
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return 0;
> + }
You could save a level of indentation if you inverted the outer if()'s
condition and made the code above "else if".
> + }
> + else
> + {
> + /*
> + * Step 3:
> + * If fail to find, we need allocate a new COS ID.
> + * If multiple domains are using same COS ID, its ref is more
> + * than 1. That means we cannot free this COS to make current domain
> + * use it. Because other domains are using the value saved in the COS.
> + * Unless the ref is changed to 1 (mean only current domain is using
> + * it), we cannot allocate the COS ID to current domain.
I think I had been confused by this already before, and I continue to
be: How could ref be "changed to 1" here, and then have said
meaning? If you refer to the value after a possible decrement, the
value then being 1 means there is another domain using it. Hence ...
> + * So, only the COS ID which ref is 1 or 0 can be allocated.
... I think this is not generally correct either: A COS with ref 1 can
only be re-used it that's old_cos. In all other cases only ref 0 ones
are candidates. But anyway I think the comment belongs into the
function, which would then allow for seeing it be added along with
the actual code, making it possible to check that both match up.
> + */
> + cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
> + if ( cos < 0 )
> + {
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return cos;
> + }
> +
> + /*
> + * Step 4:
> + * Write all features MSRs according to the COS ID.
> + */
> + ret = write_psr_msr(socket, cos, val_array);
> + if ( ret )
> + {
> + spin_unlock(&info->ref_lock);
> + xfree(val_array);
> + return ret;
> + }
These recurring error paths could certainly do with folding.
> + }
> +
> + /*
> + * Step 5:
> + * Update ref according to COS ID.
> + */
> + ref[cos]++;
> + ASSERT(ref[cos] || cos == 0);
ASSERT(!cos || ref[cos]);
ASSERT(!old_cos || ref[old_cos]);
> + ref[old_cos]--;
> + spin_unlock(&info->ref_lock);
> +
> + /*
> + * Step 6:
> + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> + * which COS the domain is using on the socket. One domain can only use
> + * one COS ID at same time on each socket.
> + */
> + d->arch.psr_cos_ids[socket] = cos;
So the domain has not been paused, i.e. some of its vCPU-s may
be running on other pCPU-s (including ones on the socket in
question). How come it is safe to update this value here?
> /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> static void psr_free_cos(struct domain *d)
> {
> - if( !d->arch.psr_cos_ids )
> + unsigned int socket, cos;
> +
> + if ( !d->arch.psr_cos_ids )
> return;
As in an earlier patch I've asked for this check to be removed, I
think you will need to add a check on socket_info to be non-
NULL somewhere in this function.
> + /* Domain is free so its cos_ref should be decreased. */
"Domain is free" ? DYM "is being destroyed"?
> + for ( socket = 0; socket < nr_sockets; socket++ )
> + {
> + struct psr_socket_info *info;
> +
> + /* cos 0 is default one which does not need be handled. */
> + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
> + continue;
> +
> + /*
> + * If domain uses other cos ids, all corresponding refs must have been
> + * increased 1 for this domain. So, we need decrease them.
> + */
> + info = socket_info + socket;
> + ASSERT(info->cos_ref[cos] || cos == 0);
> + spin_lock(&info->ref_lock);
> + info->cos_ref[cos]--;
> + spin_unlock(&info->ref_lock);
The ASSERT() is useful only inside the locked region.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-02-15 8:49 ` [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
2017-02-26 17:43 ` Wei Liu
@ 2017-03-08 16:54 ` Jan Beulich
2017-03-10 3:21 ` Yi Sun
1 sibling, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-08 16:54 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -119,6 +119,32 @@ struct feat_ops {
> /* get_val is used to get feature COS register value. */
> bool (*get_val)(const struct feat_node *feat, unsigned int cos,
> enum cbm_type type, uint64_t *val);
> + /*
> + * get_cos_num is used to get the COS registers amount used by the
> + * feature for one setting, e.g. CDP uses 2 COSs but CAT uses 1.
> + */
> + unsigned int (*get_cos_num)(const struct feat_node *feat);
Please have blank lines ahead of every separating comment. (Of
course this then may need to start already in earlier patches.)
> + /*
> + * get_old_val and set_new_val are a pair of functions called in order.
> + * The caller will traverse all features in the list and call both
> + * functions for every feature to do below two things:
> + * 1. get old_cos register value of all supported features and
> + * 2. set the new value for the feature.
This is misleading, I think: I don't think a new value is being set for
every feature. This should be worded less ambiguously.
> @@ -207,6 +233,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> return feat_type;
> }
>
> +static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
> +{
> + unsigned int first_bit, zero_bit;
> +
> + /* Set bits should only in the range of [0, cbm_len). */
> + if ( cbm & (~0ull << cbm_len) )
This will not do what you intend for cbm_len == 64.
> +static int l3_cat_get_old_val(uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int old_cos)
> +{
> + if ( old_cos > feat->info.l3_cat_info.cos_max )
Afaics this condition is the only L3 CAT specific thing in this function.
Should more of it be moved into common code? Same below for
set_new_val.
> -static int assemble_val_array(uint64_t *val,
> +static int combine_val_array(uint64_t *val,
Same comment as earlier on - please decide for a final name right
when introducing a function. In fact I'd prefer it to remain
"assemble".
> {
> - return -EINVAL;
> + const struct feat_node *feat;
> + int ret;
> + uint64_t *val_tmp = val;
I don't really see the need for this helper variable. Simply ...
> + if ( !val )
> + return -EINVAL;
> +
> + /* Get all features current values according to old_cos. */
> + list_for_each_entry(feat, &info->feat_list, list)
> + {
> + /* value getting order is same as feature list */
> + ret = feat->ops.get_old_val(val_tmp, feat, old_cos);
> + if ( ret )
> + return ret;
> +
> + val_tmp += feat->ops.get_cos_num(feat);
... use val here, after checking the return value against
array_len, and the also subtract from array_len. (I am averse
to _tmp suffixes, I'm sorry.)
Btw - for any of the later features, does their get_cos_num() ever
return other than a constant value? If not, there's no point in making
this a function call - you could simply have a numeric member in the
structure.
> @@ -567,7 +678,37 @@ static int set_new_val_to_array(uint64_t *val,
> enum cbm_type type,
> uint64_t m)
> {
> - return -EINVAL;
> + const struct feat_node *feat;
> + int ret;
> + uint64_t *val_tmp = val;
> +
> + /* Set new value into array according to feature's position in array. */
> + list_for_each_entry(feat, &info->feat_list, list)
> + {
> + if ( feat->feature != feat_type )
> + {
> + val_tmp += feat->ops.get_cos_num(feat);
> + if ( val_tmp - val > array_len)
> + return -ENOSPC;
> +
> + continue;
> + }
> +
> + /*
> + * Value setting position is same as feature list.
> + * Different features may have different setting behaviors, e.g. CDP
> + * has two values (DATA/CODE) which need us to save input value to
> + * different position in the array according to type, so we have to
> + * maintain a callback function.
> + */
> + ret = feat->ops.set_new_val(val_tmp, feat, type, m);
> + if ( ret )
> + return ret;
> + else
Pointless "else" again. I think I'll try to avoid pointing this out, should
more of these appear - please simply go through yourself any remove
any such uses.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow.
2017-02-15 8:49 ` [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
2017-02-26 17:43 ` Wei Liu
@ 2017-03-08 17:03 ` Jan Beulich
2017-03-10 5:35 ` Yi Sun
1 sibling, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-08 17:03 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> Continue with patch:
???
> 'x86: refactor psr: set value: assemble features value array'
Or did you perhaps mean "continue from"?
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -145,6 +145,19 @@ struct feat_ops {
> const struct feat_node *feat,
> enum cbm_type type,
> uint64_t m);
> + /*
> + * compare_val is used in set value process to compare if the
> + * input value array can match all the features' COS registers values
> + * according to input cos id.
> + *
> + * The return value is the amount of entries to skip in the value array
> + * or error.
> + * 1 - one entry in value array.
> + * 2 - two entries in value array, e.g. CDP uses two entries.
Isn't this get_cos_num()'s result again? Or, looking at the function
below, is the comment simply stale?
> @@ -356,6 +369,34 @@ static int l3_cat_set_new_val(uint64_t val[],
> return 0;
> }
>
> +static int l3_cat_compare_val(const uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int cos, bool *found)
> +{
> + uint64_t l3_def_cbm;
> +
> + l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
Please only calculate the value on the path you need it. Also this
will again degenerate of cbm_len == 64.
> + /*
> + * Different features' cos_max are different. If cos id of the feature
> + * being set exceeds other feature's cos_max, the val of other feature
> + * must be default value. HW supports such case.
> + */
> + if ( cos > feat->info.l3_cat_info.cos_max )
> + {
> + if ( val[0] != l3_def_cbm )
> + {
> + *found = false;
> + return -ENOENT;
What is the difference between this "not found" and ...
> + }
> + *found = true;
> + }
> + else
> + *found = (val[0] == feat->cos_reg_val[cos]);
> +
> + return 0;
... the possible one here? I.e. why once -ENOENT and once 0 as
return value?
> @@ -715,6 +757,57 @@ static int find_cos(const uint64_t *val, uint32_t array_len,
> enum psr_feat_type feat_type,
> const struct psr_socket_info *info)
> {
> + unsigned int cos;
> + const unsigned int *ref = info->cos_ref;
> + const struct feat_node *feat;
> + const uint64_t *val_tmp = val;
> + int ret;
> + bool found = false;
> + unsigned int cos_max = 0;
> +
> + /* cos_max is the one of the feature which is being set. */
> + list_for_each_entry(feat, &info->feat_list, list)
> + {
> + if ( feat->feature != feat_type )
> + continue;
> +
> + cos_max = feat->ops.get_cos_max(feat);
> + if ( cos_max > 0 )
What's the purpose of this check? I.e. why do you continue the loop
in case you find zero? You won't find another node with the same
feat_type, will you?
> + break;
> + }
> +
> + for ( cos = 0; cos <= cos_max; cos++ )
> + {
> + if ( cos && !ref[cos] )
> + continue;
> +
> + /* Not found, need find again from beginning. */
You may not even have looked yet, so how can you say "not found"?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow.
2017-02-15 8:49 ` [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
2017-02-26 17:43 ` Wei Liu
@ 2017-03-09 14:10 ` Jan Beulich
2017-03-10 5:40 ` Yi Sun
1 sibling, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-09 14:10 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> @@ -397,6 +408,25 @@ static int l3_cat_compare_val(const uint64_t val[],
> return 0;
> }
>
> +static bool l3_cat_fits_cos_max(const uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int cos)
> +{
> + uint64_t l3_def_cbm;
> +
> + l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
Seeing this pattern recur I wonder whether this also wouldn't be
something that could be stored once in a generic manner, avoiding
the need for this per-feature callback (cos_max should be common
to all features too - not the values, but the abstract notion - so
perhaps get_cos_max() isn't needed as a callback either).
> static int pick_avail_cos(const struct psr_socket_info *info,
> const uint64_t *val, uint32_t array_len,
> unsigned int old_cos,
> enum psr_feat_type feat_type)
> {
> + unsigned int cos;
> + unsigned int cos_max = 0;
> + const struct feat_node *feat;
> + const unsigned int *ref = info->cos_ref;
> +
> + /*
> + * cos_max is the one of the feature which is being set.
> + */
This is a single line comment.
> + list_for_each_entry(feat, &info->feat_list, list)
> + {
> + if ( feat->feature != feat_type )
> + continue;
> +
> + cos_max = feat->ops.get_cos_max(feat);
> + if ( cos_max > 0 )
> + break;
> + }
I think I've seen this a number of times by now - please make this a
helper function, or store the result (which isn't going to change
afaict).
Other than these comments given to earlier patches apply here too,
as I think is obvious.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 13/24] x86: refactor psr: implement CPU init and free flow for CDP.
2017-02-15 8:49 ` [PATCH v8 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
2017-02-28 14:52 ` Roger Pau Monné
@ 2017-03-09 14:53 ` Jan Beulich
2017-03-10 5:50 ` Yi Sun
1 sibling, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-09 14:53 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -97,6 +97,7 @@ struct psr_cat_hw_info {
> struct feat_hw_info {
> union {
> struct psr_cat_hw_info l3_cat_info;
> + struct psr_cat_hw_info l3_cdp_info;
Two union members of the same type? What's the union good for
then? (I've peeked ahead, and L2 CAT adds yet another one. A
strong sign that you've gone too far with what needs to be per-
feature vs what can be common.)
> @@ -195,6 +196,22 @@ struct feat_node {
> struct list_head list;
> };
>
> +/*
> + * get_data - get DATA COS register value from input COS ID.
> + * @feat: the feature list entry.
> + * @cos: the COS ID.
> + */
> +#define get_cdp_data(feat, cos) \
> + ( feat->cos_reg_val[cos * 2] )
Stray blanks inside parentheses. Macro parameters need to be
parenthesized.
> +/*
> + * get_cdp_code - get CODE COS register value from input COS ID.
> + * @feat: the feature list entry.
> + * @cos: the COS ID.
> + */
> +#define get_cdp_code(feat, cos) \
> + ( feat->cos_reg_val[cos * 2 + 1] )
Same here.
> @@ -1213,12 +1288,21 @@ static void cpu_init_work(void)
> {
> cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
>
> - feat = feat_l3_cat;
> - /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
> - feat_l3_cat = NULL;
> - feat->ops = l3_cat_ops;
> -
> - l3_cat_init_feature(regs, feat, info);
> + if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
> + !test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) )
> + {
> + feat = feat_l3_cdp;
> + /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
> + feat_l3_cdp = NULL;
> + feat->ops = l3_cdp_ops;
> + l3_cdp_init_feature(regs, feat, info);
> + } else {
I think someone else has already pointed out the style issue here, so
just in case ...
> @@ -1267,6 +1351,14 @@ static int psr_cpu_prepare(void)
> (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
> return -ENOMEM;
>
> + if ( feat_l3_cdp == NULL &&
> + (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
> + {
> + xfree(feat_l3_cat);
> + feat_l3_cat = NULL;
Why the freeing? We've decided to allow for one node to be kept,
so no reason to free it on the error path here.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow.
2017-02-15 8:49 ` [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
2017-02-28 15:15 ` Roger Pau Monné
@ 2017-03-09 15:04 ` Jan Beulich
2017-03-10 5:52 ` Yi Sun
1 sibling, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-09 15:04 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> @@ -684,6 +686,53 @@ struct feat_ops l3_cdp_ops = {
> .write_msr = l3_cdp_write_msr,
> };
>
> +/* L2 CAT callback functions implementation. */
> +static void l2_cat_init_feature(struct cpuid_leaf regs,
const struct cpuid_leaf * (and of course I should have noticed this
already in earlier patches, but I didn't)
> + struct feat_node *feat,
> + struct psr_socket_info *info)
> +{
> + struct psr_cat_hw_info l2_cat = { };
I realize Konrad did ask for the initializer here, but it's really pointless
to have such when both structure fields get very obviously set right
below. If you want an initializer, just set the fields to their final values.
> + unsigned int socket;
> +
> + /* No valid values so do not enable the feature. */
> + if ( !regs.a || !regs.d )
> + return;
> +
> + l2_cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
> + l2_cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
> +
> + /* cos=0 is reserved as default cbm(all ones). */
> + feat->cos_reg_val[0] = (1ull << l2_cat.cbm_len) - 1;
> +
> + feat->feature = PSR_SOCKET_L2_CAT;
> + ASSERT(!test_bit(PSR_SOCKET_L2_CAT, &info->feat_mask));
> + __set_bit(PSR_SOCKET_L2_CAT, &info->feat_mask);
> +
> + feat->info.l2_cat_info = l2_cat;
> +
> + info->nr_feat++;
Is this field really used for anything? I don't recall seeing other than
these increments.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow.
2017-02-15 8:49 ` [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
2017-02-28 15:18 ` Roger Pau Monné
@ 2017-03-09 15:13 ` Jan Beulich
2017-03-10 5:57 ` Yi Sun
1 sibling, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-09 15:13 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> +static bool l2_cat_get_feat_info(const struct feat_node *feat,
> + uint32_t data[], uint32_t array_len)
> +{
> + if ( !data || 2 > array_len )
> + return false;
> +
> + data[CBM_LEN] = feat->info.l2_cat_info.cbm_len;
> + data[COS_MAX] = feat->info.l2_cat_info.cos_max;
What about PSR_FLAG? Which puts under question the
array_len check at the start of this and its sibling functions: If
more array entries are provided, they'd all be left uninitialized
without the caller having a way to know. Coming back to the
data structures being the same for all features, there should
presumably be a structure instead of an array be used for
communication here, and ...
> @@ -754,6 +755,11 @@ struct xen_sysctl_psr_cat_op {
> #define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
> uint32_t flags; /* OUT: CAT flags */
> } l3_info;
> +
> + struct {
> + uint32_t cbm_len; /* OUT: CBM length */
> + uint32_t cos_max; /* OUT: Maximum COS */
> + } l2_info;
... this should use the same structure as l3_info (flags being
set to zero until a use arises). This would then also allow for
more code to be shared instead of duplicated.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
2017-03-08 14:56 ` Jan Beulich
@ 2017-03-10 1:32 ` Yi Sun
2017-03-10 8:56 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-10 1:32 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-08 07:56:54, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
[...]
> > -static int psr_cpu_prepare(unsigned int cpu)
> > +static void cpu_init_work(void)
> > +{
> > + struct psr_socket_info *info;
> > + unsigned int socket;
> > + unsigned int cpu = smp_processor_id();
> > + struct feat_node *feat;
> > + struct cpuid_leaf regs = { .a = 0, .b = 0, .c = 0, .d = 0 };
>
> I don't see you needing an initializer here at all, but if you really
> want one for some reason, the same effect can be had with just
> {}.
>
Konrad suggested me to initialize it like this in v5 patch review comments.
I think he has experienced some strange issues when he forgot to set _all_
the entries a structure allocated on the stack.
> > + if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
>
> Do you really mean to not universally check the global (boot CPU)
> flag? I.e. possibly differing behavior on different CPUs?
>
Yes, different sockets may have different configurations. E.g. sockt 0 has
PQE support but socket 1 does not. Per my info, there is only one boot cpu
no matter how many sockets there are.
> > + return;
> > + else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
>
> Pointless "else".
>
Thanks, will remove it.
> > + {
> > + __clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
>
> setup_clear_cpu_cap() if you use boot_cpu_has() above.
>
> > + return;
> > + }
> > +
> > + socket = cpu_to_socket(cpu);
> > + info = socket_info + socket;
> > + if ( info->feat_mask )
> > + return;
> > +
> > + INIT_LIST_HEAD(&info->feat_list);
> > + spin_lock_init(&info->ref_lock);
> > +
> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> > + if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> > + {
> > + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> > +
> > + feat = feat_l3_cat;
> > + /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
> > + feat_l3_cat = NULL;
>
> I don't think the comment is very useful: You've consumed the object,
> so you simply should not leave a dangling pointer (or else you'd risk
> multiple use).
>
Will remove it.
> > static void psr_cpu_init(void)
> > {
> > + if ( socket_info )
> > + cpu_init_work();
> > +
> > psr_assoc_init();
> > }
> >
> > static void psr_cpu_fini(unsigned int cpu)
> > {
> > + if ( socket_info )
> > + cpu_fini_work(cpu);
> > return;
> > }
>
> Is it really useful to use another layer of helper functions here?
>
The reason we define 'cpu_fini_work' is to match 'cpu_init_work'. If we move
codes of 'cpu_init_work' into 'psr_cpu_init', the codes look messy. That is
the reason we define 'cpu_init_work'. Do you think if it is acceptable to you?
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 06/24] x86: refactor psr: implement get hw info flow.
2017-03-08 15:15 ` Jan Beulich
@ 2017-03-10 1:43 ` Yi Sun
2017-03-10 8:57 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-10 1:43 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-08 08:15:33, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -84,6 +84,7 @@ enum psr_feat_type {
> > PSR_SOCKET_L3_CAT = 0,
> > PSR_SOCKET_L3_CDP,
> > PSR_SOCKET_L2_CAT,
> > + PSR_SOCKET_UNKNOWN = 0xFFFF,
>
> Any reason to use this value, instead of just the next sequential one?
>
This is an error value used below, in 'psr_cbm_type_to_feat_type'. To make it
explicitly different, I assign a big value to it.
> > @@ -182,6 +186,24 @@ static void free_feature(struct psr_socket_info *info)
> > }
> > }
> >
> > +static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> > +{
> > + enum psr_feat_type feat_type;
> > +
> > + /* Judge if feature is enabled. */
> > + switch ( type )
> > + {
> > + case PSR_CBM_TYPE_L3:
> > + feat_type = PSR_SOCKET_L3_CAT;
> > + break;
> > + default:
> > + feat_type = PSR_SOCKET_UNKNOWN;
> > + break;
> > + }
> > +
> > + return feat_type;
> > +}
>
> The comment ahead of the switch() doesn't seem to describe what's
> being done - there certainly is no check here whether anything is
> enabled or disabled.
>
Sorry for that, will remove the comment.
> > @@ -225,8 +247,22 @@ static unsigned int l3_cat_get_cos_max(const struct
> > feat_node *feat)
> > return feat->info.l3_cat_info.cos_max;
> > }
> >
> > +static bool l3_cat_get_feat_info(const struct feat_node *feat,
> > + uint32_t data[], unsigned int array_len)
> > +{
> > + if ( !data || 3 > array_len )
>
> I think the 3 here was picked upon already: This check does not
> guarantee there's no array overrun ...
>
Yes, Roger has suggested to change it to 'array_len != PSR_INFO_SIZE'.
> > + return false;
> > +
> > + data[CBM_LEN] = feat->info.l3_cat_info.cbm_len;
> > + data[COS_MAX] = feat->info.l3_cat_info.cos_max;
> > + data[PSR_FLAG] = 0;
>
> ... anywhere here. For that you'd need a *_MAX- or *_NUM-type
> constant (defined next to the array index constants).
>
This is defined in next version.
[...]
> > --- a/xen/include/asm-x86/psr.h
> > +++ b/xen/include/asm-x86/psr.h
> > @@ -19,19 +19,24 @@
> > #include <xen/types.h>
> >
> > /* CAT cpuid level */
> > -#define PSR_CPUID_LEVEL_CAT 0x10
> > +#define PSR_CPUID_LEVEL_CAT 0x10
> >
> > /* Resource Type Enumeration */
> > -#define PSR_RESOURCE_TYPE_L3 0x2
> > +#define PSR_RESOURCE_TYPE_L3 0x2
> >
> > /* L3 Monitoring Features */
> > -#define PSR_CMT_L3_OCCUPANCY 0x1
> > +#define PSR_CMT_L3_OCCUPANCY 0x1
> >
> > /* CDP Capability */
> > -#define PSR_CAT_CDP_CAPABILITY (1u << 2)
> > +#define PSR_CAT_CDP_CAPABILITY (1u << 2)
> >
> > /* L3 CDP Enable bit*/
> > -#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
> > +#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
>
> Are all these adjustments really needed here?
>
Per Wei's suggestion to make codes neat.
> > +/* Used by psr_get_info() */
> > +#define CBM_LEN 0
> > +#define COS_MAX 1
> > +#define PSR_FLAG 2
>
> Neither comment nor names are helpful to understand the purpose of
> these constants. How about PSR_INFO_IDX_* or some such?
>
Ok will do it in next version.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 07/24] x86: refactor psr: implement get value flow.
2017-03-08 15:35 ` Jan Beulich
@ 2017-03-10 1:50 ` Yi Sun
2017-03-10 9:05 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-10 1:50 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel
On 17-03-08 08:35:53, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> > @@ -260,9 +263,22 @@ static bool l3_cat_get_feat_info(const struct feat_node *feat,
> > return true;
> > }
> >
> > +static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
> > + enum cbm_type type, uint64_t *val)
> > +{
> > + if ( cos > feat->info.l3_cat_info.cos_max )
> > + /* Use default value. */
> > + cos = 0;
> > +
> > + *val = feat->cos_reg_val[cos];
> > +
> > + return true;
>
> This one never failing I wonder whether the same will apply to the
> later ones. If so, there's little point in returning a boolean here, but
> instead you could return the value instead of using indirection.
>
I have modified this function to 'void' in next version.
> > static void __init parse_psr_bool(char *s, char *value, char *feature,
> > @@ -482,12 +498,14 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
> > return socket_info + socket;
> > }
> >
> > -int psr_get_info(unsigned int socket, enum cbm_type type,
> > - uint32_t data[], unsigned int array_len)
> > +static int psr_get(unsigned int socket, enum cbm_type type,
>
> The immediately preceding patch introduced thus function, and
> now you're changing its name. Please give it the intended final
> name right away.
>
The name is not changed. You can see below lines. Here is just to implement
a helper function which is called by 'psr_get_info'.
> > + uint32_t data[], unsigned int array_len,
> > + struct domain *d, uint64_t *val)
>
> const struct domain *, but I'm not even sure that's an appropriate
> parameter here:
>
> > @@ -498,6 +516,15 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> > if ( feat->feature != feat_type )
> > continue;
> >
> > + if ( d )
> > + {
> > + cos = d->arch.psr_cos_ids[socket];
>
> You could equally well pass a more constrained pointer, like
> psr_cos_ids[] on its own. But of course much depends on whether
> you'll need d for other things in this function in later patches.
>
Ok, thanks! Will consider the parameter.
> > + if ( feat->ops.get_val(feat, cos, type, val) )
> > + return 0;
> > + else
> > + break;
> > + }
> > +
> > if ( feat->ops.get_feat_info(feat, data, array_len) )
> > return 0;
> > else
>
> Looking at the context here - is it really a good idea to overload the
> function in this way, rather than creating a second one? Your
> only complicating the live of the callers, as can be seen e.g. ...
>
These codes were separated into two functions before, 'psr_get_info' and
'psr_get_val'. But there are some common codes. So, Konrad suggested me
to create a helper function to reduce redundant codes.
> > @@ -507,10 +534,16 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> > return -ENOENT;
> > }
> >
> > -int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> > - uint64_t *cbm, enum cbm_type type)
> > +int psr_get_info(unsigned int socket, enum cbm_type type,
> > + uint32_t data[], unsigned int array_len)
> > +{
> > + return psr_get(socket, type, data, array_len, NULL, NULL);
>
> ... here and ...
>
> > +}
> > +
> > +int psr_get_val(struct domain *d, unsigned int socket,
> > + uint64_t *val, enum cbm_type type)
> > {
> > - return 0;
> > + return psr_get(socket, type, NULL, 0, d, val);
> > }
>
> ... here (it is a bad sign that both pass NULL on either side).
>
Yes, these things look not good. But to keep a common helper I have to pass
all necessary parameters into it. What is your suggestion?
> Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-08 16:07 ` Jan Beulich
@ 2017-03-10 2:54 ` Yi Sun
2017-03-10 9:09 ` Jan Beulich
2017-03-10 7:46 ` Yi Sun
1 sibling, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-10 2:54 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-08 09:07:10, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> > As set value flow is the most complicated one in psr, it will be
> > divided to some patches to make things clearer. This patch
> > implements the set value framework to show a whole picture firstly.
> >
> > It also changes domctl interface to make it more general.
> >
> > To make the set value flow be general and can support multiple features
> > at same time, it includes below steps:
> > 1. Get COS ID of current domain using.
>
> What is the "using" here supposed to mean?
>
My meaning is to get the cos id that current domain is using. Sorry for this.
Will make it better.
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -546,18 +546,214 @@ int psr_get_val(struct domain *d, unsigned int socket,
> > return psr_get(socket, type, NULL, 0, d, val);
> > }
> >
> > -int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> > - uint64_t cbm, enum cbm_type type)
> > +/* Set value functions */
> > +static unsigned int get_cos_num(const struct psr_socket_info *info)
> > {
> > return 0;
> > }
> >
> > +static int assemble_val_array(uint64_t *val,
> > + uint32_t array_len,
> > + const struct psr_socket_info *info,
> > + unsigned int old_cos)
> > +{
> > + return -EINVAL;
> > +}
> > +
> > +static int set_new_val_to_array(uint64_t *val,
>
> insert_new_val() ? And when talking about arrays, as indicated
> before, please use [] notation instead of pointers. This is
> particularly relevant when the function name suggests that it
> would be "val" which gets inserted, but aiui it is really ...
>
> > + uint32_t array_len,
> > + const struct psr_socket_info *info,
> > + enum psr_feat_type feat_type,
> > + enum cbm_type type,
> > + uint64_t m)
>
> ... "m". Therefore please also consider better naming of parameters.
>
Sure, thanks!
> > +static int write_psr_msr(unsigned int socket, unsigned int cos,
> > + const uint64_t *val)
> > +{
> > + return -ENOENT;
> > +}
>
> Is this function intended you write just one MSR, or potentially many?
> In the latter case the name would perhaps better be "write_psr_msrs()".
>
For one feature, it does set one MSR.
> > +int psr_set_val(struct domain *d, unsigned int socket,
> > + uint64_t val, enum cbm_type type)
> > +{
> > + unsigned int old_cos;
> > + int cos, ret;
> > + unsigned int *ref;
> > + uint64_t *val_array;
> > + struct psr_socket_info *info = get_socket_info(socket);
> > + uint32_t array_len;
> > + enum psr_feat_type feat_type;
> > +
> > + if ( IS_ERR(info) )
> > + return PTR_ERR(info);
> > +
> > + feat_type = psr_cbm_type_to_feat_type(type);
> > + if ( !test_bit(feat_type, &info->feat_mask) )
> > + return -ENOENT;
> > +
> > + /*
> > + * Step 0:
> > + * old_cos means the COS ID current domain is using. By default, it is 0.
> > + *
> > + * For every COS ID, there is a reference count to record how many domains
> > + * are using the COS register corresponding to this COS ID.
> > + * - If ref[old_cos] is 0, that means this COS is not used by any domain.
> > + * - If ref[old_cos] is 1, that means this COS is only used by current
> > + * domain.
> > + * - If ref[old_cos] is more than 1, that mean multiple domains are using
> > + * this COS.
> > + */
> > + old_cos = d->arch.psr_cos_ids[socket];
> > + if ( old_cos > MAX_COS_REG_CNT )
> > + return -EOVERFLOW;
>
> Doesn't this need to be >= ? And isn't this happening an indication
> of a bug, i.e. shouldn't there be an ASSERT_UNREACHABLE() ahead
> of the return?
>
Sorry. This has been corrected in next version. Thanks!
> > + ref = info->cos_ref;
> > +
> > + /*
> > + * Step 1:
> > + * Assemle a value array to store all featues cos_reg_val[old_cos].
>
> Assemble ... features ...
>
Oh, sorry.
[...]
> > + /*
> > + * Step 2:
> > + * Try to find if there is already a COS ID on which all features' values
> > + * are same as the array. Then, we can reuse this COS ID.
> > + */
> > + cos = find_cos(val_array, array_len, feat_type, info);
> > + if ( cos >= 0 )
> > + {
> > + if ( cos == old_cos )
> > + {
> > + spin_unlock(&info->ref_lock);
> > + xfree(val_array);
> > + return 0;
> > + }
>
> You could save a level of indentation if you inverted the outer if()'s
> condition and made the code above "else if".
>
Will consider it.
> > + }
> > + else
> > + {
> > + /*
> > + * Step 3:
> > + * If fail to find, we need allocate a new COS ID.
> > + * If multiple domains are using same COS ID, its ref is more
> > + * than 1. That means we cannot free this COS to make current domain
> > + * use it. Because other domains are using the value saved in the COS.
> > + * Unless the ref is changed to 1 (mean only current domain is using
> > + * it), we cannot allocate the COS ID to current domain.
>
> I think I had been confused by this already before, and I continue to
> be: How could ref be "changed to 1" here, and then have said
> meaning? If you refer to the value after a possible decrement, the
> value then being 1 means there is another domain using it. Hence ...
>
> > + * So, only the COS ID which ref is 1 or 0 can be allocated.
>
> ... I think this is not generally correct either: A COS with ref 1 can
> only be re-used it that's old_cos. In all other cases only ref 0 ones
> are candidates. But anyway I think the comment belongs into the
> function, which would then allow for seeing it be added along with
> the actual code, making it possible to check that both match up.
>
Sorry, the expression is not good. In fact, only COS ID which ref is 1 or 0
can be allocated to current domain. If old_cos is not 0 and its ref==1 means
that only current domain is using this old_cos ID. So, this old_cos ID is
certainly can be reused by current domain. Ref==0 means there is no any domain
using this COS ID. So it can be used too.
I will polish the comments.
> > + */
> > + cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
> > + if ( cos < 0 )
> > + {
> > + spin_unlock(&info->ref_lock);
> > + xfree(val_array);
> > + return cos;
> > + }
> > +
> > + /*
> > + * Step 4:
> > + * Write all features MSRs according to the COS ID.
> > + */
> > + ret = write_psr_msr(socket, cos, val_array);
> > + if ( ret )
> > + {
> > + spin_unlock(&info->ref_lock);
> > + xfree(val_array);
> > + return ret;
> > + }
>
> These recurring error paths could certainly do with folding.
>
Yes, Wei has suggested to use goto to handle them. This has been refined in
next version.
> > + }
> > +
> > + /*
> > + * Step 5:
> > + * Update ref according to COS ID.
> > + */
> > + ref[cos]++;
> > + ASSERT(ref[cos] || cos == 0);
>
> ASSERT(!cos || ref[cos]);
> ASSERT(!old_cos || ref[old_cos]);
>
Ok, thanks!
> > + ref[old_cos]--;
> > + spin_unlock(&info->ref_lock);
> > +
> > + /*
> > + * Step 6:
> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> > + * which COS the domain is using on the socket. One domain can only use
> > + * one COS ID at same time on each socket.
> > + */
> > + d->arch.psr_cos_ids[socket] = cos;
>
> So the domain has not been paused, i.e. some of its vCPU-s may
> be running on other pCPU-s (including ones on the socket in
> question). How come it is safe to update this value here?
>
This is a domctl operation. It is protected by domctl_lock which is locked in
do_domctl().
> > /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> > static void psr_free_cos(struct domain *d)
> > {
> > - if( !d->arch.psr_cos_ids )
> > + unsigned int socket, cos;
> > +
> > + if ( !d->arch.psr_cos_ids )
> > return;
>
> As in an earlier patch I've asked for this check to be removed, I
> think you will need to add a check on socket_info to be non-
> NULL somewhere in this function.
>
Ok, will do it in the loop.
> > + /* Domain is free so its cos_ref should be decreased. */
>
> "Domain is free" ? DYM "is being destroyed"?
>
Yes.
> > + for ( socket = 0; socket < nr_sockets; socket++ )
> > + {
> > + struct psr_socket_info *info;
> > +
> > + /* cos 0 is default one which does not need be handled. */
> > + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
> > + continue;
> > +
> > + /*
> > + * If domain uses other cos ids, all corresponding refs must have been
> > + * increased 1 for this domain. So, we need decrease them.
> > + */
> > + info = socket_info + socket;
> > + ASSERT(info->cos_ref[cos] || cos == 0);
> > + spin_lock(&info->ref_lock);
> > + info->cos_ref[cos]--;
> > + spin_unlock(&info->ref_lock);
>
> The ASSERT() is useful only inside the locked region.
>
Ok, thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-03-08 16:54 ` Jan Beulich
@ 2017-03-10 3:21 ` Yi Sun
2017-03-10 9:15 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-10 3:21 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-08 09:54:04, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
[...]
> > + /*
> > + * get_old_val and set_new_val are a pair of functions called in order.
> > + * The caller will traverse all features in the list and call both
> > + * functions for every feature to do below two things:
> > + * 1. get old_cos register value of all supported features and
> > + * 2. set the new value for the feature.
>
> This is misleading, I think: I don't think a new value is being set for
> every feature. This should be worded less ambiguously.
>
The expression is not accurate. I will correct this. Thanks!
> > @@ -207,6 +233,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> > return feat_type;
> > }
> >
> > +static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
> > +{
> > + unsigned int first_bit, zero_bit;
> > +
> > + /* Set bits should only in the range of [0, cbm_len). */
> > + if ( cbm & (~0ull << cbm_len) )
>
> This will not do what you intend for cbm_len == 64.
>
cbm_len is not 64. cbm_len means the CBM value length, how many bits. For L3
CAT, it may be 11 bits. For L2 CAT, it may be 8 bits.
> > +static int l3_cat_get_old_val(uint64_t val[],
> > + const struct feat_node *feat,
> > + unsigned int old_cos)
> > +{
> > + if ( old_cos > feat->info.l3_cat_info.cos_max )
>
> Afaics this condition is the only L3 CAT specific thing in this function.
> Should more of it be moved into common code? Same below for
> set_new_val.
>
Sorry, I may not understand your intention. For different features, they have
different cos_max. Do you mean I should abstract a callback function for all
features to handle this cos_max check? Thanks!
> > -static int assemble_val_array(uint64_t *val,
> > +static int combine_val_array(uint64_t *val,
>
> Same comment as earlier on - please decide for a final name right
> when introducing a function. In fact I'd prefer it to remain
> "assemble".
>
This is corrected in next version. I will change name back to assemble if you
like it.
> > {
> > - return -EINVAL;
> > + const struct feat_node *feat;
> > + int ret;
> > + uint64_t *val_tmp = val;
>
> I don't really see the need for this helper variable. Simply ...
>
> > + if ( !val )
> > + return -EINVAL;
> > +
> > + /* Get all features current values according to old_cos. */
> > + list_for_each_entry(feat, &info->feat_list, list)
> > + {
> > + /* value getting order is same as feature list */
> > + ret = feat->ops.get_old_val(val_tmp, feat, old_cos);
> > + if ( ret )
> > + return ret;
> > +
> > + val_tmp += feat->ops.get_cos_num(feat);
>
> ... use val here, after checking the return value against
> array_len, and the also subtract from array_len. (I am averse
> to _tmp suffixes, I'm sorry.)
>
Ok, no problem.
> Btw - for any of the later features, does their get_cos_num() ever
> return other than a constant value? If not, there's no point in making
> this a function call - you could simply have a numeric member in the
> structure.
>
Good idea. Thanks!
[...]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow.
2017-03-08 17:03 ` Jan Beulich
@ 2017-03-10 5:35 ` Yi Sun
2017-03-10 9:21 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-10 5:35 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-08 10:03:10, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> > Continue with patch:
>
> ???
>
> > 'x86: refactor psr: set value: assemble features value array'
>
> Or did you perhaps mean "continue from"?
>
Should be 'from'. Thanks!
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -145,6 +145,19 @@ struct feat_ops {
> > const struct feat_node *feat,
> > enum cbm_type type,
> > uint64_t m);
> > + /*
> > + * compare_val is used in set value process to compare if the
> > + * input value array can match all the features' COS registers values
> > + * according to input cos id.
> > + *
> > + * The return value is the amount of entries to skip in the value array
> > + * or error.
> > + * 1 - one entry in value array.
> > + * 2 - two entries in value array, e.g. CDP uses two entries.
>
> Isn't this get_cos_num()'s result again? Or, looking at the function
> below, is the comment simply stale?
>
Sorry, forgot to update it.
> > @@ -356,6 +369,34 @@ static int l3_cat_set_new_val(uint64_t val[],
> > return 0;
> > }
> >
> > +static int l3_cat_compare_val(const uint64_t val[],
> > + const struct feat_node *feat,
> > + unsigned int cos, bool *found)
> > +{
> > + uint64_t l3_def_cbm;
> > +
> > + l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
>
> Please only calculate the value on the path you need it. Also this
> will again degenerate of cbm_len == 64.
>
Sorry, what do you mean? I need get the default value of L3 CAT here
to check if input val equals the default one if cos exceeds cos_max.
As explanation in previous patch, cbm_len is not 64. It means how many bits
the CBM value has. E.g. L3 CAT may have 11 bits. L2 CAT may have 8 bits.
> > + /*
> > + * Different features' cos_max are different. If cos id of the feature
> > + * being set exceeds other feature's cos_max, the val of other feature
> > + * must be default value. HW supports such case.
> > + */
> > + if ( cos > feat->info.l3_cat_info.cos_max )
> > + {
> > + if ( val[0] != l3_def_cbm )
> > + {
> > + *found = false;
> > + return -ENOENT;
>
> What is the difference between this "not found" and ...
>
This is an error case. If input cos exceeds the cos_max and the input val is
not default value, that means the input parameters exceed HW ability. We should
report error back.
> > + }
> > + *found = true;
> > + }
> > + else
> > + *found = (val[0] == feat->cos_reg_val[cos]);
> > +
> > + return 0;
>
> ... the possible one here? I.e. why once -ENOENT and once 0 as
> return value?
>
0 means success. '*found' means if the val is found or not. Per Roger's
suggestion, I will refine this function to use return value to check
if it is found or not.
> > @@ -715,6 +757,57 @@ static int find_cos(const uint64_t *val, uint32_t array_len,
> > enum psr_feat_type feat_type,
> > const struct psr_socket_info *info)
> > {
> > + unsigned int cos;
> > + const unsigned int *ref = info->cos_ref;
> > + const struct feat_node *feat;
> > + const uint64_t *val_tmp = val;
> > + int ret;
> > + bool found = false;
> > + unsigned int cos_max = 0;
> > +
> > + /* cos_max is the one of the feature which is being set. */
> > + list_for_each_entry(feat, &info->feat_list, list)
> > + {
> > + if ( feat->feature != feat_type )
> > + continue;
> > +
> > + cos_max = feat->ops.get_cos_max(feat);
> > + if ( cos_max > 0 )
>
> What's the purpose of this check? I.e. why do you continue the loop
> in case you find zero? You won't find another node with the same
> feat_type, will you?
>
This is corrected in next version. Thanks!
> > + break;
> > + }
> > +
> > + for ( cos = 0; cos <= cos_max; cos++ )
> > + {
> > + if ( cos && !ref[cos] )
> > + continue;
> > +
> > + /* Not found, need find again from beginning. */
>
> You may not even have looked yet, so how can you say "not found"?
>
Sorry, it should be "If not found, need find again...".
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow.
2017-03-09 14:10 ` Jan Beulich
@ 2017-03-10 5:40 ` Yi Sun
2017-03-10 9:24 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-10 5:40 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-09 07:10:33, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> > @@ -397,6 +408,25 @@ static int l3_cat_compare_val(const uint64_t val[],
> > return 0;
> > }
> >
> > +static bool l3_cat_fits_cos_max(const uint64_t val[],
> > + const struct feat_node *feat,
> > + unsigned int cos)
> > +{
> > + uint64_t l3_def_cbm;
> > +
> > + l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
>
> Seeing this pattern recur I wonder whether this also wouldn't be
> something that could be stored once in a generic manner, avoiding
> the need for this per-feature callback (cos_max should be common
> to all features too - not the values, but the abstract notion - so
> perhaps get_cos_max() isn't needed as a callback either).
>
DYM to create a member in 'struct feat_node'? E.g:
uint64_t def_val;
> > static int pick_avail_cos(const struct psr_socket_info *info,
> > const uint64_t *val, uint32_t array_len,
> > unsigned int old_cos,
> > enum psr_feat_type feat_type)
> > {
> > + unsigned int cos;
> > + unsigned int cos_max = 0;
> > + const struct feat_node *feat;
> > + const unsigned int *ref = info->cos_ref;
> > +
> > + /*
> > + * cos_max is the one of the feature which is being set.
> > + */
>
> This is a single line comment.
>
> > + list_for_each_entry(feat, &info->feat_list, list)
> > + {
> > + if ( feat->feature != feat_type )
> > + continue;
> > +
> > + cos_max = feat->ops.get_cos_max(feat);
> > + if ( cos_max > 0 )
> > + break;
> > + }
>
> I think I've seen this a number of times by now - please make this a
> helper function, or store the result (which isn't going to change
> afaict).
>
This behavior is changed in next version.
> Other than these comments given to earlier patches apply here too,
> as I think is obvious.
>
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 13/24] x86: refactor psr: implement CPU init and free flow for CDP.
2017-03-09 14:53 ` Jan Beulich
@ 2017-03-10 5:50 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-03-10 5:50 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-09 07:53:16, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -97,6 +97,7 @@ struct psr_cat_hw_info {
> > struct feat_hw_info {
> > union {
> > struct psr_cat_hw_info l3_cat_info;
> > + struct psr_cat_hw_info l3_cdp_info;
>
> Two union members of the same type? What's the union good for
> then? (I've peeked ahead, and L2 CAT adds yet another one. A
> strong sign that you've gone too far with what needs to be per-
> feature vs what can be common.)
>
I have corrected this. L3 CAT/CDP and L2 CAT will use a common HW info
in next version.
> > @@ -195,6 +196,22 @@ struct feat_node {
> > struct list_head list;
> > };
> >
> > +/*
> > + * get_data - get DATA COS register value from input COS ID.
> > + * @feat: the feature list entry.
> > + * @cos: the COS ID.
> > + */
> > +#define get_cdp_data(feat, cos) \
> > + ( feat->cos_reg_val[cos * 2] )
>
> Stray blanks inside parentheses. Macro parameters need to be
> parenthesized.
>
It has been corrected in next version per Roger's comment.
[...]
> > @@ -1267,6 +1351,14 @@ static int psr_cpu_prepare(void)
> > (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
> > return -ENOMEM;
> >
> > + if ( feat_l3_cdp == NULL &&
> > + (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
> > + {
> > + xfree(feat_l3_cat);
> > + feat_l3_cat = NULL;
>
> Why the freeing? We've decided to allow for one node to be kept,
> so no reason to free it on the error path here.
>
Ok, will correct this.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow.
2017-03-09 15:04 ` Jan Beulich
@ 2017-03-10 5:52 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-03-10 5:52 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-09 08:04:35, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> > @@ -684,6 +686,53 @@ struct feat_ops l3_cdp_ops = {
> > .write_msr = l3_cdp_write_msr,
> > };
> >
> > +/* L2 CAT callback functions implementation. */
> > +static void l2_cat_init_feature(struct cpuid_leaf regs,
>
> const struct cpuid_leaf * (and of course I should have noticed this
> already in earlier patches, but I didn't)
>
Will fixed this from beginning.
> > + struct feat_node *feat,
> > + struct psr_socket_info *info)
> > +{
> > + struct psr_cat_hw_info l2_cat = { };
>
> I realize Konrad did ask for the initializer here, but it's really pointless
> to have such when both structure fields get very obviously set right
> below. If you want an initializer, just set the fields to their final values.
>
Got it. Thanks!
> > + unsigned int socket;
> > +
> > + /* No valid values so do not enable the feature. */
> > + if ( !regs.a || !regs.d )
> > + return;
> > +
> > + l2_cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
> > + l2_cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
> > +
> > + /* cos=0 is reserved as default cbm(all ones). */
> > + feat->cos_reg_val[0] = (1ull << l2_cat.cbm_len) - 1;
> > +
> > + feat->feature = PSR_SOCKET_L2_CAT;
> > + ASSERT(!test_bit(PSR_SOCKET_L2_CAT, &info->feat_mask));
> > + __set_bit(PSR_SOCKET_L2_CAT, &info->feat_mask);
> > +
> > + feat->info.l2_cat_info = l2_cat;
> > +
> > + info->nr_feat++;
>
> Is this field really used for anything? I don't recall seeing other than
> these increments.
>
Just for a record. Will remove it.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow.
2017-03-09 15:13 ` Jan Beulich
@ 2017-03-10 5:57 ` Yi Sun
2017-03-10 9:26 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-10 5:57 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-09 08:13:59, Jan Beulich wrote:
> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> > +static bool l2_cat_get_feat_info(const struct feat_node *feat,
> > + uint32_t data[], uint32_t array_len)
> > +{
> > + if ( !data || 2 > array_len )
> > + return false;
> > +
> > + data[CBM_LEN] = feat->info.l2_cat_info.cbm_len;
> > + data[COS_MAX] = feat->info.l2_cat_info.cos_max;
>
> What about PSR_FLAG? Which puts under question the
> array_len check at the start of this and its sibling functions: If
> more array entries are provided, they'd all be left uninitialized
> without the caller having a way to know. Coming back to the
> data structures being the same for all features, there should
> presumably be a structure instead of an array be used for
> communication here, and ...
>
> > @@ -754,6 +755,11 @@ struct xen_sysctl_psr_cat_op {
> > #define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
> > uint32_t flags; /* OUT: CAT flags */
> > } l3_info;
> > +
> > + struct {
> > + uint32_t cbm_len; /* OUT: CBM length */
> > + uint32_t cos_max; /* OUT: Maximum COS */
> > + } l2_info;
>
> ... this should use the same structure as l3_info (flags being
> set to zero until a use arises). This would then also allow for
> more code to be shared instead of duplicated.
>
Ok, will reuse l3_info for L2 CAT. May modify its name to cat_info.
But for MBA or future feature, its info is different with CAT. So, it may
not be approriate to use a structure as parameter for 'psr_get_info'.
So I would prefer to keep data[]. Of course, I will correct array_len check
per Roger's suggestion, like "array_len != PSR_INFO_SIZE". Is that good to
you?
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-08 16:07 ` Jan Beulich
2017-03-10 2:54 ` Yi Sun
@ 2017-03-10 7:46 ` Yi Sun
2017-03-10 9:10 ` Jan Beulich
1 sibling, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-10 7:46 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-08 09:07:10, Jan Beulich wrote:
[...]
> > /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> > static void psr_free_cos(struct domain *d)
> > {
> > - if( !d->arch.psr_cos_ids )
> > + unsigned int socket, cos;
> > +
> > + if ( !d->arch.psr_cos_ids )
> > return;
>
> As in an earlier patch I've asked for this check to be removed, I
> think you will need to add a check on socket_info to be non-
> NULL somewhere in this function.
>
d->arch.psr_cos_ids is used in the loop below. Shall I not check it?
> > + for ( socket = 0; socket < nr_sockets; socket++ )
> > + {
> > + struct psr_socket_info *info;
> > +
> > + /* cos 0 is default one which does not need be handled. */
> > + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
Here.
> > + continue;
> > +
BRs,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
2017-03-10 1:32 ` Yi Sun
@ 2017-03-10 8:56 ` Jan Beulich
2017-03-13 2:18 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-10 8:56 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 10.03.17 at 02:32, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-08 07:56:54, Jan Beulich wrote:
>> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> > -static int psr_cpu_prepare(unsigned int cpu)
>> > +static void cpu_init_work(void)
>> > +{
>> > + struct psr_socket_info *info;
>> > + unsigned int socket;
>> > + unsigned int cpu = smp_processor_id();
>> > + struct feat_node *feat;
>> > + struct cpuid_leaf regs = { .a = 0, .b = 0, .c = 0, .d = 0 };
>>
>> I don't see you needing an initializer here at all, but if you really
>> want one for some reason, the same effect can be had with just
>> {}.
>>
> Konrad suggested me to initialize it like this in v5 patch review comments.
> I think he has experienced some strange issues when he forgot to set _all_
> the entries a structure allocated on the stack.
I can see there being cases where this is desirable; I don't think
this is one of them. (See also a related comment I had made on
a later patch.)
>> > + if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
>>
>> Do you really mean to not universally check the global (boot CPU)
>> flag? I.e. possibly differing behavior on different CPUs?
>>
> Yes, different sockets may have different configurations. E.g. sockt 0 has
> PQE support but socket 1 does not.
For all other CPU features we assume symmetry. Why would we
not do so here? And how would things even work in that case,
when a vCPU gets moved (by the scheduler) from a more capable
to a less capable pCPU?
>> > static void psr_cpu_init(void)
>> > {
>> > + if ( socket_info )
>> > + cpu_init_work();
>> > +
>> > psr_assoc_init();
>> > }
>> >
>> > static void psr_cpu_fini(unsigned int cpu)
>> > {
>> > + if ( socket_info )
>> > + cpu_fini_work(cpu);
>> > return;
>> > }
>>
>> Is it really useful to use another layer of helper functions here?
>>
> The reason we define 'cpu_fini_work' is to match 'cpu_init_work'.
And the question was for both of them.
> If we move
> codes of 'cpu_init_work' into 'psr_cpu_init', the codes look messy.
I don't think that's the case; I could see this as a valid argument if
the calling functions above were already quite complex.
> That is
> the reason we define 'cpu_init_work'. Do you think if it is acceptable to
> you?
Well, I won't NAK the patch if you keep them, but I'd prefer the
functions to be folded into their callers.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 06/24] x86: refactor psr: implement get hw info flow.
2017-03-10 1:43 ` Yi Sun
@ 2017-03-10 8:57 ` Jan Beulich
2017-03-10 9:01 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-10 8:57 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 10.03.17 at 02:43, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-08 08:15:33, Jan Beulich wrote:
>> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> > --- a/xen/arch/x86/psr.c
>> > +++ b/xen/arch/x86/psr.c
>> > @@ -84,6 +84,7 @@ enum psr_feat_type {
>> > PSR_SOCKET_L3_CAT = 0,
>> > PSR_SOCKET_L3_CDP,
>> > PSR_SOCKET_L2_CAT,
>> > + PSR_SOCKET_UNKNOWN = 0xFFFF,
>>
>> Any reason to use this value, instead of just the next sequential one?
>>
> This is an error value used below, in 'psr_cbm_type_to_feat_type'. To make it
> explicitly different, I assign a big value to it.
How does that "big value" help?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 06/24] x86: refactor psr: implement get hw info flow.
2017-03-10 8:57 ` Jan Beulich
@ 2017-03-10 9:01 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-03-10 9:01 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-10 01:57:57, Jan Beulich wrote:
> >>> On 10.03.17 at 02:43, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-08 08:15:33, Jan Beulich wrote:
> >> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> >> > --- a/xen/arch/x86/psr.c
> >> > +++ b/xen/arch/x86/psr.c
> >> > @@ -84,6 +84,7 @@ enum psr_feat_type {
> >> > PSR_SOCKET_L3_CAT = 0,
> >> > PSR_SOCKET_L3_CDP,
> >> > PSR_SOCKET_L2_CAT,
> >> > + PSR_SOCKET_UNKNOWN = 0xFFFF,
> >>
> >> Any reason to use this value, instead of just the next sequential one?
> >>
> > This is an error value used below, in 'psr_cbm_type_to_feat_type'. To make it
> > explicitly different, I assign a big value to it.
>
> How does that "big value" help?
>
It is just my habit. I will remove this assignment if you don't like it.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 07/24] x86: refactor psr: implement get value flow.
2017-03-10 1:50 ` Yi Sun
@ 2017-03-10 9:05 ` Jan Beulich
0 siblings, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-10 9:05 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 10.03.17 at 02:50, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-08 08:35:53, Jan Beulich wrote:
>> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> > @@ -498,6 +516,15 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
>> > if ( feat->feature != feat_type )
>> > continue;
>> >
>> > + if ( d )
>> > + {
>> > + cos = d->arch.psr_cos_ids[socket];
>> > + if ( feat->ops.get_val(feat, cos, type, val) )
>> > + return 0;
>> > + else
>> > + break;
>> > + }
>> > +
>> > if ( feat->ops.get_feat_info(feat, data, array_len) )
>> > return 0;
>> > else
>>
>> Looking at the context here - is it really a good idea to overload the
>> function in this way, rather than creating a second one? Your
>> only complicating the live of the callers, as can be seen e.g. ...
>>
> These codes were separated into two functions before, 'psr_get_info' and
> 'psr_get_val'. But there are some common codes. So, Konrad suggested me
> to create a helper function to reduce redundant codes.
I think you went too far - breaking out common code is nice, but if
the two callers now pass NULL/0 each as two of the last four
arguments, this is a clear indication that you've folded more code
than was actually common.
>> > @@ -507,10 +534,16 @@ int psr_get_info(unsigned int socket, enum cbm_type
> type,
>> > return -ENOENT;
>> > }
>> >
>> > -int psr_get_l3_cbm(struct domain *d, unsigned int socket,
>> > - uint64_t *cbm, enum cbm_type type)
>> > +int psr_get_info(unsigned int socket, enum cbm_type type,
>> > + uint32_t data[], unsigned int array_len)
>> > +{
>> > + return psr_get(socket, type, data, array_len, NULL, NULL);
>>
>> ... here and ...
>>
>> > +}
>> > +
>> > +int psr_get_val(struct domain *d, unsigned int socket,
>> > + uint64_t *val, enum cbm_type type)
>> > {
>> > - return 0;
>> > + return psr_get(socket, type, NULL, 0, d, val);
>> > }
>>
>> ... here (it is a bad sign that both pass NULL on either side).
>>
> Yes, these things look not good. But to keep a common helper I have to pass
> all necessary parameters into it. What is your suggestion?
If there's really that much code to be shared (which I continue not
to be convinced of), use of a function pointer may make this look a
little better. But I think when having tried to carry out the earlier
suggestion, you should have responded back right away indicating
this would result in other ugliness.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-10 2:54 ` Yi Sun
@ 2017-03-10 9:09 ` Jan Beulich
2017-03-13 2:36 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-10 9:09 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 10.03.17 at 03:54, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-08 09:07:10, Jan Beulich wrote:
>> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> > +static int write_psr_msr(unsigned int socket, unsigned int cos,
>> > + const uint64_t *val)
>> > +{
>> > + return -ENOENT;
>> > +}
>>
>> Is this function intended you write just one MSR, or potentially many?
>> In the latter case the name would perhaps better be "write_psr_msrs()".
>>
> For one feature, it does set one MSR.
In which case - why is the value being passed by pointer?
>> > + ref[old_cos]--;
>> > + spin_unlock(&info->ref_lock);
>> > +
>> > + /*
>> > + * Step 6:
>> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
>> > + * which COS the domain is using on the socket. One domain can only use
>> > + * one COS ID at same time on each socket.
>> > + */
>> > + d->arch.psr_cos_ids[socket] = cos;
>>
>> So the domain has not been paused, i.e. some of its vCPU-s may
>> be running on other pCPU-s (including ones on the socket in
>> question). How come it is safe to update this value here?
>>
> This is a domctl operation. It is protected by domctl_lock which is locked in
> do_domctl().
But that lock doesn't keep the subject domain's vCPU-s from
running on other pCPU-s at the same time.
>> > /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
>> > static void psr_free_cos(struct domain *d)
>> > {
>> > - if( !d->arch.psr_cos_ids )
>> > + unsigned int socket, cos;
>> > +
>> > + if ( !d->arch.psr_cos_ids )
>> > return;
>>
>> As in an earlier patch I've asked for this check to be removed, I
>> think you will need to add a check on socket_info to be non-
>> NULL somewhere in this function.
>>
> Ok, will do it in the loop.
Please don't - loop invariants should be put outside of loops.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-10 7:46 ` Yi Sun
@ 2017-03-10 9:10 ` Jan Beulich
0 siblings, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-10 9:10 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 10.03.17 at 08:46, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-08 09:07:10, Jan Beulich wrote:
> [...]
>> > /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
>> > static void psr_free_cos(struct domain *d)
>> > {
>> > - if( !d->arch.psr_cos_ids )
>> > + unsigned int socket, cos;
>> > +
>> > + if ( !d->arch.psr_cos_ids )
>> > return;
>>
>> As in an earlier patch I've asked for this check to be removed, I
>> think you will need to add a check on socket_info to be non-
>> NULL somewhere in this function.
>>
> d->arch.psr_cos_ids is used in the loop below. Shall I not check it?
>
>> > + for ( socket = 0; socket < nr_sockets; socket++ )
>> > + {
>> > + struct psr_socket_info *info;
>> > +
>> > + /* cos 0 is default one which does not need be handled. */
>> > + if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
> Here.
Oh, yes, you should. But the check should be added here instead
of in the earlier patch.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-03-10 3:21 ` Yi Sun
@ 2017-03-10 9:15 ` Jan Beulich
2017-03-13 2:43 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-10 9:15 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 10.03.17 at 04:21, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-08 09:54:04, Jan Beulich wrote:
>> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> > @@ -207,6 +233,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
>> > return feat_type;
>> > }
>> >
>> > +static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
>> > +{
>> > + unsigned int first_bit, zero_bit;
>> > +
>> > + /* Set bits should only in the range of [0, cbm_len). */
>> > + if ( cbm & (~0ull << cbm_len) )
>>
>> This will not do what you intend for cbm_len == 64.
>>
> cbm_len is not 64. cbm_len means the CBM value length, how many bits. For L3
> CAT, it may be 11 bits. For L2 CAT, it may be 8 bits.
And there is an _architectural_ guarantee for them to never
reach 64 bits?
>> > +static int l3_cat_get_old_val(uint64_t val[],
>> > + const struct feat_node *feat,
>> > + unsigned int old_cos)
>> > +{
>> > + if ( old_cos > feat->info.l3_cat_info.cos_max )
>>
>> Afaics this condition is the only L3 CAT specific thing in this function.
>> Should more of it be moved into common code? Same below for
>> set_new_val.
>>
> Sorry, I may not understand your intention. For different features, they have
> different cos_max. Do you mean I should abstract a callback function for all
> features to handle this cos_max check? Thanks!
Yes. All features have a cos_max (even if the precise values may
differ), so common code can do the check if the values is stored
suitably in a field outside of any feature dependent data structures.
>> > -static int assemble_val_array(uint64_t *val,
>> > +static int combine_val_array(uint64_t *val,
>>
>> Same comment as earlier on - please decide for a final name right
>> when introducing a function. In fact I'd prefer it to remain
>> "assemble".
>>
> This is corrected in next version. I will change name back to assemble if you
> like it.
I'm fine with it. Personally I'd like "gather" even better, but the
naming choice in the end is up to you as long as it reflects the
purpose of the function (which imo "combine" doesn't).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow.
2017-03-10 5:35 ` Yi Sun
@ 2017-03-10 9:21 ` Jan Beulich
0 siblings, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-10 9:21 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 10.03.17 at 06:35, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-08 10:03:10, Jan Beulich wrote:
>> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> > @@ -356,6 +369,34 @@ static int l3_cat_set_new_val(uint64_t val[],
>> > return 0;
>> > }
>> >
>> > +static int l3_cat_compare_val(const uint64_t val[],
>> > + const struct feat_node *feat,
>> > + unsigned int cos, bool *found)
>> > +{
>> > + uint64_t l3_def_cbm;
>> > +
>> > + l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
>>
>> Please only calculate the value on the path you need it. Also this
>> will again degenerate of cbm_len == 64.
>>
> Sorry, what do you mean? I need get the default value of L3 CAT here
> to check if input val equals the default one if cos exceeds cos_max.
No, you don't need it at function scope. You only need it ...
>> > + /*
>> > + * Different features' cos_max are different. If cos id of the feature
>> > + * being set exceeds other feature's cos_max, the val of other feature
>> > + * must be default value. HW supports such case.
>> > + */
>> > + if ( cos > feat->info.l3_cat_info.cos_max )
>> > + {
>> > + if ( val[0] != l3_def_cbm )
... in this more narrow one. Specifically no code outside the outer
if() needs the variable.
>> > + {
>> > + *found = false;
>> > + return -ENOENT;
>>
>> What is the difference between this "not found" and ...
>>
> This is an error case. If input cos exceeds the cos_max and the input val is
> not default value, that means the input parameters exceed HW ability. We should
> report error back.
>
>> > + }
>> > + *found = true;
>> > + }
>> > + else
>> > + *found = (val[0] == feat->cos_reg_val[cos]);
>> > +
>> > + return 0;
>>
>> ... the possible one here? I.e. why once -ENOENT and once 0 as
>> return value?
>>
> 0 means success. '*found' means if the val is found or not.
I still don't understand the difference, but perhaps this will all sort
itself out once the cos_max checks get done in common code and
the default values (suitably stored, as suggested elsewhere) can
also be checked by common code. In fact with that I'm not even
sure you will continue to require this feature dependent actor
anymore.
> Per Roger's
> suggestion, I will refine this function to use return value to check
> if it is found or not.
Well, let's see how this ends up being.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow.
2017-03-10 5:40 ` Yi Sun
@ 2017-03-10 9:24 ` Jan Beulich
0 siblings, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-10 9:24 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 10.03.17 at 06:40, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-09 07:10:33, Jan Beulich wrote:
>> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> > @@ -397,6 +408,25 @@ static int l3_cat_compare_val(const uint64_t val[],
>> > return 0;
>> > }
>> >
>> > +static bool l3_cat_fits_cos_max(const uint64_t val[],
>> > + const struct feat_node *feat,
>> > + unsigned int cos)
>> > +{
>> > + uint64_t l3_def_cbm;
>> > +
>> > + l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
>>
>> Seeing this pattern recur I wonder whether this also wouldn't be
>> something that could be stored once in a generic manner, avoiding
>> the need for this per-feature callback (cos_max should be common
>> to all features too - not the values, but the abstract notion - so
>> perhaps get_cos_max() isn't needed as a callback either).
>>
> DYM to create a member in 'struct feat_node'? E.g:
> uint64_t def_val;
Yes. All data items applicable to all features should be outside
the feature dependent data structures. And which pieces of
data to store you should determine by the overall aim of
preferably as little feature-specific code as possible, i.e.
whatever can be made common with reasonable effort should
be made common.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow.
2017-03-10 5:57 ` Yi Sun
@ 2017-03-10 9:26 ` Jan Beulich
0 siblings, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-10 9:26 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 10.03.17 at 06:57, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-09 08:13:59, Jan Beulich wrote:
>> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> > +static bool l2_cat_get_feat_info(const struct feat_node *feat,
>> > + uint32_t data[], uint32_t array_len)
>> > +{
>> > + if ( !data || 2 > array_len )
>> > + return false;
>> > +
>> > + data[CBM_LEN] = feat->info.l2_cat_info.cbm_len;
>> > + data[COS_MAX] = feat->info.l2_cat_info.cos_max;
>>
>> What about PSR_FLAG? Which puts under question the
>> array_len check at the start of this and its sibling functions: If
>> more array entries are provided, they'd all be left uninitialized
>> without the caller having a way to know. Coming back to the
>> data structures being the same for all features, there should
>> presumably be a structure instead of an array be used for
>> communication here, and ...
>>
>> > @@ -754,6 +755,11 @@ struct xen_sysctl_psr_cat_op {
>> > #define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
>> > uint32_t flags; /* OUT: CAT flags */
>> > } l3_info;
>> > +
>> > + struct {
>> > + uint32_t cbm_len; /* OUT: CBM length */
>> > + uint32_t cos_max; /* OUT: Maximum COS */
>> > + } l2_info;
>>
>> ... this should use the same structure as l3_info (flags being
>> set to zero until a use arises). This would then also allow for
>> more code to be shared instead of duplicated.
>>
> Ok, will reuse l3_info for L2 CAT. May modify its name to cat_info.
Of course.
> But for MBA or future feature, its info is different with CAT. So, it may
> not be approriate to use a structure as parameter for 'psr_get_info'.
> So I would prefer to keep data[]. Of course, I will correct array_len check
> per Roger's suggestion, like "array_len != PSR_INFO_SIZE". Is that good to
> you?
Yes, if there is a need for it to be that way down the road, then I'm
fine as long as both the risk of array overrun and that of using
uninitialized data are minimized as much as possible.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow.
2017-03-10 8:56 ` Jan Beulich
@ 2017-03-13 2:18 ` Yi Sun
0 siblings, 0 replies; 122+ messages in thread
From: Yi Sun @ 2017-03-13 2:18 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-10 01:56:04, Jan Beulich wrote:
> >>> On 10.03.17 at 02:32, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-08 07:56:54, Jan Beulich wrote:
> >> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
[...]
> >> > + if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
> >>
> >> Do you really mean to not universally check the global (boot CPU)
> >> flag? I.e. possibly differing behavior on different CPUs?
> >>
> > Yes, different sockets may have different configurations. E.g. sockt 0 has
> > PQE support but socket 1 does not.
>
> For all other CPU features we assume symmetry. Why would we
> not do so here? And how would things even work in that case,
> when a vCPU gets moved (by the scheduler) from a more capable
> to a less capable pCPU?
>
Hmm, ok, I will use boot cpu to check PQE here. Thanks!
> >> > static void psr_cpu_init(void)
> >> > {
> >> > + if ( socket_info )
> >> > + cpu_init_work();
> >> > +
> >> > psr_assoc_init();
> >> > }
> >> >
> >> > static void psr_cpu_fini(unsigned int cpu)
> >> > {
> >> > + if ( socket_info )
> >> > + cpu_fini_work(cpu);
> >> > return;
> >> > }
> >>
> >> Is it really useful to use another layer of helper functions here?
> >>
> > The reason we define 'cpu_fini_work' is to match 'cpu_init_work'.
>
> And the question was for both of them.
>
> > If we move
> > codes of 'cpu_init_work' into 'psr_cpu_init', the codes look messy.
>
> I don't think that's the case; I could see this as a valid argument if
> the calling functions above were already quite complex.
>
> > That is
> > the reason we define 'cpu_init_work'. Do you think if it is acceptable to
> > you?
>
> Well, I won't NAK the patch if you keep them, but I'd prefer the
> functions to be folded into their callers.
>
I will move the codes.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-10 9:09 ` Jan Beulich
@ 2017-03-13 2:36 ` Yi Sun
2017-03-13 12:35 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-13 2:36 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-10 02:09:55, Jan Beulich wrote:
> >>> On 10.03.17 at 03:54, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-08 09:07:10, Jan Beulich wrote:
> >> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> >> > +static int write_psr_msr(unsigned int socket, unsigned int cos,
> >> > + const uint64_t *val)
> >> > +{
> >> > + return -ENOENT;
> >> > +}
> >>
> >> Is this function intended you write just one MSR, or potentially many?
> >> In the latter case the name would perhaps better be "write_psr_msrs()".
> >>
> > For one feature, it does set one MSR.
>
> In which case - why is the value being passed by pointer?
>
Will change it. My original meaning is, for example, when setting L3 CAT, only
one MSR is set.
> >> > + ref[old_cos]--;
> >> > + spin_unlock(&info->ref_lock);
> >> > +
> >> > + /*
> >> > + * Step 6:
> >> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> >> > + * which COS the domain is using on the socket. One domain can only use
> >> > + * one COS ID at same time on each socket.
> >> > + */
> >> > + d->arch.psr_cos_ids[socket] = cos;
> >>
> >> So the domain has not been paused, i.e. some of its vCPU-s may
> >> be running on other pCPU-s (including ones on the socket in
> >> question). How come it is safe to update this value here?
> >>
> > This is a domctl operation. It is protected by domctl_lock which is locked in
> > do_domctl().
>
> But that lock doesn't keep the subject domain's vCPU-s from
> running on other pCPU-s at the same time.
>
Yes, you are right. But only 'psr_ctxt_switch_to()' can access
psr_cos_ids[socket] when psr_cos_ids[socket] is being set. 'psr_ctxt_switch_to()'
read the cos and set it to ASSOC register. Context switch is short so that the
correct cos can be set to ASSOC register in a short time.
BRs,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-03-10 9:15 ` Jan Beulich
@ 2017-03-13 2:43 ` Yi Sun
2017-03-13 12:37 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-13 2:43 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-10 02:15:20, Jan Beulich wrote:
> >>> On 10.03.17 at 04:21, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-08 09:54:04, Jan Beulich wrote:
> >> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> >> > @@ -207,6 +233,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> >> > return feat_type;
> >> > }
> >> >
> >> > +static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
> >> > +{
> >> > + unsigned int first_bit, zero_bit;
> >> > +
> >> > + /* Set bits should only in the range of [0, cbm_len). */
> >> > + if ( cbm & (~0ull << cbm_len) )
> >>
> >> This will not do what you intend for cbm_len == 64.
> >>
> > cbm_len is not 64. cbm_len means the CBM value length, how many bits. For L3
> > CAT, it may be 11 bits. For L2 CAT, it may be 8 bits.
>
> And there is an _architectural_ guarantee for them to never
> reach 64 bits?
>
Per SDM, 'EAX[4:0] reports the length of the capacity bitmask length'. So, we
get cbm_len in '*_init_feature' function as below.
#define CAT_CBM_LEN_MASK 0x1f
cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
So, the max cbm_len is '32'.
BRs,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-13 2:36 ` Yi Sun
@ 2017-03-13 12:35 ` Jan Beulich
2017-03-14 2:43 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-13 12:35 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 13.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-10 02:09:55, Jan Beulich wrote:
>> >>> On 10.03.17 at 03:54, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-08 09:07:10, Jan Beulich wrote:
>> >> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> >> > + ref[old_cos]--;
>> >> > + spin_unlock(&info->ref_lock);
>> >> > +
>> >> > + /*
>> >> > + * Step 6:
>> >> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
>> >> > + * which COS the domain is using on the socket. One domain can only use
>> >> > + * one COS ID at same time on each socket.
>> >> > + */
>> >> > + d->arch.psr_cos_ids[socket] = cos;
>> >>
>> >> So the domain has not been paused, i.e. some of its vCPU-s may
>> >> be running on other pCPU-s (including ones on the socket in
>> >> question). How come it is safe to update this value here?
>> >>
>> > This is a domctl operation. It is protected by domctl_lock which is locked in
>> > do_domctl().
>>
>> But that lock doesn't keep the subject domain's vCPU-s from
>> running on other pCPU-s at the same time.
>>
> Yes, you are right. But only 'psr_ctxt_switch_to()' can access
> psr_cos_ids[socket] when psr_cos_ids[socket] is being set. 'psr_ctxt_switch_to()'
> read the cos and set it to ASSOC register. Context switch is short so that the
> correct cos can be set to ASSOC register in a short time.
That's a reply you should never give: No matter how short a time
window, eventually it'll be hit. A very good example of this is the
VMCS race we've recently fixed (commit 2f4d2198a9), and which
had been there for years until it was first observed (and after
that it took another year or so until we've actually managed to
figure out what's going on).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-03-13 2:43 ` Yi Sun
@ 2017-03-13 12:37 ` Jan Beulich
2017-03-14 2:20 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-13 12:37 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 13.03.17 at 03:43, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-10 02:15:20, Jan Beulich wrote:
>> >>> On 10.03.17 at 04:21, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-08 09:54:04, Jan Beulich wrote:
>> >> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> >> > @@ -207,6 +233,29 @@ static enum psr_feat_type
> psr_cbm_type_to_feat_type(enum cbm_type type)
>> >> > return feat_type;
>> >> > }
>> >> >
>> >> > +static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
>> >> > +{
>> >> > + unsigned int first_bit, zero_bit;
>> >> > +
>> >> > + /* Set bits should only in the range of [0, cbm_len). */
>> >> > + if ( cbm & (~0ull << cbm_len) )
>> >>
>> >> This will not do what you intend for cbm_len == 64.
>> >>
>> > cbm_len is not 64. cbm_len means the CBM value length, how many bits. For L3
>> > CAT, it may be 11 bits. For L2 CAT, it may be 8 bits.
>>
>> And there is an _architectural_ guarantee for them to never
>> reach 64 bits?
>>
> Per SDM, 'EAX[4:0] reports the length of the capacity bitmask length'. So, we
> get cbm_len in '*_init_feature' function as below.
>
> #define CAT_CBM_LEN_MASK 0x1f
> cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
>
> So, the max cbm_len is '32'.
Great, yet in that case why do you use 64-bit calculations at all?
Avoiding the undefinedness when cbm_len == 32 can be
achieved without 64-bit arithmetic.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-03-13 12:37 ` Jan Beulich
@ 2017-03-14 2:20 ` Yi Sun
2017-03-14 6:32 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-14 2:20 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel
On 17-03-13 06:37:41, Jan Beulich wrote:
> >>> On 13.03.17 at 03:43, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-10 02:15:20, Jan Beulich wrote:
> >> >>> On 10.03.17 at 04:21, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-03-08 09:54:04, Jan Beulich wrote:
> >> >> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> >> >> > @@ -207,6 +233,29 @@ static enum psr_feat_type
> > psr_cbm_type_to_feat_type(enum cbm_type type)
> >> >> > return feat_type;
> >> >> > }
> >> >> >
> >> >> > +static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
> >> >> > +{
> >> >> > + unsigned int first_bit, zero_bit;
> >> >> > +
> >> >> > + /* Set bits should only in the range of [0, cbm_len). */
> >> >> > + if ( cbm & (~0ull << cbm_len) )
> >> >>
> >> >> This will not do what you intend for cbm_len == 64.
> >> >>
> >> > cbm_len is not 64. cbm_len means the CBM value length, how many bits. For L3
> >> > CAT, it may be 11 bits. For L2 CAT, it may be 8 bits.
> >>
> >> And there is an _architectural_ guarantee for them to never
> >> reach 64 bits?
> >>
> > Per SDM, 'EAX[4:0] reports the length of the capacity bitmask length'. So, we
> > get cbm_len in '*_init_feature' function as below.
> >
> > #define CAT_CBM_LEN_MASK 0x1f
> > cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
> >
> > So, the max cbm_len is '32'.
>
> Great, yet in that case why do you use 64-bit calculations at all?
> Avoiding the undefinedness when cbm_len == 32 can be
> achieved without 64-bit arithmetic.
>
Considering to support future feature, we use 64bit as input parameter type for
psr_get_val/psr_set_val. So, we handle all values as 64bit.
For psr_check_cbm case, I think we can add an ASSERT as below. Is that good for
you? Thanks!
ASSERT(cbm_len <= 32);
> Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-13 12:35 ` Jan Beulich
@ 2017-03-14 2:43 ` Yi Sun
2017-03-14 6:29 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-14 2:43 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-13 06:35:33, Jan Beulich wrote:
> >>> On 13.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-10 02:09:55, Jan Beulich wrote:
> >> >>> On 10.03.17 at 03:54, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-03-08 09:07:10, Jan Beulich wrote:
> >> >> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
> >> >> > + ref[old_cos]--;
> >> >> > + spin_unlock(&info->ref_lock);
> >> >> > +
> >> >> > + /*
> >> >> > + * Step 6:
> >> >> > + * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> >> >> > + * which COS the domain is using on the socket. One domain can only use
> >> >> > + * one COS ID at same time on each socket.
> >> >> > + */
> >> >> > + d->arch.psr_cos_ids[socket] = cos;
> >> >>
> >> >> So the domain has not been paused, i.e. some of its vCPU-s may
> >> >> be running on other pCPU-s (including ones on the socket in
> >> >> question). How come it is safe to update this value here?
> >> >>
> >> > This is a domctl operation. It is protected by domctl_lock which is locked in
> >> > do_domctl().
> >>
> >> But that lock doesn't keep the subject domain's vCPU-s from
> >> running on other pCPU-s at the same time.
> >>
> > Yes, you are right. But only 'psr_ctxt_switch_to()' can access
> > psr_cos_ids[socket] when psr_cos_ids[socket] is being set. 'psr_ctxt_switch_to()'
> > read the cos and set it to ASSOC register. Context switch is short so that the
> > correct cos can be set to ASSOC register in a short time.
>
> That's a reply you should never give: No matter how short a time
> window, eventually it'll be hit. A very good example of this is the
> VMCS race we've recently fixed (commit 2f4d2198a9), and which
> had been there for years until it was first observed (and after
> that it took another year or so until we've actually managed to
> figure out what's going on).
>
Sorry for that. Let me explain in details.
There are three scenarios. E.g.
1. User calls domctl interface on Dom0 to set a COS ID 1 for Dom1 into its
psr_cos_ids[]. Then, Dom1 is scheduled so that 'psr_ctxt_switch_to()' is
called which makes COS ID 1 work. For this case, we do not any action.
2. Dom1 runs on CPU 1 and COS ID 1 is working. At same time, user calls domctl
interface on Dom0 to set a new COS ID 2 for Dom1 into psr_cos_ids[]. After
time slice ends, the Dom1 is scheduled again, the new COS ID 2 will work.
For this case, we don't need any action either.
3. When a new COS ID is being set to psr_cos_ids[], 'psr_ctxt_switch_to()'
is called to access the same psr_cos_ids[] member through 'psr_assoc_cos'.
The COS ID is constrained by cos_mask so that it cannot exceeds the cos_max.
So even the COS ID got here is wrong, it is still a workable ID (within
cos_max). The functionality is still workable but of course the COS ID may
not be the one that user intends to use.
If you think scenario 3 is not acceptable, I suggest to add read write lock as
below. How do you think? Thanks!
static void psr_assoc_cos()
{
read_lock(&rwlock);
*reg = (*reg & ~cos_mask) |
(((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
read_unlock(&rwlock);
}
int psr_set_val()
{
......
write_lock(&rwlock);
d->arch.psr_cos_ids[socket] = cos;
write_unlock(&rwlock);
......
}
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-14 2:43 ` Yi Sun
@ 2017-03-14 6:29 ` Jan Beulich
2017-03-14 9:21 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-14 6:29 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> Yi Sun <yi.y.sun@linux.intel.com> 03/14/17 3:42 AM >>>
>There are three scenarios. E.g.
>1. User calls domctl interface on Dom0 to set a COS ID 1 for Dom1 into its
>psr_cos_ids[]. Then, Dom1 is scheduled so that 'psr_ctxt_switch_to()' is
>called which makes COS ID 1 work. For this case, we do not any action.
>
>2. Dom1 runs on CPU 1 and COS ID 1 is working. At same time, user calls domctl
>interface on Dom0 to set a new COS ID 2 for Dom1 into psr_cos_ids[]. After
>time slice ends, the Dom1 is scheduled again, the new COS ID 2 will work.
>For this case, we don't need any action either.
And that's because of? I'd think the domctl caller can expect the new COS ID
to take effect immediately for all vCPU-s of the target domain.
>3. When a new COS ID is being set to psr_cos_ids[], 'psr_ctxt_switch_to()'
>is called to access the same psr_cos_ids[] member through 'psr_assoc_cos'.
>The COS ID is constrained by cos_mask so that it cannot exceeds the cos_max.
>So even the COS ID got here is wrong, it is still a workable ID (within
>cos_max). The functionality is still workable but of course the COS ID may
>not be the one that user intends to use.
>
>If you think scenario 3 is not acceptable, I suggest to add read write lock as
>below. How do you think? Thanks!
>
>static void psr_assoc_cos()
>{
>read_lock(&rwlock);
>*reg = (*reg & ~cos_mask) |
>(((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
>read_unlock(&rwlock);
>}
>
>int psr_set_val()
>{
>......
>write_lock(&rwlock);
>d->arch.psr_cos_ids[socket] = cos;
>write_unlock(&rwlock);
>......
>}
I don't see how that would help. The domain could then still use a stale COS
ID. I see only two valid approaches: Either you pause the domain during the
update, or you IPI CPUs running vCPU-s of that domain to reload their MSRs.
The latter could become tricky afaict ...
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array.
2017-03-14 2:20 ` Yi Sun
@ 2017-03-14 6:32 ` Jan Beulich
0 siblings, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-14 6:32 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> Yi Sun <yi.y.sun@linux.intel.com> 03/14/17 3:20 AM >>>
>On 17-03-13 06:37:41, Jan Beulich wrote:
>> >>> On 13.03.17 at 03:43, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-10 02:15:20, Jan Beulich wrote:
>> >> >>> On 10.03.17 at 04:21, <yi.y.sun@linux.intel.com> wrote:
>> >> > On 17-03-08 09:54:04, Jan Beulich wrote:
>> >> >> >>> On 15.02.17 at 09:49, <yi.y.sun@linux.intel.com> wrote:
>> >> >> > @@ -207,6 +233,29 @@ static enum psr_feat_type
>> > psr_cbm_type_to_feat_type(enum cbm_type type)
>> >> >> > return feat_type;
>> >> >> > }
>> >> >> >
>> >> >> > +static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
>> >> >> > +{
>> >> >> > + unsigned int first_bit, zero_bit;
>> >> >> > +
>> >> >> > + /* Set bits should only in the range of [0, cbm_len). */
>> >> >> > + if ( cbm & (~0ull << cbm_len) )
>> >> >>
>> >> >> This will not do what you intend for cbm_len == 64.
>> >> >>
>> >> > cbm_len is not 64. cbm_len means the CBM value length, how many bits. For L3
>> >> > CAT, it may be 11 bits. For L2 CAT, it may be 8 bits.
>> >>
>> >> And there is an _architectural_ guarantee for them to never
>> >> reach 64 bits?
>> >>
>> > Per SDM, 'EAX[4:0] reports the length of the capacity bitmask length'. So, we
>> > get cbm_len in '*_init_feature' function as below.
>> >
>> > #define CAT_CBM_LEN_MASK 0x1f
>> > cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
>> >
>> > So, the max cbm_len is '32'.
>>
>> Great, yet in that case why do you use 64-bit calculations at all?
>> Avoiding the undefinedness when cbm_len == 32 can be
>> achieved without 64-bit arithmetic.
>>
>Considering to support future feature, we use 64bit as input parameter type for
>psr_get_val/psr_set_val. So, we handle all values as 64bit.
But you quote the SDM above as limiting values to 32 bits. Plus I can't see why
making the code ready for 64-bit values is any help when you assume the
architecture might change - if it does, values may as well become wider. Allowing
for some slack in the public interface is certainly reasonable, but let's not make
internal code deal with needlessly wide numbers. After all dealing with 32-bit
quantities is more efficient.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-14 6:29 ` Jan Beulich
@ 2017-03-14 9:21 ` Yi Sun
2017-03-14 10:24 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-14 9:21 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-14 00:29:09, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 03/14/17 3:42 AM >>>
> >There are three scenarios. E.g.
> >1. User calls domctl interface on Dom0 to set a COS ID 1 for Dom1 into its
> >psr_cos_ids[]. Then, Dom1 is scheduled so that 'psr_ctxt_switch_to()' is
> >called which makes COS ID 1 work. For this case, we do not any action.
> >
> >2. Dom1 runs on CPU 1 and COS ID 1 is working. At same time, user calls domctl
> >interface on Dom0 to set a new COS ID 2 for Dom1 into psr_cos_ids[]. After
> >time slice ends, the Dom1 is scheduled again, the new COS ID 2 will work.
> >For this case, we don't need any action either.
>
> And that's because of? I'd think the domctl caller can expect the new COS ID
> to take effect immediately for all vCPU-s of the target domain.
>
> >3. When a new COS ID is being set to psr_cos_ids[], 'psr_ctxt_switch_to()'
> >is called to access the same psr_cos_ids[] member through 'psr_assoc_cos'.
> >The COS ID is constrained by cos_mask so that it cannot exceeds the cos_max.
> >So even the COS ID got here is wrong, it is still a workable ID (within
> >cos_max). The functionality is still workable but of course the COS ID may
> >not be the one that user intends to use.
> >
> >If you think scenario 3 is not acceptable, I suggest to add read write lock as
> >below. How do you think? Thanks!
> >
> >static void psr_assoc_cos()
> >{
> >read_lock(&rwlock);
> >*reg = (*reg & ~cos_mask) |
> >(((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
> >read_unlock(&rwlock);
> >}
> >
> >int psr_set_val()
> >{
> >......
> >write_lock(&rwlock);
> >d->arch.psr_cos_ids[socket] = cos;
> >write_unlock(&rwlock);
> >......
> >}
>
> I don't see how that would help. The domain could then still use a stale COS
> ID. I see only two valid approaches: Either you pause the domain during the
> update, or you IPI CPUs running vCPU-s of that domain to reload their MSRs.
> The latter could become tricky afaict ...
>
For IPI solution, could you please help to check if below codes can work?
Thanks!
struct assoc_write_info
{
struct domain *d;
};
static void do_write_assoc_msr(void *data)
{
......
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
......
}
static void write_psr_assoc_msr(struct domain *d)
{
struct assoc_write_info data = { .d = d };
cpumask_t *online = cpupool_domain_cpumask(d);
/* Make all valid cpus execute do_write_assoc_msr. */
on_selected_cpus(online, do_write_assoc_msr, &data, 0);
}
int psr_set_val(...)
{
......
d->arch.psr_cos_ids[socket] = cos;
write_psr_assoc_msr(d);
......
}
BRs,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-14 9:21 ` Yi Sun
@ 2017-03-14 10:24 ` Jan Beulich
2017-03-15 2:52 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-14 10:24 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 14.03.17 at 10:21, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-14 00:29:09, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 03/14/17 3:42 AM >>>
>> >There are three scenarios. E.g.
>> >1. User calls domctl interface on Dom0 to set a COS ID 1 for Dom1 into its
>> >psr_cos_ids[]. Then, Dom1 is scheduled so that 'psr_ctxt_switch_to()' is
>> >called which makes COS ID 1 work. For this case, we do not any action.
>> >
>> >2. Dom1 runs on CPU 1 and COS ID 1 is working. At same time, user calls
> domctl
>> >interface on Dom0 to set a new COS ID 2 for Dom1 into psr_cos_ids[]. After
>> >time slice ends, the Dom1 is scheduled again, the new COS ID 2 will work.
>> >For this case, we don't need any action either.
>>
>> And that's because of? I'd think the domctl caller can expect the new COS ID
>> to take effect immediately for all vCPU-s of the target domain.
>>
>> >3. When a new COS ID is being set to psr_cos_ids[], 'psr_ctxt_switch_to()'
>> >is called to access the same psr_cos_ids[] member through 'psr_assoc_cos'.
>> >The COS ID is constrained by cos_mask so that it cannot exceeds the cos_max.
>> >So even the COS ID got here is wrong, it is still a workable ID (within
>> >cos_max). The functionality is still workable but of course the COS ID may
>> >not be the one that user intends to use.
>> >
>> >If you think scenario 3 is not acceptable, I suggest to add read write lock
> as
>> >below. How do you think? Thanks!
>> >
>> >static void psr_assoc_cos()
>> >{
>> >read_lock(&rwlock);
>> >*reg = (*reg & ~cos_mask) |
>> >(((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
>> >read_unlock(&rwlock);
>> >}
>> >
>> >int psr_set_val()
>> >{
>> >......
>> >write_lock(&rwlock);
>> >d->arch.psr_cos_ids[socket] = cos;
>> >write_unlock(&rwlock);
>> >......
>> >}
>>
>> I don't see how that would help. The domain could then still use a stale COS
>> ID. I see only two valid approaches: Either you pause the domain during the
>> update, or you IPI CPUs running vCPU-s of that domain to reload their MSRs.
>> The latter could become tricky afaict ...
>>
> For IPI solution, could you please help to check if below codes can work?
Well, I did say "tricky", didn't I? What you suggest still leaves a
time window, as the result of cpupool_domain_cpumask() may
change behind your back. The simplest (and least tricky) solution
would be to IPI all CPUs, and check whether any adjustment is
needed inside the handler. However, you'd then need to make
sure there's no context switch related race, similar to the VMCS
one I've referred you to as example.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-14 10:24 ` Jan Beulich
@ 2017-03-15 2:52 ` Yi Sun
2017-03-15 7:40 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-15 2:52 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel
On 17-03-14 04:24:51, Jan Beulich wrote:
> >>> On 14.03.17 at 10:21, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-14 00:29:09, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 03/14/17 3:42 AM >>>
> >> >There are three scenarios. E.g.
> >> >1. User calls domctl interface on Dom0 to set a COS ID 1 for Dom1 into its
> >> >psr_cos_ids[]. Then, Dom1 is scheduled so that 'psr_ctxt_switch_to()' is
> >> >called which makes COS ID 1 work. For this case, we do not any action.
> >> >
> >> >2. Dom1 runs on CPU 1 and COS ID 1 is working. At same time, user calls
> > domctl
> >> >interface on Dom0 to set a new COS ID 2 for Dom1 into psr_cos_ids[]. After
> >> >time slice ends, the Dom1 is scheduled again, the new COS ID 2 will work.
> >> >For this case, we don't need any action either.
> >>
> >> And that's because of? I'd think the domctl caller can expect the new COS ID
> >> to take effect immediately for all vCPU-s of the target domain.
> >>
> >> >3. When a new COS ID is being set to psr_cos_ids[], 'psr_ctxt_switch_to()'
> >> >is called to access the same psr_cos_ids[] member through 'psr_assoc_cos'.
> >> >The COS ID is constrained by cos_mask so that it cannot exceeds the cos_max.
> >> >So even the COS ID got here is wrong, it is still a workable ID (within
> >> >cos_max). The functionality is still workable but of course the COS ID may
> >> >not be the one that user intends to use.
> >> >
> >> >If you think scenario 3 is not acceptable, I suggest to add read write lock
> > as
> >> >below. How do you think? Thanks!
> >> >
> >> >static void psr_assoc_cos()
> >> >{
> >> >read_lock(&rwlock);
> >> >*reg = (*reg & ~cos_mask) |
> >> >(((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
> >> >read_unlock(&rwlock);
> >> >}
> >> >
> >> >int psr_set_val()
> >> >{
> >> >......
> >> >write_lock(&rwlock);
> >> >d->arch.psr_cos_ids[socket] = cos;
> >> >write_unlock(&rwlock);
> >> >......
> >> >}
> >>
> >> I don't see how that would help. The domain could then still use a stale COS
> >> ID. I see only two valid approaches: Either you pause the domain during the
> >> update, or you IPI CPUs running vCPU-s of that domain to reload their MSRs.
> >> The latter could become tricky afaict ...
> >>
> > For IPI solution, could you please help to check if below codes can work?
>
> Well, I did say "tricky", didn't I? What you suggest still leaves a
> time window, as the result of cpupool_domain_cpumask() may
> change behind your back. The simplest (and least tricky) solution
> would be to IPI all CPUs, and check whether any adjustment is
> needed inside the handler. However, you'd then need to make
> sure there's no context switch related race, similar to the VMCS
> one I've referred you to as example.
>
Sorry, I may not fully understand your meaning. My thoughts are below.
1. We set 'd->arch.psr_cos_ids[socket] = cos;' in 'psr_set_val';
2. After that, we get valid cpumask through cpupool_domain_cpumask();
3. If the actual valid cpumask changed after that, the new cpu is valid so
that the context switch happens. Then 'psr_ctxt_switch_to' is called to
update the new cpu's ASSOC register with the new COS ID which has been
set in step 1.
4. Send IPI to all cpus on cpumask got in step 2. They will update their
ASSOC registers according to their domains psr_cos_ids[].
So I think this flow can cover all cpus which ASSOC registers need be updated.
What is your idea? Thanks!
BRs,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-15 2:52 ` Yi Sun
@ 2017-03-15 7:40 ` Jan Beulich
2017-03-15 8:18 ` Yi Sun
0 siblings, 1 reply; 122+ messages in thread
From: Jan Beulich @ 2017-03-15 7:40 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.03.17 at 03:52, <yi.y.sun@linux.intel.com> wrote:
> Sorry, I may not fully understand your meaning. My thoughts are below.
> 1. We set 'd->arch.psr_cos_ids[socket] = cos;' in 'psr_set_val';
>
> 2. After that, we get valid cpumask through cpupool_domain_cpumask();
>
> 3. If the actual valid cpumask changed after that, the new cpu is valid so
> that the context switch happens. Then 'psr_ctxt_switch_to' is called to
> update the new cpu's ASSOC register with the new COS ID which has been
> set in step 1.
>
> 4. Send IPI to all cpus on cpumask got in step 2. They will update their
> ASSOC registers according to their domains psr_cos_ids[].
>
> So I think this flow can cover all cpus which ASSOC registers need be
> updated.
You writing it down this way makes me realize that the IPI approach
can't work this way at all: It leaves a time window (between 1 and 2)
where the domain may have vCPU-s running with the wrong COS ID.
I think pausing the vCPU is unavoidable, unless it can be explained
that running with the wrong COS ID for a brief period of time is not
really a problem. I do realize that the pausing approach has its own
difficulty wrt Dom0, but I think that's solvable (e.g. by doing the
actual work in a tasklet instead of in the context of a Dom0 vCPU).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-15 7:40 ` Jan Beulich
@ 2017-03-15 8:18 ` Yi Sun
2017-03-15 8:32 ` Jan Beulich
0 siblings, 1 reply; 122+ messages in thread
From: Yi Sun @ 2017-03-15 8:18 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
On 17-03-15 01:40:06, Jan Beulich wrote:
> >>> On 15.03.17 at 03:52, <yi.y.sun@linux.intel.com> wrote:
> > Sorry, I may not fully understand your meaning. My thoughts are below.
> > 1. We set 'd->arch.psr_cos_ids[socket] = cos;' in 'psr_set_val';
> >
> > 2. After that, we get valid cpumask through cpupool_domain_cpumask();
> >
> > 3. If the actual valid cpumask changed after that, the new cpu is valid so
> > that the context switch happens. Then 'psr_ctxt_switch_to' is called to
> > update the new cpu's ASSOC register with the new COS ID which has been
> > set in step 1.
> >
> > 4. Send IPI to all cpus on cpumask got in step 2. They will update their
> > ASSOC registers according to their domains psr_cos_ids[].
> >
> > So I think this flow can cover all cpus which ASSOC registers need be
> > updated.
>
> You writing it down this way makes me realize that the IPI approach
> can't work this way at all: It leaves a time window (between 1 and 2)
> where the domain may have vCPU-s running with the wrong COS ID.
> I think pausing the vCPU is unavoidable, unless it can be explained
> that running with the wrong COS ID for a brief period of time is not
> really a problem. I do realize that the pausing approach has its own
> difficulty wrt Dom0, but I think that's solvable (e.g. by doing the
> actual work in a tasklet instead of in the context of a Dom0 vCPU).
>
I have below thoughts.
1. We can use domain_pause for all domains except Dom0 to update COS ID.
2. PSR features are to set cache capacity for a domain. The setting to
cache is progressively effective. When the cache setting becomes really
effective, the time slice to schedule a domain may have passed. Moreover,
even a wrong COS ID is used to set ASSOC, only another CBM be effective
for a short time. In next Dom0 schedule, the correct CBM will take effect.
So can we just leave Dom0 setting as current implementation?
Thanks,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v8 08/24] x86: refactor psr: set value: implement framework.
2017-03-15 8:18 ` Yi Sun
@ 2017-03-15 8:32 ` Jan Beulich
0 siblings, 0 replies; 122+ messages in thread
From: Jan Beulich @ 2017-03-15 8:32 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng
>>> On 15.03.17 at 09:18, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-15 01:40:06, Jan Beulich wrote:
>> >>> On 15.03.17 at 03:52, <yi.y.sun@linux.intel.com> wrote:
>> > Sorry, I may not fully understand your meaning. My thoughts are below.
>> > 1. We set 'd->arch.psr_cos_ids[socket] = cos;' in 'psr_set_val';
>> >
>> > 2. After that, we get valid cpumask through cpupool_domain_cpumask();
>> >
>> > 3. If the actual valid cpumask changed after that, the new cpu is valid so
>> > that the context switch happens. Then 'psr_ctxt_switch_to' is called to
>> > update the new cpu's ASSOC register with the new COS ID which has been
>> > set in step 1.
>> >
>> > 4. Send IPI to all cpus on cpumask got in step 2. They will update their
>> > ASSOC registers according to their domains psr_cos_ids[].
>> >
>> > So I think this flow can cover all cpus which ASSOC registers need be
>> > updated.
>>
>> You writing it down this way makes me realize that the IPI approach
>> can't work this way at all: It leaves a time window (between 1 and 2)
>> where the domain may have vCPU-s running with the wrong COS ID.
>> I think pausing the vCPU is unavoidable, unless it can be explained
>> that running with the wrong COS ID for a brief period of time is not
>> really a problem. I do realize that the pausing approach has its own
>> difficulty wrt Dom0, but I think that's solvable (e.g. by doing the
>> actual work in a tasklet instead of in the context of a Dom0 vCPU).
>>
> I have below thoughts.
> 1. We can use domain_pause for all domains except Dom0 to update COS ID.
> 2. PSR features are to set cache capacity for a domain. The setting to
> cache is progressively effective. When the cache setting becomes really
> effective, the time slice to schedule a domain may have passed. Moreover,
> even a wrong COS ID is used to set ASSOC, only another CBM be effective
> for a short time. In next Dom0 schedule, the correct CBM will take effect.
> So can we just leave Dom0 setting as current implementation?
If at all possible I'd like to avoid special casing Dom0. Please keep in
mind that in disaggregated environments domctl-s may be issued by
other than Dom0. So either the argumentation above to use the
current implementation is okay for all domains (and is reasonable to
expect to hold for possible future extensions), or all domains need
to be taken care of by a changed approach.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 122+ messages in thread
end of thread, other threads:[~2017-03-15 8:32 UTC | newest]
Thread overview: 122+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-15 8:49 [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-02-15 8:49 ` [PATCH v8 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-02-15 16:49 ` Konrad Rzeszutek Wilk
2017-02-26 17:40 ` Wei Liu
2017-02-15 8:49 ` [PATCH v8 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
2017-02-26 17:40 ` Wei Liu
2017-02-15 8:49 ` [PATCH v8 03/24] x86: refactor psr: implement main data structures Yi Sun
2017-02-28 11:58 ` Roger Pau Monné
2017-03-01 5:10 ` Yi Sun
2017-03-01 8:17 ` Jan Beulich
2017-03-01 8:28 ` Yi Sun
2017-03-01 8:39 ` Jan Beulich
2017-03-01 8:49 ` Roger Pau Monn�
2017-03-01 8:54 ` Jan Beulich
2017-03-01 9:00 ` Roger Pau Monn�
2017-02-15 8:49 ` [PATCH v8 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
2017-02-26 17:41 ` Wei Liu
2017-02-27 6:42 ` Yi Sun
2017-02-27 11:45 ` Wei Liu
2017-02-27 8:41 ` Jan Beulich
2017-03-08 14:56 ` Jan Beulich
2017-03-10 1:32 ` Yi Sun
2017-03-10 8:56 ` Jan Beulich
2017-03-13 2:18 ` Yi Sun
2017-02-15 8:49 ` [PATCH v8 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
2017-02-26 17:41 ` Wei Liu
2017-03-08 15:04 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 06/24] x86: refactor psr: implement get hw info flow Yi Sun
2017-02-26 17:41 ` Wei Liu
2017-02-28 12:34 ` Roger Pau Monné
2017-03-08 15:15 ` Jan Beulich
2017-03-10 1:43 ` Yi Sun
2017-03-10 8:57 ` Jan Beulich
2017-03-10 9:01 ` Yi Sun
2017-02-15 8:49 ` [PATCH v8 07/24] x86: refactor psr: implement get value flow Yi Sun
2017-02-28 12:44 ` Roger Pau Monné
2017-03-01 5:21 ` Yi Sun
2017-03-08 15:35 ` Jan Beulich
2017-03-10 1:50 ` Yi Sun
2017-03-10 9:05 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 08/24] x86: refactor psr: set value: implement framework Yi Sun
2017-02-26 17:41 ` Wei Liu
2017-02-27 7:06 ` Yi Sun
2017-02-27 10:55 ` Jan Beulich
2017-02-28 13:58 ` Roger Pau Monné
2017-03-01 6:23 ` Yi Sun
2017-03-08 16:07 ` Jan Beulich
2017-03-10 2:54 ` Yi Sun
2017-03-10 9:09 ` Jan Beulich
2017-03-13 2:36 ` Yi Sun
2017-03-13 12:35 ` Jan Beulich
2017-03-14 2:43 ` Yi Sun
2017-03-14 6:29 ` Jan Beulich
2017-03-14 9:21 ` Yi Sun
2017-03-14 10:24 ` Jan Beulich
2017-03-15 2:52 ` Yi Sun
2017-03-15 7:40 ` Jan Beulich
2017-03-15 8:18 ` Yi Sun
2017-03-15 8:32 ` Jan Beulich
2017-03-10 7:46 ` Yi Sun
2017-03-10 9:10 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-02-27 7:11 ` Yi Sun
2017-02-27 11:45 ` Wei Liu
2017-03-08 16:54 ` Jan Beulich
2017-03-10 3:21 ` Yi Sun
2017-03-10 9:15 ` Jan Beulich
2017-03-13 2:43 ` Yi Sun
2017-03-13 12:37 ` Jan Beulich
2017-03-14 2:20 ` Yi Sun
2017-03-14 6:32 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-02-27 7:16 ` Yi Sun
2017-03-08 17:03 ` Jan Beulich
2017-03-10 5:35 ` Yi Sun
2017-03-10 9:21 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-03-09 14:10 ` Jan Beulich
2017-03-10 5:40 ` Yi Sun
2017-03-10 9:24 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 12/24] x86: refactor psr: set value: implement write msr flow Yi Sun
2017-02-15 8:49 ` [PATCH v8 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
2017-02-28 14:52 ` Roger Pau Monné
2017-03-09 14:53 ` Jan Beulich
2017-03-10 5:50 ` Yi Sun
2017-02-15 8:49 ` [PATCH v8 14/24] x86: refactor psr: implement get hw info " Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-02-28 14:54 ` Roger Pau Monné
2017-02-15 8:49 ` [PATCH v8 15/24] x86: refactor psr: implement get value " Yi Sun
2017-02-28 14:59 ` Roger Pau Monné
2017-02-15 8:49 ` [PATCH v8 16/24] x86: refactor psr: implement set value callback functions " Yi Sun
2017-02-26 17:43 ` Wei Liu
2017-02-27 7:19 ` Yi Sun
2017-02-15 8:49 ` [PATCH v8 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
2017-02-28 15:15 ` Roger Pau Monné
2017-03-01 6:35 ` Yi Sun
2017-03-09 15:04 ` Jan Beulich
2017-03-10 5:52 ` Yi Sun
2017-02-15 8:49 ` [PATCH v8 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
2017-02-28 15:18 ` Roger Pau Monné
2017-03-09 15:13 ` Jan Beulich
2017-03-10 5:57 ` Yi Sun
2017-03-10 9:26 ` Jan Beulich
2017-02-15 8:49 ` [PATCH v8 19/24] x86: L2 CAT: implement get value flow Yi Sun
2017-02-28 15:20 ` Roger Pau Monné
2017-02-15 8:49 ` [PATCH v8 20/24] x86: L2 CAT: implement set " Yi Sun
2017-02-28 15:25 ` Roger Pau Monné
2017-03-01 6:59 ` Yi Sun
2017-03-01 11:31 ` Dario Faggioli
2017-02-15 8:49 ` [PATCH v8 21/24] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
2017-02-15 8:49 ` [PATCH v8 22/24] tools: L2 CAT: support show cbm " Yi Sun
2017-02-15 8:49 ` [PATCH v8 23/24] tools: L2 CAT: support set " Yi Sun
2017-02-15 8:49 ` [PATCH v8 24/24] docs: add L2 CAT description in docs Yi Sun
2017-02-15 16:14 ` [PATCH v8 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Konrad Rzeszutek Wilk
2017-02-26 18:00 ` Wei Liu
2017-02-28 11:02 ` Roger Pau Monné
2017-03-01 4:54 ` Yi Sun
2017-03-01 8:35 ` Roger Pau Monn�
2017-03-01 8:40 ` Yi Sun
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.