* [PATCH v13 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
@ 2017-07-06 1:52 ` Yi Sun
2017-07-07 16:37 ` Meng Xu
2017-07-06 1:52 ` [PATCH v13 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
` (21 subsequent siblings)
22 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:52 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch creates CAT and CDP feature document in doc/features/. It describes
key points to implement L3 CAT/CDP and L2 CAT which is described in details in
Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES".
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v11:
- modify data structure descriptions according to code changes.
- modify revision.
v9:
- add 'CMT' to the list of acronyms.
(suggested by Wei Liu)
- change feature list to feature array.
- modify data structure descriptions according to latest codes.
- modify revision.
v8:
- change revision info.
(suggested by Konrad Rzeszutek Wilk)
- add content int 'Areas for improvement'.
(suggested by Konrad Rzeszutek Wilk)
v7:
- correct typo.
(suggested by Konrad Rzeszutek Wilk)
- replace application/VM to domain.
(suggested by Konrad Rzeszutek Wilk)
- amend description of `feat_mask` to make it clearer.
(suggested by Konrad Rzeszutek Wilk)
- update revision.
(suggested by Konrad Rzeszutek Wilk)
- other minor fixes.
(suggested by Konrad Rzeszutek Wilk)
v6:
- write a new feature document to cover L3 CAT/CDP and L2 CAT.
(suggested by Kevin Tian)
- adjust 'Terminology' position in document.
(suggested by Dario Faggioli)
- fix wordings.
(suggested by Dario Faggioli, Kevin Tian and Konrad Rzeszutek Wilk)
- add SDM chapter title in commit message.
(suggested by Konrad Rzeszutek Wilk)
- add more explanations.
(suggested by Kevin Tian)
v4:
- change file name to be more descriptive, 'intel_psr_l2_cat.pandoc'.
(suggested by Dario Faggioli)
- remove 'Ext' and 'New' prefixes.
(suggested by Dario Faggioli)
- remove change log in Revison part.
(suggested by Dario Faggioli)
- adjust Xen release number to 4.9 to show this feature targets 4.9.
(suggested by Dario Faggioli)
- provide 'Terminology' and more sections.
(suggested by Dario Faggioli)
- fix wordings.
(suggested by Konrad Rzeszutek Wilk)
- remove chapter number.
(suggested by Konrad Rzeszutek Wilk)
v3:
- make design document be a patch.
(suggested by Konrad Rzeszutek Wilk)
v2:
- provide chapter for the L2 CAT.
(suggested by Meng Xu)
---
docs/features/intel_psr_cat_cdp.pandoc | 461 +++++++++++++++++++++++++++++++++
1 file changed, 461 insertions(+)
create mode 100644 docs/features/intel_psr_cat_cdp.pandoc
diff --git a/docs/features/intel_psr_cat_cdp.pandoc b/docs/features/intel_psr_cat_cdp.pandoc
new file mode 100644
index 0000000..79b4739
--- /dev/null
+++ b/docs/features/intel_psr_cat_cdp.pandoc
@@ -0,0 +1,461 @@
+% Intel Cache Allocation Technology and Code and Data Prioritization Features
+% Revision 1.11
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+ Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+ Component(s): Hypervisor, toolstack
+
+ Hardware: L3 CAT: Haswell and beyond CPUs
+ CDP : Broadwell and beyond CPUs
+ L2 CAT: Atom codename Goldmont and beyond CPUs
+---------------- ----------------------------------------------------
+
+# Terminology
+
+* CAT Cache Allocation Technology
+* CBM Capacity BitMasks
+* CDP Code and Data Prioritization
+* CMT Cache Monitoring Technology
+* COS/CLOS Class of Service
+* MSRs Machine Specific Registers
+* PSR Intel Platform Shared Resource
+
+# Overview
+
+Intel provides a set of allocation capabilities including Cache Allocatation
+Technology (CAT) and Code and Data Prioritization (CDP).
+
+CAT allows an OS or hypervisor to control allocation of a CPU's shared cache
+based on application/domain priority or Class of Service (COS). Each COS is
+configured using capacity bitmasks (CBMs) which represent cache capacity and
+indicate the degree of overlap and isolation between classes. Once CAT is co-
+nfigured, the processor allows access to portions of cache according to the
+established COS. Intel Xeon processor E5 v4 family (and some others) introduce
+capabilities to configure and make use of the CAT mechanism on the L3 cache.
+Intel Goldmont processor provides support for control over the L2 cache.
+
+Code and Data Prioritization (CDP) Technology is an extension of CAT. CDP
+enables isolation and separate prioritization of code and data fetches to
+the L3 cache in a SW configurable manner, which can enable workload priorit-
+ization and tuning of cache capacity to the characteristics of the workload.
+CDP extends CAT by providing separate code and data masks per Class of Service
+(COS). When SW configures to enable CDP, L3 CAT is disabled.
+
+# User details
+
+* Feature Enabling:
+
+ Add "psr=cat" to boot line parameter to enable all supported level CAT featu-
+ res. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW.
+
+* xl interfaces:
+
+ 1. `psr-cat-show [OPTIONS] domain-id`:
+
+ Show L2 CAT or L3 CAT/CDP CBM of the domain designated by Xen domain-id.
+
+ Option `-l`:
+ `-l2`: Show cbm for L2 cache.
+ `-l3`: Show cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ 2. `psr-cat-set [OPTIONS] domain-id cbm`:
+
+ Set L2 CAT or L3 CAT/CDP CBM to the domain designated by Xen domain-id.
+
+ Option `-s`: Specify the socket to process, otherwise all sockets are
+ processed.
+
+ Option `-l`:
+ `-l2`: Specify cbm for L2 cache.
+ `-l3`: Specify cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ Option `-c` or `-d`:
+ `-c`: Set L3 CDP code cbm.
+ `-d`: Set L3 CDP data cbm.
+
+ 3. `psr-hwinfo [OPTIONS]`:
+
+ Show CMT & L2 CAT & L3 CAT/CDP HW information on every socket.
+
+ Option `-m, --cmt`: Show Cache Monitoring Technology (CMT) hardware info.
+
+ Option `-a, --cat`: Show CAT/CDP hardware info.
+
+# Technical details
+
+L3 CAT/CDP and L2 CAT are all members of Intel PSR features, they share the base
+PSR infrastructure in Xen.
+
+## Hardware perspective
+
+ CAT/CDP defines a range of MSRs to assign different cache access patterns
+ which are known as CBMs, each CBM is associated with a COS.
+
+ ```
+ E.g. L2 CAT:
+ +----------------------------+----------------+
+ IA32_PQR_ASSOC | MSR (per socket) | Address |
+ +----+---+-------+ +----------------------------+----------------+
+ | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
+ +----+---+-------+ +----------------------------+----------------+
+ └-------------> | ... | ... |
+ +----------------------------+----------------+
+ | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
+ +----------------------------+----------------+
+ ```
+
+ L3 CAT/CDP uses a range of MSRs from 0xC90 ~ 0xC90+n (n<128).
+
+ L2 CAT uses a range of MSRs from 0xD10 ~ 0xD10+n (n<64), following the L3
+ CAT/CDP MSRs, setting different L2 cache accessing patterns from L3 cache is
+ supported.
+
+ Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the
+ hardware indicating the cache space a domain should be limited to as well as
+ providing an indication of overlap and isolation in the CAT-capable cache from
+ other domains contending for the cache.
+
+ Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
+ note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
+ 0FF0H, 003CH, etc.).
+
+ ```
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Default Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Overlapped Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | | | | | Isolated Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | | |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+ ```
+
+ We can get the CBM length through CPUID. The default value of CBM is calcul-
+ ated by `(1ull << cbm_len) - 1`. That is a fully open bitmask, all ones bitm-
+ ask. The COS[0] always stores the default value without change.
+
+ There is a `IA32_PQR_ASSOC` register which stores the COS ID of the VCPU. HW
+ enforces cache allocation according to the corresponding CBM.
+
+## The relationship between L3 CAT/CDP and L2 CAT
+
+ HW may support all features. By default, CDP is disabled on the processor.
+ If the L3 CAT MSRs are used without enabling CDP, the processor operates in
+ a traditional CAT-only mode. When CDP is enabled:
+ * the CAT mask MSRs are re-mapped into interleaved pairs of mask MSRs for
+ data or code fetches.
+ * the range of COS for CAT is re-indexed, with the lower-half of the COS
+ range available for CDP.
+
+ L2 CAT is independent of L3 CAT/CDP, which means L2 CAT can be enabled while
+ L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are both enabled.
+
+ As a requirement, the bits of CBM of CAT/CDP must be continuous.
+
+ N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same associate
+ register `IA32_PQR_ASSOC`, which means one COS is associated with a pair of
+ L2 CAT CBM and L3 CAT/CDP CBM.
+
+ Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or other
+ PSR features in future). In some cases, a domain is permitted to have a COS
+ that is beyond one (or more) of PSR features but within the others. For
+ instance, let's assume the max COS of L2 CAT is 8 but the max COS of L3
+ CAT is 16, when a domain is assigned 9 as COS, the L3 CAT CBM associated to
+ COS 9 would be enforced, but for L2 CAT, the HW works as default value is
+ set since COS 9 is beyond the max COS (8) of L2 CAT.
+
+## Design Overview
+
+* Core COS/CBM association
+
+ When enforcing CAT/CDP, all cores of domains have the same default COS (COS0)
+ which is associated with the fully open CBM (all ones bitmask) to access all
+ cache. The default COS is used only in hypervisor and is transparent to tool
+ stack and user.
+
+ System administrator can change PSR allocation policy at runtime by tool stack.
+ Since L2 CAT shares COS with L3 CAT/CDP, a COS corresponds to a 2-tuple, like
+ [L2 CBM, L3 CBM] with only-CAT enabled, when CDP is enabled, one COS correspo-
+ nds to a 3-tuple, like [L2 CBM, L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT
+ nor L3 CDP is enabled, things would be easier, one COS corresponds to one L2
+ CBM.
+
+* VCPU schedule
+
+ When context switch happens, the COS of VCPU is written to per-thread MSR
+ `IA32_PQR_ASSOC`, and then hardware enforces cache allocation according to
+ the corresponding CBM.
+
+* Multi-sockets
+
+ Different sockets may have different CAT/CDP capability (e.g. max COS) alth-
+ ough it is consistent on the same socket. So the capability of per-socket CAT/
+ CDP is specified.
+
+ 'psr-cat-set' can set CBM for one domain per socket. On each socket, we main-
+ tain a COS array for all domains. One domain uses one COS at one time. One COS
+ stores the CBM of the domain to work. So, when a VCPU of the domain is migrat-
+ ed from socket 1 to socket 2, it follows configuration on socket 2.
+
+ E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets do-
+ main 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
+ is migrated from socket 1 to 2, the COS ID used is 7, that means 0x3f is the
+ CBM to work for this domain 1 now.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+ 1. Boot line parameter "psr=cat" enables L2 CAT and L3 CAT if hardware suppo-
+ rted. "psr=cdp" enables CDP if hardware supported.
+
+ 2. SYSCTL:
+ - XEN_SYSCTL_PSR_CAT_get_l3_info: Get L3 CAT/CDP information.
+ - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
+
+ 3. DOMCTL:
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: Get L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: Set L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE: Get CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: Set CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA: Get CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: Set CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
+
+* xl interfaces:
+
+ 1. psr-cat-show -lX domain-id
+ Show LX cbm for a domain.
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
+
+ 2. psr-cat-set -lX domain-id cbm
+ Set LX cbm for a domain.
+ => XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
+
+ 3. psr-hwinfo
+ Show PSR HW information, including L3 CAT/CDP/L2 CAT
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info
+
+* Key data structure:
+
+ 1. Feature properties
+
+ ```
+ static const struct feat_props {
+ unsigned int cos_num;
+ enum cbm_type type[PSR_MAX_COS_NUM];
+ bool (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len);
+ void (*write_msr)(unsigned int cos, uint32_t val, enum cbm_type type);
+ } *feat_props[PSR_SOCKET_FEAT_NUM];
+ ```
+
+ Every feature has its own properties, e.g. some data and actions. A
+ feature property pointer array is declared to save every feature's
+ properties.
+
+ - Member `cos_num`
+
+ `cos_num` is the number of COS registers the feature uses, e.g. L3/L2
+ CAT uses 1 register but CDP uses 2 registers.
+
+ - Member `type`
+
+ `type` is an array to save all 'enum cbm_type' values of the feature.
+ It is used with cos_num together to get/write a feature's COS registers
+ values one by one.
+
+ - Member `get_feat_info`
+
+ `get_feat_info` is used to return feature HW info through sysctl.
+
+ - Member `write_msr`
+
+ `write_msr` is used to write out feature MSR register.
+
+ 2. Feature node
+
+ ```
+ struct feat_node {
+ unsigned int cos_max;
+ unsigned int cbm_len;
+ uint32_t cos_reg_val[MAX_COS_REG_CNT];
+ };
+ ```
+
+ When a PSR enforcement feature is enabled, it will be added into a
+ feature array.
+
+ - Member `cos_max`
+
+ `cos_max` is one of the hardware info of CAT. It means the max number
+ of COS registers. As L3 CAT/CDP/L2 CAT all have it, it is declared in
+ `feat_node`.
+
+ - Member `cbm_len`
+
+ `cbm_len` is one of the hardware info of CAT. It means the max number
+ of bits to set.
+
+ - Member `cos_reg_val`
+
+ `cos_reg_val` is an array to maintain the value set in all COS registers
+ of the feature. The array is indexed by COS ID.
+
+ 3. Per-socket PSR features information structure
+
+ ```
+ struct psr_socket_info {
+ bool feat_init;
+ struct feat_node *features[PSR_SOCKET_FEAT_NUM];
+ spinlock_t ref_lock;
+ unsigned int cos_ref[MAX_COS_REG_CNT];
+ DECLARE_BITMAP(dom_ids, DOMID_IDLE + 1);
+ };
+ ```
+
+ We collect all PSR allocation features information of a socket in this
+ `struct psr_socket_info`.
+
+ - Member `feat_init`
+
+ `feat_init` is a flag, to indicate whether the CPU init on a socket
+ has been done.
+
+ - Member `features`
+
+ `features` is a pointer array to save all enabled features poniters
+ according to feature position defined in `enum psr_feat_type`.
+
+ - Member `ref_lock`
+
+ `ref_lock` is a spin lock to protect `cos_ref`.
+
+ - Member `cos_ref`
+
+ `cos_ref` is an array which maintains the reference of one COS. It maps
+ to cos_reg_val[MAX_COS_REG_NUM] in `struct feat_node`. If one COS is
+ used by one domain, the corresponding reference will increase by one. If
+ a domain releases the COS, the reference will decrease by one. The array
+ is indexed by COS ID.
+
+ - Member `dom_ids`
+
+ `dom_ids` is a bitmap, every bit corresponds to a domain. Index is
+ domain_id. It is used to help restore domain_id to 0 when a socket is
+ offline and then online again.
+
+# Limitations
+
+CAT/CDP can only work on HW which enables it(check by CPUID). So far, there is
+no HW which enables both L2 CAT and L3 CAT/CDP. But SW implementation has cons-
+idered such scenario to enable both L2 CAT and L3 CAT/CDP.
+
+# Testing
+
+We can execute above xl commands to verify L2 CAT and L3 CAT/CDP on different
+HWs support them.
+
+For example:
+ root@:~$ xl psr-hwinfo --cat
+ Cache Allocation Technology (CAT): L2
+ Socket ID : 0
+ Maximum COS : 3
+ CBM length : 8
+ Default CBM : 0xff
+
+ root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
+
+ root@:~$ xl psr-cat-show -l2 1
+ Socket ID : 0
+ Default CBM : 0xff
+ ID NAME CBM
+ 1 ubuntu14 0x7f
+
+# Areas for improvement
+
+A hexadecimal number is used to set/show CBM for a domain now. Although this
+is convenient to cover overlap/isolated bitmask requirement, it is not user-
+friendly.
+
+To improve this, the libxl interfaces can be wrapped in libvirt to provide more
+usr-friendly interfaces to user, e.g. a percentage number of the cache to set
+and show.
+
+# Known issues
+
+N/A
+
+# References
+
+"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date Revision Version Notes
+---------- -------- -------- -------------------------------------------
+2016-08-12 1.0 Xen 4.9 Design document written
+2017-02-13 1.7 Xen 4.9 Changes:
+ 1. Modify the design document to cover L3
+ CAT/CDP and L2 CAT;
+ 2. Fix typos;
+ 3. Amend description of `feat_mask` to make
+ it clearer;
+ 4. Other minor changes.
+2017-02-15 1.8 Xen 4.9 Changes:
+ 1. Add content in 'Areas for improvement';
+ 2. Adjust revision number.
+2017-03-16 1.9 Xen 4.9 Changes:
+ 1. Add 'CMT' in 'Terminology';
+ 2. Change 'feature list' to 'feature array'.
+ 3. Modify data structure descriptions.
+ 4. Adjust revision number.
+2017-05-03 1.11 Xen 4.9 Changes:
+ 1. Modify data structure descriptions.
+ 2. Adjust revision number.
+---------- -------- -------- -------------------------------------------
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v13 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-07-06 1:52 ` [PATCH v13 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
@ 2017-07-07 16:37 ` Meng Xu
2017-07-10 5:25 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Meng Xu @ 2017-07-07 16:37 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, Wei Liu, Andrew Cooper, Dario Faggioli, Ian Jackson,
Jan Beulich, Chao Peng, xen-devel, roger.pau
[I just add some of my thoughts when I read this document.]
> +## Hardware perspective
> +
> + CAT/CDP defines a range of MSRs to assign different cache access patterns
> + which are known as CBMs, each CBM is associated with a COS.
> +
> + ```
> + E.g. L2 CAT:
> + +----------------------------+----------------+
> + IA32_PQR_ASSOC | MSR (per socket) | Address |
> + +----+---+-------+ +----------------------------+----------------+
> + | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
> + +----+---+-------+ +----------------------------+----------------+
> + └-------------> | ... | ... |
> + +----------------------------+----------------+
> + | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
> + +----------------------------+----------------+
> + ```
> +
> + L3 CAT/CDP uses a range of MSRs from 0xC90 ~ 0xC90+n (n<128).
> +
> + L2 CAT uses a range of MSRs from 0xD10 ~ 0xD10+n (n<64), following the L3
> + CAT/CDP MSRs, setting different L2 cache accessing patterns from L3 cache is
> + supported.
> +
> + Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the
> + hardware indicating the cache space a domain should be limited to as well as
> + providing an indication of overlap and isolation in the CAT-capable cache from
> + other domains contending for the cache.
> +
> + Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
> + note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
> + 0FF0H, 003CH, etc.).
IIRC, the number of contiguous '1's in CBM should be at least 2 at
least on my machine (Intel(R) Xeon(R) CPU E5-2618L v3).
I'm unsure if this constraint exist for all CAT-capable processors.
For those processors that have such constraint, the system may crash
when the user sets only 1 bit to the CBM.
> + 3. Per-socket PSR features information structure
> +
> + ```
> + struct psr_socket_info {
> + bool feat_init;
> + struct feat_node *features[PSR_SOCKET_FEAT_NUM];
> + spinlock_t ref_lock;
> + unsigned int cos_ref[MAX_COS_REG_CNT];
> + DECLARE_BITMAP(dom_ids, DOMID_IDLE + 1);
> + };
> + ```
> +
> + We collect all PSR allocation features information of a socket in this
> + `struct psr_socket_info`.
> +
> + - Member `feat_init`
> +
> + `feat_init` is a flag, to indicate whether the CPU init on a socket
> + has been done.
> +
> + - Member `features`
> +
> + `features` is a pointer array to save all enabled features poniters
> + according to feature position defined in `enum psr_feat_type`.
> +
> + - Member `ref_lock`
> +
> + `ref_lock` is a spin lock to protect `cos_ref`.
> +
> + - Member `cos_ref`
> +
> + `cos_ref` is an array which maintains the reference of one COS. It maps
> + to cos_reg_val[MAX_COS_REG_NUM] in `struct feat_node`. If one COS is
> + used by one domain, the corresponding reference will increase by one. If
> + a domain releases the COS, the reference will decrease by one. The array
> + is indexed by COS ID.
> +
> + - Member `dom_ids`
> +
> + `dom_ids` is a bitmap, every bit corresponds to a domain. Index is
> + domain_id. It is used to help restore domain_id to 0 when a socket is
> + offline and then online again.
Did you mean "it is used to help restore domain_id to COS0, which has
all 1s in CBM, when a socket is offline and then online again."
Best,
Meng
-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-07-07 16:37 ` Meng Xu
@ 2017-07-10 5:25 ` Yi Sun
2017-07-10 13:09 ` Meng Xu
0 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-10 5:25 UTC (permalink / raw)
To: Meng Xu
Cc: kevin.tian, Wei Liu, Andrew Cooper, Dario Faggioli, Ian Jackson,
Jan Beulich, Chao Peng, xen-devel, roger.pau
On 17-07-07 12:37:28, Meng Xu wrote:
> > + Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
> > + note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
> > + 0FF0H, 003CH, etc.).
>
> IIRC, the number of contiguous '1's in CBM should be at least 2 at
> least on my machine (Intel(R) Xeon(R) CPU E5-2618L v3).
> I'm unsure if this constraint exist for all CAT-capable processors.
> For those processors that have such constraint, the system may crash
> when the user sets only 1 bit to the CBM.
>
It seems your machine does not officially support CAT. Per my info, some
machines, e.g. Haswell, do not officially support CAT but you can enable
CAT through some actions. On these machines, you may encounter such issue.
Per SDM, we do not have such limitation. Per my test on SKL, 1 bit setting
works well.
>
> > + - Member `dom_ids`
> > +
> > + `dom_ids` is a bitmap, every bit corresponds to a domain. Index is
> > + domain_id. It is used to help restore domain_id to 0 when a socket is
> > + offline and then online again.
>
> Did you mean "it is used to help restore domain_id to COS0, which has
> all 1s in CBM, when a socket is offline and then online again."
>
Sorry, a typo here, should be:
"It is used to help restore 'd->arch.psr_cos_ids[socket]' to 0 when a socket is
offline and then online again."
If you think it is still not clear, I may add your explanation:
", which has all 1s in CBM, "
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v13 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-07-10 5:25 ` Yi Sun
@ 2017-07-10 13:09 ` Meng Xu
0 siblings, 0 replies; 42+ messages in thread
From: Meng Xu @ 2017-07-10 13:09 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, Wei Liu, Andrew Cooper, Dario Faggioli, Ian Jackson,
Jan Beulich, Chao Peng, xen-devel, roger.pau
On Mon, Jul 10, 2017 at 1:25 AM, Yi Sun <yi.y.sun@linux.intel.com> wrote:
> On 17-07-07 12:37:28, Meng Xu wrote:
>> > + Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
>> > + note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
>> > + 0FF0H, 003CH, etc.).
>>
>> IIRC, the number of contiguous '1's in CBM should be at least 2 at
>> least on my machine (Intel(R) Xeon(R) CPU E5-2618L v3).
>> I'm unsure if this constraint exist for all CAT-capable processors.
>> For those processors that have such constraint, the system may crash
>> when the user sets only 1 bit to the CBM.
>>
> It seems your machine does not officially support CAT. Per my info, some
> machines, e.g. Haswell, do not officially support CAT but you can enable
> CAT through some actions. On these machines, you may encounter such issue.
>
> Per SDM, we do not have such limitation. Per my test on SKL, 1 bit setting
> works well.
I see.
IIRC, Xen CAT won't be enabled on my haswell machine. As long as all
those machines that does not support 1 bit setting are not enabled by
default, that is fine. Otherwise, this may cause a problem.
>
>>
>> > + - Member `dom_ids`
>> > +
>> > + `dom_ids` is a bitmap, every bit corresponds to a domain. Index is
>> > + domain_id. It is used to help restore domain_id to 0 when a socket is
>> > + offline and then online again.
>>
>> Did you mean "it is used to help restore domain_id to COS0, which has
>> all 1s in CBM, when a socket is offline and then online again."
>>
> Sorry, a typo here, should be:
> "It is used to help restore 'd->arch.psr_cos_ids[socket]' to 0 when a socket is
> offline and then online again."
I think this is more clear.
Another statement could be:
It is used to help restore the cos_id of the domain_id to 0 when a
socket is offline and then online again.
>
> If you think it is still not clear, I may add your explanation:
> ", which has all 1s in CBM, "
If you need to send another version, I think it's better to correct it.
Thanks,
Meng
-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v13 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-07-06 1:52 ` [PATCH v13 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
@ 2017-07-06 1:52 ` Yi Sun
2017-07-06 1:52 ` [PATCH v13 03/23] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
` (20 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:52 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch moves 'cpuid_count_leaf' from cpuid.c to processor.h to
make it available to external codes.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
v9:
- create this patch alone to move 'cpuid_count_leaf'.
(suggested by Wei Liu)
v6:
- use 'struct cpuid_leaf' in psr.c. So we have to access 'cpuid_count_leaf'
which has to be moved to processor.h.
(suggested by Andrew Cooper)
---
xen/arch/x86/cpuid.c | 6 ------
xen/include/asm-x86/processor.h | 7 +++++++
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/cpuid.c b/xen/arch/x86/cpuid.c
index d359e09..f1a6e9f 100644
--- a/xen/arch/x86/cpuid.c
+++ b/xen/arch/x86/cpuid.c
@@ -35,12 +35,6 @@ static void cpuid_leaf(uint32_t leaf, struct cpuid_leaf *data)
cpuid(leaf, &data->a, &data->b, &data->c, &data->d);
}
-static void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
- struct cpuid_leaf *data)
-{
- cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
-}
-
static void sanitise_featureset(uint32_t *fs)
{
/* for_each_set_bit() uses unsigned longs. Extend with zeroes. */
diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h
index 6a335d3..63a836d 100644
--- a/xen/include/asm-x86/processor.h
+++ b/xen/include/asm-x86/processor.h
@@ -13,6 +13,7 @@
#include <asm/types.h>
#include <asm/cpufeature.h>
#include <asm/desc.h>
+#include <asm/x86_emulate.h>
#endif
#include <asm/x86-defns.h>
@@ -264,6 +265,12 @@ static always_inline unsigned int cpuid_count_ebx(
return ebx;
}
+static always_inline void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
+ struct cpuid_leaf *data)
+{
+ cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
+}
+
static inline unsigned long read_cr0(void)
{
unsigned long cr0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 03/23] x86: refactor psr: remove L3 CAT/CDP codes.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-07-06 1:52 ` [PATCH v13 01/23] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-07-06 1:52 ` [PATCH v13 02/23] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
@ 2017-07-06 1:52 ` Yi Sun
2017-07-06 1:52 ` [PATCH v13 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
` (19 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:52 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
The current cache allocation codes in psr.c do not consider
future features addition and are not friendly to extend.
To make psr.c be more flexible to add new features and fulfill
the program principle, open for extension but closed for
modification, we have to refactor the psr.c:
1. Analyze cache allocation features and abstract general data
structures.
2. Analyze the init and all other functions flow, abstract all
steps that different features may have different implementations.
Make these steps be callback functions and register feature
specific fuctions. Then, the main processes will not be changed
when introducing a new feature.
Because the quantity of refactor codes is big and the logics are
changed a lot, it will cause reviewers confused if just change
old codes. Reviewers have to understand both old codes and new
implementations. After review iterations from V1 to V3, Jan has
proposed to remove all old cache allocation codes firstly, then
implement new codes step by step. This will help to make codes
be more easily reviewable.
There is no construction without destruction. So, this patch
removes all current L3 CAT/CDP codes in psr.c. The following
patches will introduce the new mechanism.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v4:
- create this patch to make codes easily understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 470 +----------------------------------------------------
1 file changed, 5 insertions(+), 465 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0b5073c..96a8589 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -23,24 +23,6 @@
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
-struct psr_cat_cbm {
- union {
- uint64_t cbm;
- struct {
- uint64_t code;
- uint64_t data;
- };
- };
- unsigned int ref;
-};
-
-struct psr_cat_socket_info {
- unsigned int cbm_len;
- unsigned int cos_max;
- struct psr_cat_cbm *cos_to_cbm;
- spinlock_t cbm_lock;
-};
-
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -48,26 +30,11 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
-static unsigned long *__read_mostly cat_socket_enable;
-static struct psr_cat_socket_info *__read_mostly cat_socket_info;
-static unsigned long *__read_mostly cdp_socket_enable;
-
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
-static unsigned int __read_mostly opt_cos_max = 255;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
-static struct psr_cat_cbm *temp_cos_to_cbm;
-
-static unsigned int get_socket_cpu(unsigned int socket)
-{
- if ( likely(socket < nr_sockets) )
- return cpumask_any(socket_cpumask[socket]);
-
- return nr_cpu_ids;
-}
-
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -107,9 +74,6 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
- if ( val_str && !strcmp(s, "cos_max") )
- opt_cos_max = simple_strtoul(val_str, NULL, 0);
-
s = ss + 1;
} while ( ss );
}
@@ -213,16 +177,7 @@ static inline void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( cat_socket_info )
- {
- unsigned int socket = cpu_to_socket(smp_processor_id());
-
- if ( test_bit(socket, cat_socket_enable) )
- psra->cos_mask = ((1ull << get_count_order(
- cat_socket_info[socket].cos_max)) - 1) << 32;
- }
-
- if ( psr_cmt_enabled() || psra->cos_mask )
+ if ( psr_cmt_enabled() )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -231,12 +186,6 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
-static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
- uint64_t cos_mask)
-{
- *reg = (*reg & ~cos_mask) | (((uint64_t)cos << 32) & cos_mask);
-}
-
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -245,459 +194,54 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
- if ( psra->cos_mask )
- psr_assoc_cos(®, d->arch.psr_cos_ids ?
- d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
- 0, psra->cos_mask);
-
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
psra->val = reg;
}
}
-static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
-{
- if ( !cat_socket_info )
- return ERR_PTR(-ENODEV);
-
- if ( socket >= nr_sockets )
- return ERR_PTR(-ENOTSOCK);
-
- if ( !test_bit(socket, cat_socket_enable) )
- return ERR_PTR(-ENOENT);
-
- return cat_socket_info + socket;
-}
-
-static inline bool_t cdp_is_enabled(unsigned int socket)
-{
- return cdp_socket_enable && test_bit(socket, cdp_socket_enable);
-}
int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
uint32_t *cos_max, uint32_t *flags)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- *cbm_len = info->cbm_len;
- *cos_max = info->cos_max;
-
- *flags = 0;
- if ( cdp_is_enabled(socket) )
- *flags |= XEN_SYSCTL_PSR_CAT_L3_CDP;
-
return 0;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
- bool_t cdp_enabled = cdp_is_enabled(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- if ( cdp_enabled )
- return -EXDEV;
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].code;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].data;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- }
-
- return 0;
-}
-
-static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
-{
- unsigned int first_bit, zero_bit;
-
- /* Set bits should only in the range of [0, cbm_len). */
- if ( cbm & (~0ull << cbm_len) )
- return 0;
-
- /* At least one bit need to be set. */
- if ( cbm == 0 )
- return 0;
-
- first_bit = find_first_bit(&cbm, cbm_len);
- zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
-
- /* Set bits should be contiguous. */
- if ( zero_bit < cbm_len &&
- find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
- return 0;
-
- return 1;
-}
-
-struct cos_cbm_info
-{
- unsigned int cos;
- bool_t cdp;
- uint64_t cbm_code;
- uint64_t cbm_data;
-};
-
-static void do_write_l3_cbm(void *data)
-{
- struct cos_cbm_info *info = data;
-
- if ( info->cdp )
- {
- wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(info->cos), info->cbm_code);
- wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(info->cos), info->cbm_data);
- }
- else
- wrmsrl(MSR_IA32_PSR_L3_MASK(info->cos), info->cbm_code);
-}
-
-static int write_l3_cbm(unsigned int socket, unsigned int cos,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp)
-{
- struct cos_cbm_info info =
- {
- .cos = cos,
- .cbm_code = cbm_code,
- .cbm_data = cbm_data,
- .cdp = cdp,
- };
-
- if ( socket == cpu_to_socket(smp_processor_id()) )
- do_write_l3_cbm(&info);
- else
- {
- unsigned int cpu = get_socket_cpu(socket);
-
- if ( cpu >= nr_cpu_ids )
- return -ENOTSOCK;
- on_selected_cpus(cpumask_of(cpu), do_write_l3_cbm, &info, 1);
- }
-
return 0;
}
-static int find_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
-{
- unsigned int cos;
-
- for ( cos = 0; cos <= cos_max; cos++ )
- {
- if ( (map[cos].ref || cos == 0) &&
- ((!cdp_enabled && map[cos].cbm == cbm_code) ||
- (cdp_enabled && map[cos].code == cbm_code &&
- map[cos].data == cbm_data)) )
- return cos;
- }
-
- return -ENOENT;
-}
-
-static int pick_avail_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- unsigned int old_cos)
-{
- unsigned int cos;
-
- /* If old cos is referred only by the domain, then use it. */
- if ( map[old_cos].ref == 1 && old_cos != 0 )
- return old_cos;
-
- /* Find an unused one other than cos0. */
- for ( cos = 1; cos <= cos_max; cos++ )
- if ( map[cos].ref == 0 )
- return cos;
-
- return -ENOENT;
-}
-
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type)
{
- unsigned int old_cos, cos_max;
- int cos, ret;
- uint64_t cbm_data, cbm_code;
- bool_t cdp_enabled = cdp_is_enabled(socket);
- struct psr_cat_cbm *map;
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- if ( !psr_check_cbm(info->cbm_len, cbm) )
- return -EINVAL;
-
- if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
- type == PSR_CBM_TYPE_L3_DATA) )
- return -ENXIO;
-
- cos_max = info->cos_max;
- old_cos = d->arch.psr_cos_ids[socket];
- map = info->cos_to_cbm;
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- cbm_code = cbm;
- cbm_data = cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- cbm_code = cbm;
- cbm_data = map[old_cos].data;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- cbm_code = map[old_cos].code;
- cbm_data = cbm;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- return -EINVAL;
- }
-
- spin_lock(&info->cbm_lock);
- cos = find_cos(map, cos_max, cbm_code, cbm_data, cdp_enabled);
- if ( cos >= 0 )
- {
- if ( cos == old_cos )
- {
- spin_unlock(&info->cbm_lock);
- return 0;
- }
- }
- else
- {
- cos = pick_avail_cos(map, cos_max, old_cos);
- if ( cos < 0 )
- {
- spin_unlock(&info->cbm_lock);
- return cos;
- }
-
- /* We try to avoid writing MSR. */
- if ( (cdp_enabled &&
- (map[cos].code != cbm_code || map[cos].data != cbm_data)) ||
- (!cdp_enabled && map[cos].cbm != cbm_code) )
- {
- ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
- if ( ret )
- {
- spin_unlock(&info->cbm_lock);
- return ret;
- }
- map[cos].code = cbm_code;
- map[cos].data = cbm_data;
- }
- }
-
- map[cos].ref++;
- map[old_cos].ref--;
- spin_unlock(&info->cbm_lock);
-
- d->arch.psr_cos_ids[socket] = cos;
-
return 0;
}
-/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
-static void psr_free_cos(struct domain *d)
-{
- unsigned int socket;
- unsigned int cos;
- struct psr_cat_socket_info *info;
-
- if( !d->arch.psr_cos_ids )
- return;
-
- for_each_set_bit(socket, cat_socket_enable, nr_sockets)
- {
- if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
- continue;
-
- info = cat_socket_info + socket;
- spin_lock(&info->cbm_lock);
- info->cos_to_cbm[cos].ref--;
- spin_unlock(&info->cbm_lock);
- }
-
- xfree(d->arch.psr_cos_ids);
- d->arch.psr_cos_ids = NULL;
-}
-
int psr_domain_init(struct domain *d)
{
- if ( cat_socket_info )
- {
- d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
- if ( !d->arch.psr_cos_ids )
- return -ENOMEM;
- }
-
return 0;
}
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
- psr_free_cos(d);
-}
-
-static int cat_cpu_prepare(unsigned int cpu)
-{
- if ( !cat_socket_info )
- return 0;
-
- if ( temp_cos_to_cbm == NULL &&
- (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
- opt_cos_max + 1UL)) == NULL )
- return -ENOMEM;
-
- return 0;
-}
-
-static void cat_cpu_init(void)
-{
- unsigned int eax, ebx, ecx, edx;
- struct psr_cat_socket_info *info;
- unsigned int socket;
- unsigned int cpu = smp_processor_id();
- uint64_t val;
- const struct cpuinfo_x86 *c = cpu_data + cpu;
-
- if ( !cpu_has(c, X86_FEATURE_PQE) || c->cpuid_level < PSR_CPUID_LEVEL_CAT )
- return;
-
- socket = cpu_to_socket(cpu);
- if ( test_bit(socket, cat_socket_enable) )
- return;
-
- cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
- if ( ebx & PSR_RESOURCE_TYPE_L3 )
- {
- cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
- info = cat_socket_info + socket;
- info->cbm_len = (eax & 0x1f) + 1;
- info->cos_max = min(opt_cos_max, edx & 0xffff);
-
- info->cos_to_cbm = temp_cos_to_cbm;
- temp_cos_to_cbm = NULL;
- /* cos=0 is reserved as default cbm(all ones). */
- info->cos_to_cbm[0].cbm = (1ull << info->cbm_len) - 1;
-
- spin_lock_init(&info->cbm_lock);
-
- set_bit(socket, cat_socket_enable);
-
- if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
- cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
- {
- info->cos_to_cbm[0].code = (1ull << info->cbm_len) - 1;
- info->cos_to_cbm[0].data = (1ull << info->cbm_len) - 1;
-
- /* We only write mask1 since mask0 is always all ones by default. */
- wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->cbm_len) - 1);
-
- rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
- wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
-
- /* Cut half of cos_max when CDP is enabled. */
- info->cos_max >>= 1;
-
- set_bit(socket, cdp_socket_enable);
- }
- printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
- socket, info->cos_max, info->cbm_len,
- cdp_is_enabled(socket) ? "on" : "off");
- }
-}
-
-static void cat_cpu_fini(unsigned int cpu)
-{
- unsigned int socket = cpu_to_socket(cpu);
-
- if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
- {
- struct psr_cat_socket_info *info = cat_socket_info + socket;
-
- if ( info->cos_to_cbm )
- {
- xfree(info->cos_to_cbm);
- info->cos_to_cbm = NULL;
- }
-
- if ( cdp_is_enabled(socket) )
- clear_bit(socket, cdp_socket_enable);
-
- clear_bit(socket, cat_socket_enable);
- }
-}
-
-static void __init psr_cat_free(void)
-{
- xfree(cat_socket_enable);
- cat_socket_enable = NULL;
- xfree(cat_socket_info);
- cat_socket_info = NULL;
-}
-
-static void __init init_psr_cat(void)
-{
- if ( opt_cos_max < 1 )
- {
- printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
- return;
- }
-
- cat_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
- cat_socket_info = xzalloc_array(struct psr_cat_socket_info, nr_sockets);
- cdp_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
-
- if ( !cat_socket_enable || !cat_socket_info )
- psr_cat_free();
}
static int psr_cpu_prepare(unsigned int cpu)
{
- return cat_cpu_prepare(cpu);
+ return 0;
}
static void psr_cpu_init(void)
{
- if ( cat_socket_info )
- cat_cpu_init();
-
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
- if ( cat_socket_info )
- cat_cpu_fini(cpu);
+ return;
}
static int cpu_callback(
@@ -738,14 +282,10 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- if ( opt_psr & PSR_CAT )
- init_psr_cat();
-
- if ( psr_cpu_prepare(0) )
- psr_cat_free();
+ psr_cpu_prepare(0);
psr_cpu_init();
- if ( psr_cmt_enabled() || cat_socket_info )
+ if ( psr_cmt_enabled() )
register_cpu_notifier(&cpu_nfb);
return 0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (2 preceding siblings ...)
2017-07-06 1:52 ` [PATCH v13 03/23] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
@ 2017-07-06 1:52 ` Yi Sun
2017-07-06 1:52 ` [PATCH v13 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
` (18 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:52 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
To construct an extendible framework, we need analyze PSR features
and abstract the common things and feature specific things. Then,
encapsulate them into different data structures.
By analyzing PSR features, we can get below map.
+------+------+------+
--------->| Dom0 | Dom1 | ... |
| +------+------+------+
| |
|Dom ID | cos_id of domain
| V
| +-----------------------------------------------------------------------------+
User --------->| PSR |
Socket ID | +--------------+---------------+---------------+ |
| | Socket0 Info | Socket 1 Info | ... | |
| +--------------+---------------+---------------+ |
| | cos_id=0 cos_id=1 ... |
| | +-----------------------+-----------------------+-----------+ |
| |->Ref : | ref 0 | ref 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L3 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L2 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------+-----------+-----------+-----------+-----------+ |
| |->CDP : | cos0 code | cos0 data | cos1 code | cos1 data | ... | |
| +-----------+-----------+-----------+-----------+-----------+ |
+-----------------------------------------------------------------------------+
So, we need define a socket info data structure, 'struct
psr_socket_info' to manage information per socket. It contains a
reference count array according to COS ID and a feature array to
manage all features enabled. Every entry of the reference count
array is used to record how many domains are using the COS registers
according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
Dom1 uses COS_ID=1 registers of both features to save CBM values, like
below.
+-------+-------+-------+-----+
| COS 0 | COS 1 | COS 2 | ... |
+-------+-------+-------+-----+
L3 CAT | 0x7ff | 0x1ff | ... | ... |
+-------+-------+-------+-----+
L2 CAT | 0xff | 0xff | ... | ... |
+-------+-------+-------+-----+
If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
That means, both Dom1 and Dom2 use same COS registers(ID=1) to keep same
L3/L2 values. So, the value of ref[1] is 2 which means 2 domains are using
COS_ID 1.
To manage a feature, we need define a feature node data structure,
'struct feat_node', to manage feature's specific HW info, and an array of all
COS registers values of this feature.
To manage feature properties, we need define a feature property data structure,
'struct feat_props', to manage common properties (callback functions - all
feature's specific behaviors are encapsulated into these callback functions,
and generic values - e.g. the cos_max), the feature independent values.
CDP is a special feature which uses two entries of the array
for one COS ID. So, the number of CDP COS registers is the half of L3
CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
it is enabled. CDP uses the COS registers array as below.
+-----------+-----------+-----------+-----------+-----------+
CDP cos_reg_val[] index: | 0 | 1 | 2 | 3 | ... |
+-----------+-----------+-----------+-----------+-----------+
value: | cos0 code | cos0 data | cos1 code | cos1 data | ... |
+-----------+-----------+-----------+-----------+-----------+
For more details, please refer SDM and patches to implement 'get value' and
'set value'.
This patch also implements the CPU init and free flow including L3 CAT
initialization and some resources free. It includes below flows:
1. presmp init:
- parse command line parameter.
- allocate socket info for every socket.
- allocate feature resource.
- initialize socket info, get feature info and add feature into feature
array per cpuid result.
- free resources allocated if error happens.
- register cpu notifier to handle cpu events.
2. cpu notifier:
- handle cpu online events, if initialization work has been done before,
do nothing.
- handle cpu offline events, if it is the last cpu offline, free some
socket resources.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v13:
- modify macros and enum identifiers names to be consistent.
(suggested by Jan Beulich)
- correct comment.
(suggested by Jan Beulich)
- move 'cos_ref' array clearning into this patch.
(suggested by Jan Beulich)
- add 'alt_type' in 'feat_props' to handle some special operation in the
future.
(suggested by Jan Beulich)
- move 'feat_init' out from the statement.
(suggested by Jan Beulich)
- change 'cat_init_feat' type to 'int' to return error back if something
wrong. Then, we will not add feat_props into list.
v12:
- add comment for 'feat_init'.
(suggested by Jan Beulich)
- use 'ARRAY_SIZE()' to check array boundary.
(suggested by Jan Beulich)
- use 'XENLOG_WARNING' for error message.
(suggested by Jan Beulich)
- move 'type[]' assignment for feat_props object into its delcaration
to make the object be 'const'.
(suggested by Jan Beulich)
- remove "L3" and "L2" indication in printk.
(suggested by Jan Beulich)
- move position of 'ref_lock' definition.
- restore mask(0) MSR to default value.
(suggested by Jan Beulich)
v11:
- handle 'feat_init'.
- merge main data structures implementation into CPU init/free patch,
including commit messages and change history.
(suggested by Jan Beulich)
- remove MSR restore action which is unnecessary.
(suggested by Jan Beulich)
- move 'type[]' declaration into this patch.
(suggested by Jan Beulich)
- modify comment.
(suggested by Jan Beulich)
v10:
- remove initialization for 'PSR_SOCKET_L3_CAT'.
(suggested by Jan Beulich)
- rename 'feat_ops' to 'feat_props'.
(suggested by Jan Beulich)
- move 'cbm_len' to 'feat_props' because it is feature independent so far.
(suggested by Jan Beulich)
- move 'cos_max' to 'feat_props' because it is feature independent.
(suggested by Jan Beulich)
- move 'cos_num' to 'feat_props' because it is feature independent.
(suggested by Jan Beulich)
- remove union 'info' and struct 'psr_cat_hw_info'.
- remove 'get_cos_max' from 'feat_props'.
(suggested by Jan Beulich)
- remove 'feat_mask' from 'psr_socket_info' because we can use 'features[]'
to check if any feature is initialized.
(suggested by Jan Beulich)
- move 'ref_lock' above 'cos_ref'.
(suggested by Jan Beulich)
- adjust comments and commit message according to above changes.
- remove 'asm/x86_emulate.h' inclusion as it has been indirectly included.
(suggested by Jan Beulich)
- remove 'CAT_COS_NUM' as it is only used once.
(suggested by Jan Beulich)
- remove 'feat_mask'.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- remove 'get_cos_max' hook declaration.
(suggested by Jan Beulich)
- modify 'cat_default_val' implementation.
(suggested by Jan Beulich)
- modify 'psr_alloc_feat_enabled' implementation to make it simple.
(suggested by Jan Beulich)
- rename 'free_feature' to 'free_socket_resources' because it is executed
when socket is offline. It needs free resources related to the socket.
(suggested by Jan Beulich)
- define 'feat_init_done' to iterate feature array to check if any feature
has been initialized.
(suggested by Jan Beulich)
- input 'struct cpuid_leaf' pointer into 'cat_init_feature' to avoid memory
copy.
(suggested by Jan Beulich)
- modify 'cat_init_feature' to use switch and things related to above
changes.
(suggested by Jan Beulich)
- add an indentation for label.
(suggested by Jan Beulich)
v9:
- replace feature list to a feature pointer array.
(suggested by Roger Pau)
- add 'PSR_SOCKET_MAX_FEAT' in 'enum psr_feat_type' to know features
account.
(suggested by Roger Pau)
- move 'feat_ops' declaration into 'feat_node' structure.
(suggested by Roger Pau)
- directly use uninon for feature HW info and move its declaration into
'feat_node' structure.
(suggested by Roger Pau)
- remove 'enum psr_feat_type feature' declared in 'feat_ops' because it is
not useful after using feature pointer array.
(suggested by Roger Pau)
- rename 'l3_cat_info' to 'cat_info' to be used by all CAT/CDP features.
- remove 'nr_feat' which is only for a record.
(suggested by Jan Beulich)
- add 'cos_num' to record how many COS registers are used by a feature in
one time access.
(suggested by Jan Beulich)
- replace 'uint64_t' to 'uint32_t' for cbm value because SDM specifies the
max 32 bits for it.
(suggested by Jan Beulich)
- add commit message to explain the flows.
- handle cpu offline and online again case to read MSRs registers values
back and save them into cos array to make user can get real data.
- create a new patch about moving 'cpuid_count_leaf'.
(suggested by Wei Liu)
- modify comment to explain why not free some resource in 'free_feature'.
(suggested by Wei Liu)
- implement 'psr_alloc_feat_enabled' to check if allocation feature is
enabled in cmdline and some initialization work done.
(suggested by Wei Liu)
- implement 'cat_default_val' to set default value for CAT features.
(suggested by Wei Liu)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- implement a common 'cat_init_feature' to replace L3 CAT/L2 CAT specific
init functions.
(suggested by Roger Pau)
- modify comments for global feature node.
(suggested by Jan Beulich)
- remove unnecessary comments.
(suggested by Jan Beulich)
- remove unnecessary 'else'.
(suggested by Jan Beulich)
- remove 'nr_feat'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- check global flag with boot cpu operations.
(suggested by Jan Beulich)
- remove 'cpu_init_work' and move codes into 'psr_cpu_init'.
(suggested by Jan Beulich)
- remove 'cpu_fini_work' and move codes into 'psr_cpu_fini'.
(suggested by Jan Beulich)
- assign value for 'cos_num'.
(suggested by Jan Beulich)
- change about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- fix format issue.
(suggested by Konrad Rzeszutek Wilk)
- add comments to explain why we care about cpumask_empty when the last
cpu on socket is offline.
(suggested by Konrad Rzeszutek Wilk)
v7:
- sort inclusion files position.
(suggested by Wei Liu)
- initialize structure objects for avoiding surprise.
(suggested by Konrad Rzeszutek Wilk)
- fix typo.
(suggested by Konrad Rzeszutek Wilk)
- fix a logical mistake when handling the last cpu offline event.
(suggested by Konrad Rzeszutek Wilk)
v6:
- make commit message be clearer.
(suggested by Konrad Rzeszutek Wilk)
- fix wordings.
(suggested by Konrad Rzeszutek Wilk)
- add comments to explain relationship between 'feat_mask' and
'enum psr_feat_type'.
(suggested by Konrad Rzeszutek Wilk)
- use 'struct cpuid_leaf' introduced in Andrew's patch.
(suggested by Konrad Rzeszutek Wilk)
- add comments about cpu_add_remove_lock.
(suggested by Konrad Rzeszutek Wilk)
- change 'clear_bit' to '__clear_bit'.
(suggested by Konrad Rzeszutek Wilk)
- add 'ASSERT' check when setting 'feat_mask'.
(suggested by Konrad Rzeszutek Wilk)
- adjust 'printk' position to avoid odd spacing.
(suggested by Konrad Rzeszutek Wilk)
- add comment to explain usage of 'feat_l3_cat'.
(suggested by Konrad Rzeszutek Wilk)
- fix wording.
(suggested by Konrad Rzeszutek Wilk)
- move 'cpuid_count_leaf' helper function to 'asm-x86/processor.h'.
It cannot be moved to 'cpuid.h' which causes compilation error because
of header file loop reference.
(suggested by Andrew Cooper)
v5:
- remove section number.
(suggested by Jan Beulich)
- remove double blank.
(suggested by Jan Beulich)
- add comment to explain the reason to define 'feat_l3_cat'.
(suggested by Jan Beulich)
- use 'list_for_each_entry_safe'.
(suggested by Jan Beulich)
- remove codes to free 'feat_l3_cat' in 'free_feature' to avoid the need
for an allocation the next time a CPU comes online.
(suggested by Jan Beulich)
- define 'struct cpuid_leaf_regs' to encapsulate eax~edx.
(suggested by Jan Beulich)
- print feature info on a socket only when 'opt_cpu_info' is true.
(suggested by Jan Beulich)
- declare global variable 'l3_cat_ops' to 'static const'.
(suggested by Jan Beulich)
- use 'current_cpu_data'.
(suggested by Jan Beulich)
- rename 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- clear PQE feature bit when the maximum CPUID level is too low.
(suggested by Jan Beulich)
- directly call 'l3_cat_init_feature'. No need to make it a callback
function.
(suggested by Jan Beulich)
- remove local variable 'info'.
(suggested by Jan Beulich)
- move 'INIT_LIST_HEAD' into 'cpu_init_work' to be together with
spin_lock_init().
(suggested by Jan Beulich)
- remove 'cpu_prepare_work' and move its content into 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v4:
- create this patch because of removing all old CAT/CDP codes to make
implementation be more easily understood.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 292 +++++++++++++++++++++++++++++++++++++++++++++-
xen/include/asm-x86/psr.h | 1 +
2 files changed, 287 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 96a8589..5980275 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -13,16 +13,118 @@
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
* more details.
*/
-#include <xen/init.h>
#include <xen/cpu.h>
#include <xen/err.h>
+#include <xen/init.h>
#include <xen/sched.h>
#include <asm/psr.h>
+/*
+ * Terminology:
+ * - CAT Cache Allocation Technology
+ * - CBM Capacity BitMasks
+ * - CDP Code and Data Prioritization
+ * - CMT Cache Monitoring Technology
+ * - COS/CLOS Class of Service. Also mean COS registers.
+ * - COS_MAX Max number of COS for the feature (minus 1)
+ * - MSRs Machine Specific Registers
+ * - PSR Intel Platform Shared Resource
+ */
+
#define PSR_CMT (1<<0)
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
+#define CAT_CBM_LEN_MASK 0x1f
+#define CAT_COS_MAX_MASK 0xffff
+
+/*
+ * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
+ * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
+ * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
+ *
+ * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for
+ * up to 64 L2 CAT COS. The COS_ID=[0,63].
+ *
+ * So, the maximum COS register count of one feature is 128.
+ */
+#define MAX_COS_REG_CNT 128
+
+/*
+ * Every PSR feature uses some COS registers for each COS ID, e.g. CDP uses 2
+ * COS registers (DATA and CODE) for one COS ID, but CAT uses 1 COS register.
+ * We use below macro as the max number of COS registers used by all features.
+ * So far, it is 2 which means CDP's COS registers number.
+ */
+#define MAX_COS_NUM 2
+
+enum psr_feat_type {
+ FEAT_TYPE_L3_CAT,
+ FEAT_TYPE_NUM,
+};
+
+/*
+ * This structure represents one feature.
+ * cos_max - The max COS registers number got through CPUID.
+ * cbm_len - The length of CBM got through CPUID.
+ * cos_reg_val - Array to store the values of COS registers. One entry stores
+ * the value of one COS register.
+ * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
+ * For CDP, two entries correspond to one COS_ID. E.g.
+ * COS_ID=0 corresponds to cos_reg_val[0] (Data) and
+ * cos_reg_val[1] (Code).
+ */
+struct feat_node {
+ /* cos_max and cbm_len are common values for all features so far. */
+ unsigned int cos_max;
+ unsigned int cbm_len;
+ uint32_t cos_reg_val[MAX_COS_REG_CNT];
+};
+
+/*
+ * This structure defines feature specific values, e.g. cos_num.
+ *
+ * Array 'feat_props' is defined to save every feature's properties. We use
+ * 'enum psr_feat_type' as index.
+ */
+static const struct feat_props {
+ /*
+ * cos_num - COS registers number that feature uses for one COS ID.
+ * It is defined in SDM.
+ */
+ unsigned int cos_num;
+
+ /*
+ * An array to save all 'enum cbm_type' values of the feature. It is
+ * used with cos_num together to get/write a feature's COS registers
+ * values one by one.
+ */
+ enum cbm_type type[MAX_COS_NUM];
+
+ /*
+ * alt_type is 'alternative type'. When this 'alt_type' is input, the
+ * feature does some special operations.
+ */
+ enum cbm_type alt_type;
+} *feat_props[FEAT_TYPE_NUM];
+
+/*
+ * PSR features are managed per socket. Below structure defines the members
+ * used to manage these features.
+ * feat_init - Indicate if features on a socket have been initialized.
+ * features - A feature node array used to manage all features enabled.
+ * ref_lock - A lock to protect cos_ref.
+ * cos_ref - A reference count array to record how many domains are using the
+ * COS ID. Every entry of cos_ref corresponds to one COS ID.
+ */
+struct psr_socket_info {
+ bool feat_init;
+ /* Feature array's index is 'enum psr_feat_type' which is same as 'props' */
+ struct feat_node *features[FEAT_TYPE_NUM];
+ spinlock_t ref_lock;
+ unsigned int cos_ref[MAX_COS_REG_CNT];
+};
+
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -30,11 +132,105 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
+static struct psr_socket_info *__read_mostly socket_info;
+
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
+static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
+/*
+ * Declare global feature node for every feature to facilitate the feature
+ * array creation. It is used to transiently store a spare node.
+ */
+static struct feat_node *feat_l3_cat;
+
+/* Common functions */
+#define cat_default_val(len) (0xffffffff >> (32 - (len)))
+
+/*
+ * Use this function to check if any allocation feature has been enabled
+ * in cmdline.
+ */
+static bool psr_alloc_feat_enabled(void)
+{
+ return !!socket_info;
+}
+
+static void free_socket_resources(unsigned int socket)
+{
+ unsigned int i;
+ struct psr_socket_info *info = socket_info + socket;
+
+ if ( !info )
+ return;
+
+ /*
+ * Free resources of features. The global feature object, e.g. feat_l3_cat,
+ * may not be freed here if it is not added into array. It is simply being
+ * kept until the next CPU online attempt.
+ */
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ xfree(info->features[i]);
+ info->features[i] = NULL;
+ }
+
+ info->feat_init = false;
+
+ memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
+}
+
+/* CAT common functions implementation. */
+static int cat_init_feature(const struct cpuid_leaf *regs,
+ struct feat_node *feat,
+ struct psr_socket_info *info,
+ enum psr_feat_type type)
+{
+ /* No valid value so do not enable feature. */
+ if ( !regs->a || !regs->d )
+ return -ENOENT;
+
+ feat->cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
+ feat->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
+
+ switch ( type )
+ {
+ case FEAT_TYPE_L3_CAT:
+ if ( feat->cos_max < 1 )
+ return -ENOENT;
+
+ /* We reserve cos=0 as default cbm (all bits within cbm_len are 1). */
+ feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
+
+ wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
+
+ break;
+
+ default:
+ return -ENOENT;
+ }
+
+ /* Add this feature into array. */
+ info->features[type] = feat;
+
+ if ( !opt_cpu_info )
+ return 0;
+
+ printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
+
+ return 0;
+}
+
+/* L3 CAT props */
+static const struct feat_props l3_cat_props = {
+ .cos_num = 1,
+ .type[0] = PSR_CBM_TYPE_L3,
+ .alt_type = PSR_CBM_TYPE_UNKNOWN,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -74,6 +270,9 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
+ if ( val_str && !strcmp(s, "cos_max") )
+ opt_cos_max = simple_strtoul(val_str, NULL, 0);
+
s = ss + 1;
} while ( ss );
}
@@ -229,19 +428,96 @@ void psr_domain_free(struct domain *d)
psr_free_rmid(d);
}
-static int psr_cpu_prepare(unsigned int cpu)
+static void __init init_psr(void)
{
+ if ( opt_cos_max < 1 )
+ {
+ printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
+ return;
+ }
+
+ socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
+
+ if ( !socket_info )
+ {
+ printk(XENLOG_WARNING "Failed to alloc socket_info!\n");
+ return;
+ }
+}
+
+static void __init psr_free(void)
+{
+ xfree(socket_info);
+ socket_info = NULL;
+}
+
+static int psr_cpu_prepare(void)
+{
+ if ( !psr_alloc_feat_enabled() )
+ return 0;
+
+ /* Malloc memory for the global feature node here. */
+ if ( feat_l3_cat == NULL &&
+ (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
static void psr_cpu_init(void)
{
+ struct psr_socket_info *info;
+ unsigned int socket, cpu = smp_processor_id();
+ struct feat_node *feat;
+ struct cpuid_leaf regs;
+
+ if ( !psr_alloc_feat_enabled() || !boot_cpu_has(X86_FEATURE_PQE) )
+ goto assoc_init;
+
+ if ( boot_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
+ {
+ setup_clear_cpu_cap(X86_FEATURE_PQE);
+ goto assoc_init;
+ }
+
+ socket = cpu_to_socket(cpu);
+ info = socket_info + socket;
+ if ( info->feat_init )
+ goto assoc_init;
+
+ spin_lock_init(&info->ref_lock);
+
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L3 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
+
+ feat = feat_l3_cat;
+ feat_l3_cat = NULL;
+
+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CAT) )
+ feat_props[FEAT_TYPE_L3_CAT] = &l3_cat_props;
+ }
+
+ info->feat_init = true;
+
+ assoc_init:
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
- return;
+ unsigned int socket = cpu_to_socket(cpu);
+
+ if ( !psr_alloc_feat_enabled() )
+ return;
+
+ /*
+ * We only free when we are the last CPU in the socket. The socket_cpumask
+ * is cleared prior to this notification code by remove_siblinginfo().
+ */
+ if ( socket_cpumask[socket] && cpumask_empty(socket_cpumask[socket]) )
+ free_socket_resources(socket);
}
static int cpu_callback(
@@ -253,7 +529,7 @@ static int cpu_callback(
switch ( action )
{
case CPU_UP_PREPARE:
- rc = psr_cpu_prepare(cpu);
+ rc = psr_cpu_prepare();
break;
case CPU_STARTING:
psr_cpu_init();
@@ -282,10 +558,14 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- psr_cpu_prepare(0);
+ if ( opt_psr & PSR_CAT )
+ init_psr();
+
+ if ( psr_cpu_prepare() )
+ psr_free();
psr_cpu_init();
- if ( psr_cmt_enabled() )
+ if ( psr_cmt_enabled() || psr_alloc_feat_enabled() )
register_cpu_notifier(&cpu_nfb);
return 0;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 57f47e9..8141336 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -50,6 +50,7 @@ enum cbm_type {
PSR_CBM_TYPE_L3,
PSR_CBM_TYPE_L3_CODE,
PSR_CBM_TYPE_L3_DATA,
+ PSR_CBM_TYPE_UNKNOWN,
};
extern struct psr_cmt *psr_cmt;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (3 preceding siblings ...)
2017-07-06 1:52 ` [PATCH v13 04/23] x86: refactor psr: L3 CAT: implement main data structures, CPU init and free flows Yi Sun
@ 2017-07-06 1:52 ` Yi Sun
2017-07-06 1:52 ` [PATCH v13 06/23] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
` (17 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:52 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the Domain init/free and schedule flows.
- When domain init, its psr resource should be allocated.
- When domain free, its psr resource should be freed too.
- When domain is scheduled, its COS ID on the socket should be
set into ASSOC register to make corresponding COS MSR value
work.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v13:
- modify macro name to be consistent.
(suggested by Jan Beulich)
- fix indenting issue.
(suggested by Jan Beulich)
v12:
- fix coding style issue.
(suggested by Jan Beulich)
- in 'get_max_cos_max', invert the condition and dropping 'continue'.
(suggested by Jan Beulich)
- modify 'psr_assoc_cos' to directly return 'reg' value.
(suggested by Jan Beulich)
- modify code style.
(suggested by Jan Beulich)
- do not break domain creation if 'psr_cos_ids' allocation fails.
(suggested by Jan Beulich)
v11:
- replace 'feat_init_done()' to 'feat_init' flag.
(suggested by Jan Beulich)
- adjust parameters positions when calling 'psr_assoc_cos'.
(suggested by Jan Beulich)
- add comment to explain why to check 'psr_cos_ids'.
v10:
- remove 'cat_get_cos_max' as 'cos_max' is a feature property now which
can be directly used.
(suggested by Jan Beulich)
- replace 'info->feat_mask' check to 'feat_init_done'.
(suggested by Jan Beulich)
v9:
- rename 'l3_cat_get_cos_max' to 'cat_get_cos_max' to cover all CAT/CDP
features.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- implement 'psr_alloc_cos' to match 'psr_free_cos'.
(suggested by Wei Liu)
- use 'psr_alloc_feat_enabled'.
(suggested by Wei Liu)
- fix coding style issue.
(suggested by Wei Liu)
- remove 'inline'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- remove 'psr_cos_ids' check in 'psr_free_cos'.
(suggested by Jan Beulich)
v6:
- change 'PSR_ASSOC_REG_POS' to 'PSR_ASSOC_REG_SHIFT'.
(suggested by Konrad Rzeszutek Wilk)
v5:
- rename 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- define 'PSR_ASSOC_REG_POS'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/domain.c | 3 +--
xen/arch/x86/psr.c | 67 ++++++++++++++++++++++++++++++++++++++++++++---
xen/include/asm-x86/psr.h | 2 +-
3 files changed, 65 insertions(+), 7 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 49388f4..68b8481 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -489,8 +489,7 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags,
}
spin_lock_init(&d->arch.e820_lock);
- if ( (rc = psr_domain_init(d)) != 0 )
- goto fail;
+ psr_domain_init(d);
if ( is_hvm_domain(d) )
{
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5980275..bebc679 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -50,6 +50,8 @@
*/
#define MAX_COS_REG_CNT 128
+#define ASSOC_REG_SHIFT 32
+
/*
* Every PSR feature uses some COS registers for each COS ID, e.g. CDP uses 2
* COS registers (DATA and CODE) for one COS ID, but CAT uses 1 COS register.
@@ -372,11 +374,37 @@ void psr_free_rmid(struct domain *d)
d->arch.psr_rmid = 0;
}
-static inline void psr_assoc_init(void)
+static unsigned int get_max_cos_max(const struct psr_socket_info *info)
+{
+ unsigned int cos_max = 0, i;
+
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ const struct feat_node *feat = info->features[i];
+
+ if ( feat )
+ cos_max = max(feat->cos_max, cos_max);
+ }
+
+ return cos_max;
+}
+
+static void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( psr_cmt_enabled() )
+ if ( psr_alloc_feat_enabled() )
+ {
+ unsigned int socket = cpu_to_socket(smp_processor_id());
+ const struct psr_socket_info *info = socket_info + socket;
+ unsigned int cos_max = get_max_cos_max(info);
+
+ if ( info->feat_init )
+ psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
+ ASSOC_REG_SHIFT;
+ }
+
+ if ( psr_cmt_enabled() || psra->cos_mask )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -385,6 +413,13 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
+static uint64_t psr_assoc_cos(uint64_t reg, unsigned int cos,
+ uint64_t cos_mask)
+{
+ return (reg & ~cos_mask) |
+ (((uint64_t)cos << ASSOC_REG_SHIFT) & cos_mask);
+}
+
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -393,6 +428,14 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
+ /* If domain's 'psr_cos_ids' is NULL, we set default value for it. */
+ if ( psra->cos_mask )
+ reg = psr_assoc_cos(reg,
+ (d->arch.psr_cos_ids ?
+ d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
+ 0),
+ psra->cos_mask);
+
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
@@ -418,14 +461,30 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
return 0;
}
-int psr_domain_init(struct domain *d)
+/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
+static void psr_free_cos(struct domain *d)
{
- return 0;
+ xfree(d->arch.psr_cos_ids);
+ d->arch.psr_cos_ids = NULL;
+}
+
+static void psr_alloc_cos(struct domain *d)
+{
+ d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
+ if ( !d->arch.psr_cos_ids )
+ printk(XENLOG_WARNING "Failed to alloc psr_cos_ids!\n");
+}
+
+void psr_domain_init(struct domain *d)
+{
+ if ( psr_alloc_feat_enabled() )
+ psr_alloc_cos(d);
}
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
+ psr_free_cos(d);
}
static void __init init_psr(void)
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 8141336..a17e31a 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -71,7 +71,7 @@ int psr_get_l3_cbm(struct domain *d, unsigned int socket,
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type);
-int psr_domain_init(struct domain *d);
+void psr_domain_init(struct domain *d);
void psr_domain_free(struct domain *d);
#endif /* __ASM_PSR_H__ */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 06/23] x86: refactor psr: L3 CAT: implement get hw info flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (4 preceding siblings ...)
2017-07-06 1:52 ` [PATCH v13 05/23] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
@ 2017-07-06 1:52 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
` (16 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:52 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements get HW info flow including L3 CAT callback
function.
It also changes sysctl interface to make it more general.
With this patch, 'psr-hwinfo' can work for L3 CAT.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v12:
- use 'ASSERT_UNREACHABLE()' to record bug.
(suggested by Jan Beulich)
v11:
- changes about 'cos_max' and 'cbm_len'.
(suggested by Jan Beulich)
- change about 'feat_props'.
(suggested by Jan Beulich)
- declare 'PSR_SOCKET_FEAT_UNKNOWN' to handle error case.
(suggested by Jan Beulich)
- use 'ASSERT' to check data.
(suggested by Jan Beulich)
- fix comment of 'get_feat_info'.
(suggested by Jan Beulich)
- fix a bug when checking 'feat_type'.
(suggested by Jan Beulich)
v10:
- remove 'PSR_SOCKET_UNKNOWN' and use 'ASSERT_UNREACHABLE()' to handle
this case.
(suggested by Jan Beulich)
- check 'feat_type'.
(suggested by Jan Beulich)
- adjust macros names and values to make them more appropriate.
(suggested by Jan Beulich)
- use 'feat_init_done'.
(suggested by Jan Beulich)
- changes about 'cbm_len'.
(suggested by Jan Beulich)
v9:
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- define 'PSR_INFO_SIZE'.
(suggested by Roger Pau)
- fix coding style issue.
(suggested by Roger Pau and Jan Beulich)
- use 'ARRAY_SIZE'.
(suggested by Roger Pau)
- rename 'l3_cat_get_feat_info' to 'cat_get_feat_info' to make it a common
function for both L3/L2 CAT.
(suggested by Roger Pau)
- move constant to the right of comparison.
(suggested by Wei Liu)
- remove wrong comment.
(suggested by Jan Beulich)
- rename macros used by psr_get_info to make them meaningful.
(suggested by Jan Beulich)
- remove assignment for 'PSR_SOCKET_UNKNOWN'.
(suggested by Jan Beulich)
- retain blank line after 'case XEN_SYSCTL_PSR_CAT_get_l3_info'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- move common data check into common function.
(suggested by Jan Beulich)
v6:
- fix coding style issue.
(suggested by Konrad Rzeszutek Wilk)
- define 'PSR_SOCKET_UNKNOWN' in 'psr_feat_type'.
(suggested by Konrad Rzeszutek Wilk)
- change '-ENOTSOCK' to 'ERANGE'.
(suggested by Konrad Rzeszutek Wilk)
- modify position of macros to remove odd spacing in psr.h.
(suggested by Konrad Rzeszutek Wilk)
v5:
- change 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
- modify parameter type to avoid fixed width type when there is no such
intention.
(suggested by Jan Beulich)
- use 'const' when it is possible.
(suggested by Jan Beulich)
- check feature type outside callback function.
(suggested by Jan Beulich)
- modify macros names to add prefix 'PSR_' and change 'CDP_FLAG' to
'PSR_FLAG'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 81 +++++++++++++++++++++++++++++++++++++++++++++--
xen/arch/x86/sysctl.c | 19 ++++++++---
xen/include/asm-x86/psr.h | 16 +++++++---
3 files changed, 104 insertions(+), 12 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index bebc679..5d6b29b 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -63,6 +63,7 @@
enum psr_feat_type {
FEAT_TYPE_L3_CAT,
FEAT_TYPE_NUM,
+ FEAT_TYPE_UNKNOWN,
};
/*
@@ -108,6 +109,10 @@ static const struct feat_props {
* feature does some special operations.
*/
enum cbm_type alt_type;
+
+ /* get_feat_info is used to return feature HW info through sysctl. */
+ bool (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len);
} *feat_props[FEAT_TYPE_NUM];
/*
@@ -184,6 +189,22 @@ static void free_socket_resources(unsigned int socket)
memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
}
+static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
+{
+ enum psr_feat_type feat_type = FEAT_TYPE_UNKNOWN;
+
+ switch ( type )
+ {
+ case PSR_CBM_TYPE_L3:
+ feat_type = FEAT_TYPE_L3_CAT;
+ break;
+ default:
+ ASSERT_UNREACHABLE();
+ }
+
+ return feat_type;
+}
+
/* CAT common functions implementation. */
static int cat_init_feature(const struct cpuid_leaf *regs,
struct feat_node *feat,
@@ -226,11 +247,25 @@ static int cat_init_feature(const struct cpuid_leaf *regs,
return 0;
}
+static bool cat_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len)
+{
+ if ( array_len != PSR_INFO_ARRAY_SIZE )
+ return false;
+
+ data[PSR_INFO_IDX_COS_MAX] = feat->cos_max;
+ data[PSR_INFO_IDX_CAT_CBM_LEN] = feat->cbm_len;
+ data[PSR_INFO_IDX_CAT_FLAG] = 0;
+
+ return true;
+}
+
/* L3 CAT props */
static const struct feat_props l3_cat_props = {
.cos_num = 1,
.type[0] = PSR_CBM_TYPE_L3,
.alt_type = PSR_CBM_TYPE_UNKNOWN,
+ .get_feat_info = cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -443,10 +478,50 @@ void psr_ctxt_switch_to(struct domain *d)
}
}
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags)
+static struct psr_socket_info *get_socket_info(unsigned int socket)
{
- return 0;
+ if ( !socket_info )
+ return ERR_PTR(-ENODEV);
+
+ if ( socket >= nr_sockets )
+ return ERR_PTR(-ERANGE);
+
+ if ( !socket_info[socket].feat_init )
+ return ERR_PTR(-ENOENT);
+
+ return socket_info + socket;
+}
+
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len)
+{
+ const struct psr_socket_info *info = get_socket_info(socket);
+ const struct feat_node *feat;
+ enum psr_feat_type feat_type;
+
+ ASSERT(data);
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( feat_type >= ARRAY_SIZE(info->features) )
+ return -ENOENT;
+
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ if ( !feat_props[feat_type] )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( feat_props[feat_type]->get_feat_info(feat, data, array_len) )
+ return 0;
+
+ return -EINVAL;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 2f7056e..c23270d 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -175,14 +175,25 @@ long arch_do_sysctl(
switch ( sysctl->u.psr_cat_op.cmd )
{
case XEN_SYSCTL_PSR_CAT_get_l3_info:
- ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
- &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
- &sysctl->u.psr_cat_op.u.l3_info.cos_max,
- &sysctl->u.psr_cat_op.u.l3_info.flags);
+ {
+ uint32_t data[PSR_INFO_ARRAY_SIZE];
+
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
+ if ( ret )
+ break;
+
+ sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ data[PSR_INFO_IDX_COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ data[PSR_INFO_IDX_CAT_CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.flags =
+ data[PSR_INFO_IDX_CAT_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
ret = -EFAULT;
break;
+ }
default:
ret = -EOPNOTSUPP;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index a17e31a..8d15328 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -19,20 +19,26 @@
#include <xen/types.h>
/* CAT cpuid level */
-#define PSR_CPUID_LEVEL_CAT 0x10
+#define PSR_CPUID_LEVEL_CAT 0x10
/* Resource Type Enumeration */
#define PSR_RESOURCE_TYPE_L3 0x2
/* L3 Monitoring Features */
-#define PSR_CMT_L3_OCCUPANCY 0x1
+#define PSR_CMT_L3_OCCUPANCY 0x1
/* CDP Capability */
-#define PSR_CAT_CDP_CAPABILITY (1u << 2)
+#define PSR_CAT_CDP_CAPABILITY (1u << 2)
/* L3 CDP Enable bit*/
#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
+/* Used by psr_get_info() */
+#define PSR_INFO_IDX_COS_MAX 0
+#define PSR_INFO_IDX_CAT_CBM_LEN 1
+#define PSR_INFO_IDX_CAT_FLAG 2
+#define PSR_INFO_ARRAY_SIZE 3
+
struct psr_cmt_l3 {
unsigned int features;
unsigned int upscaling_factor;
@@ -64,8 +70,8 @@ int psr_alloc_rmid(struct domain *d);
void psr_free_rmid(struct domain *d);
void psr_ctxt_switch_to(struct domain *d);
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags);
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len);
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 07/23] x86: refactor psr: L3 CAT: implement get value flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (5 preceding siblings ...)
2017-07-06 1:52 ` [PATCH v13 06/23] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
` (15 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
There is an interface in user space to show feature value of
domains.
This patch implements get value flow in hypervisor.
It also changes domctl interface to make it more general.
With this patch, 'psr-cat-show' can work for L3 CAT but not for
L3 code/data which is implemented in CDP related patches.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v12:
- remove 'psr_get_feat_and_type' and implement its functionality in
'psr_get_val'.
- use 'ASSERT_UNREACHABLE' to record error.
(suggested by Jan Beulich)
- return success only if the 'val' is set in 'psr_get_val'.
(suggested by Jan Beulich)
v11:
- declare a 'switch()' wide variable 'val32' in domctl.
(suggested by Jan Beulich)
- remove 'get_val' callback function which is replaced by generic codes.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- do not 'ASSERT' domain pointer.
(suggested by Jan Beulich)
- modify commit message.
v10:
- use an intermediate variable to get value and avoid cast in domctl.
(suggested by Jan Beulich)
- remove 'type' in 'get_val' parameters and will add it back when
implementing CDP.
(suggested by Jan Beulich)
- remove unnecessary variable and return error about 'info' in
'psr_get_feat'.
(suggested by Jan Beulich)
- use 'ASSERT' to check input parameter in 'psr_get_val'.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
v9:
- add commit message to explain there is an user space interface.
- rename 'l3_cat_get_val' to 'cat_get_val' to cover all L3/L2 CAT features.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- change parameter of 'psr_get'. Use 'psr_cos_ids' directly to replace
domain. Also declare it to 'const'.
(suggested by Jan Beulich)
- change code flow to remove 'psr_get' but add 'psr_get_feat' to make codes
more reasonable.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- move cos check into common function because this check is required by all
features.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v7:
- rename '__psr_get' to 'psr_get'.
(suggested by Wei Liu)
v6:
- modify commit message to make it clearer.
(suggested by Konrad Rzeszutek Wilk)
- remove one extra space in code.
(suggested by Konrad Rzeszutek Wilk)
- remove unnecessary comment.
(suggested by Konrad Rzeszutek Wilk)
- write a helper function to move get info and get val functions into
it. Because most codes of 'get_info' and 'get_val' are same.
(suggested by Konrad Rzeszutek Wilk)
v5:
- rename 'dat[]' to 'data[]'
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- check if feature type match in caller of feature callback function.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/domctl.c | 20 ++++++++++---------
xen/arch/x86/psr.c | 51 ++++++++++++++++++++++++++++++++++++++++++++---
xen/include/asm-x86/psr.h | 4 ++--
3 files changed, 61 insertions(+), 14 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index f40e989..f350c66 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1419,6 +1419,8 @@ long arch_do_domctl(
case XEN_DOMCTL_psr_cat_op:
switch ( domctl->u.psr_cat_op.cmd )
{
+ uint32_t val32;
+
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
domctl->u.psr_cat_op.data,
@@ -1438,23 +1440,23 @@ long arch_do_domctl(
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L3);
+ domctl->u.psr_cat_op.data = val32;
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L3_CODE);
+ domctl->u.psr_cat_op.data = val32;
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L3_DATA);
+ domctl->u.psr_cat_op.data = val32;
copyback = 1;
break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5d6b29b..c245c9a 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -524,10 +524,55 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
return -EINVAL;
}
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type)
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint32_t *val, enum cbm_type type)
{
- return 0;
+ const struct psr_socket_info *info = get_socket_info(socket);
+ const struct feat_node *feat;
+ enum psr_feat_type feat_type;
+ unsigned int cos, i;
+
+ ASSERT(val);
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( feat_type >= ARRAY_SIZE(info->features) )
+ return -ENOENT;
+
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ if ( !feat_props[feat_type] )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ cos = d->arch.psr_cos_ids[socket];
+ /*
+ * If input cos exceeds current feature's cos_max, we should return its
+ * default value which is stored in cos 0. This case only happens
+ * when more than two features enabled concurrently and at least one
+ * features's cos_max is bigger than others. When a domain's working cos
+ * id is bigger than some features' cos_max, HW automatically works as
+ * default value for those features which cos_max is smaller.
+ */
+ if ( cos > feat->cos_max )
+ cos = 0;
+
+ for ( i = 0; i < feat_props[feat_type]->cos_num; i++ )
+ {
+ if ( type == feat_props[feat_type]->type[i] )
+ {
+ *val = feat->cos_reg_val[cos * feat_props[feat_type]->cos_num + i];
+ return 0;
+ }
+ }
+
+ return -EINVAL;
}
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 8d15328..0e960bf 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -72,8 +72,8 @@ void psr_ctxt_switch_to(struct domain *d);
int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type);
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint32_t *val, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (6 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 07/23] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-12 19:14 ` Jan Beulich
2017-07-06 1:53 ` [PATCH v13 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
` (14 subsequent siblings)
22 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
As set value flow is the most complicated one in psr, it will be
divided to some patches to make things clearer. This patch
implements the set value framework to show a whole picture firstly.
It also changes domctl interface to make it more general.
To make the set value flow be general and can support multiple features
at same time, it includes below steps:
1. Test and set dom_ids bit corresponding to the domain. If the old bit is 0
which means the domain's COS ID is invalid, restore COS ID to 0. If the
COS ID is valid, get the COS ID that current domain is using.
2. Gather a value array to store all features current value
into it and replace the current value of the feature which is
being set to the new input value.
3. Find if there is already a COS ID on which all features'
values are same as the array. Then, we can reuse this COS
ID.
4. If fail to find, we need pick an available COS ID. Only COS ID which ref
is 0 or 1 can be picked.
5. Write the feature's MSRs according to the COS ID.
6. Update ref according to COS ID.
7. Save the COS ID into current domain's psr_cos_ids[socket] so that we
can know which COS the domain is using on the socket.
So, some functions are abstracted and the callback functions will be
implemented in next patches.
Here is an example to understand the process. The CPU supports
two featuers, e.g. L3 CAT and L2 CAT. User wants to set L3 CAT
of Dom1 to 0x1ff.
1. At the initial time, the old_cos of Dom1 is 0. The COS registers values
are below at this time.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | 0x7ff | 0x7ff | ... |
-------------------------------
L2 CAT | 0xff | 0xff | 0xff | ... |
-------------------------------
2. Gather the value array and insert new value into it:
val[0]: 0x1ff
val[1]: 0xff
3. It cannot find a matching COS.
4. Pick COS 1 to store the value set.
5. Write the L3 CAT COS 1 registers. The COS registers values are
changed to below now.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | 0x1ff | ... | ... |
-------------------------------
L2 CAT | 0xff | 0xff | ... | ... |
-------------------------------
6. The ref[1] is increased to 1 because Dom1 is using it now.
7. Save 1 to Dom1's psr_cos_ids[socket].
Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
of Dom2 is 0 too. Repeat above flow.
The val array assembled is:
val[0]: 0x1ff
val[1]: 0xff
So, it can find a matching COS, COS 1. Then, it can reuse COS 1
for Dom2.
The ref[1] is increased to 2 now because both Dom1 and Dom2 are
using this COS ID. Set 1 to Dom2's psr_cos_ids[socket].
There is one thing need to emphasize that we need restore domain's COS ID to
0 when socket is offline. Otherwise, a wrong COS ID will be used when the
socket is online again. That may cause user see the wrong CBM shown. But it
takes much time to iterate all domains to restore COS ID to 0. So, we define
a 'dom_ids[]' to represents all domains, one bit corresponds to one domain.
If the bit is 0 when entering 'psr_ctxt_switch_to', that means this is the
first time the domain is switched to this socket or domain's COS ID has not
been set since the socket is online. So, the COS ID set to ASSOC register on
this socket should be default value, 0. If not, that means the domain's COS
ID has been set when the socket was online. So, this COS ID is valid and we
can directly use it. We restore the domain's COS ID to 0 if the bit
corresponding to the domain is 0 but the domain's COS ID is not 0 when
'psr_get_val' and 'psr_set_val' is called. This can avoid CPU serialization
if restoring action is exectued in 'psr_ctxt_switch_to'.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v13:
- replace 'test_bit' to 'test_and_set_bit' in 'psr_get_val/psr_set_val'.
(suggested by Jan Beulich)
- remove 'set_bit' used in 'psr_set_val' .
(suggested by Jan Beulich)
- remove unused parameter in 'get_cos_num()'.
(suggested by Jan Beulich)
v12:
- remove the memebers position changes in 'psr_socket_info'.
(suggested by Jan Beulich)
- rename 'dom_ids' to 'dom_set'.
(suggested by Jan Beulich)
- call 'bitmap_zero' to clear bitmap.
(suggested by Jan Beulich)
- combine two if()-s in 'psr_ctxt_switch_to' and add comment.
(suggested by Jan Beulich)
- remove redundant check in 'psr_get_val'.
(suggested by Jan Beulich)
- use 'domain_lock()' to protect 'psr_cos_ids' so that the codes do not
depend on 'domctl_lock'.
(suggested by Jan Beulich)
- adjust codes to avoid cast in 'psr_set_val'.
(suggested by Jan Beulich)
- adjust codes to avoid duplication of error paths in 'psr_set_val'.
(suggested by Jan Beulich)
- clear the domain bit anyway in 'psr_free_cos'.
(suggested by Jan Beulich)
- use the default cos id when getting old_cos in 'psr_set_val'.
v11:
- define 'dom_ids[]' and implement related flows.
- restore domain cos id to 0 in 'psr_get_val'.
- rename 'write_psr_msr' to 'write_psr_msrs' and change its parameters to
handle value array the feature's all MSRs.
- fix coding style issue.
(suggested by Jan Beulich)
- do not need check 'cos' in ASSERT.
(suggested by Jan Beulich)
- rename 'insert_val_to_array' to 'insert_val_into_array'.
(suggested by Jan Beulich)
- remove 'ref_lock' from parameter list in 'find_cos' and 'pick_avail_cos'.
(suggested by Jan Beulich)
- remove ASSERT check to 'ref_lock' in 'find_cos' and 'pick_avail_cos'.
(suggested by Jan Beulich)
- fix a bug for checking 'feat_type'.
(suggested by Jan Beulich)
- move 'free_array' label.
(suggested by Jan Beulich)
- modify comments and commit message.
v10:
- restore domain cos id to 0 when socket is offline.
(suggested by Jan Beulich)
- check 'psr_cat_op.data' to make sure only lower 32 bits are valid.
(suggested by Jan Beulich)
- remove unnecessary fixed width type of parameters and variables.
(suggested by Jan Beulich)
- rename 'insert_new_val_to_array' to 'insert_val_to_array'.
(suggested by Jan Beulich)
- input 'ref_lock' pointer into functions to check if it has been locked.
(suggested by Jan Beulich)
- add comment to declare the set process is protected by 'domctl_lock'.
(suggested by Jan Beulich)
- check 'feat_type'.
(suggested by Jan Beulich)
- remove 'feat_mask'.
(suggested by Jan Beulich)
- remove unnecessary criteria of ASSERT.
(suggested by Jan Beulich)
- adjust flow of 'psr_set_val' to avoid 'goto' for successful cases.
(suggested by Jan Beulich)
- use ASSERT to check 'socket_info' in 'psr_free_cos'.
(suggested by Jan Beulich)
- remove unnecessary comment in 'psr_free_cos'.
(suggested by Jan Beulich)
v9:
- use goto style error handling in 'psr_set_val'.
(suggested by Wei Liu)
- use ASSERT for checking old_cos.
(suggested by Wei Liu and Jan Beulich)
- fix coding style issue.
(suggested by Wei Liu)
- rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
(suggested by Wei Liu)
- use 'spin_is_locked' to check ref_lock.
(suggested by Roger Pau)
- add an input parameter 'array_len' for 'write_psr_msr'.
- check 'socket_info' and 'psr_cos_ids' in this patch.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- fix commit message words.
(suggested by Jan Beulich)
- change 'assemble_val_array' to 'gather_val_array'.
(suggested by Jan Beulich)
- change 'set_new_val_to_array' to 'insert_new_val_to_array'.
(suggested by Jan Beulich)
- change parameter 'm' of 'insert_new_val_to_array' to 'new_val'.
(suggested by Jan Beulich)
- change 'write_psr_msr' to 'write_psr_msrs'.
(suggested by Jan Beulich)
- correct comments.
(suggested by Jan Beulich)
- remove unnecessary comments.
(suggested by Jan Beulich)
- adjust conditions after 'find_cos' to save a level of indentation.
(suggested by Jan Beulich)
- add 'ASSERT(!old_cos || ref[old_cos])'.
(suggested by Jan Beulich)
- move ASSERT() check into locked region.
(suggested by Jan Beulich)
- replace parameter '*val' to 'val[]' in some functions.
(suggested by Jan Beulich)
- change 'write_psr_msr' parameters to prepare to only set one new value
for one feature.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
- add explanation about context switch.
(suggested by Jan Beulich)
v5:
- modify commit message.
(suggested by Jan Beulich)
- return an error for all helper functions in set flow.
(suggested by Jan Beulich)
- remove unnecessary cast.
(suggested by Jan Beulich)
- divide 'get_old_set_new' to two functions, 'assemble_val_array' and
'set_new_val_to_array'.
(suggested by Jan Beulich)
- modify comments.
(suggested by Jan Beulich)
- adjust code format.
(suggested by Jan Beulich)
- change 'alloc_new_cos' to 'pick_avail_cos' to make name accurate.
(suggested by Jan Beulich)
- check feature type when entering 'psr_set_val'.
(suggested by Jan Beulich)
- use ASSERT to check ref.
(suggested by Jan Beulich)
- rename 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/domctl.c | 18 ++--
xen/arch/x86/psr.c | 240 ++++++++++++++++++++++++++++++++++++++++++++--
xen/include/asm-x86/psr.h | 4 +-
3 files changed, 242 insertions(+), 20 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index f350c66..cea643c 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1422,21 +1422,21 @@ long arch_do_domctl(
uint32_t val32;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_CODE);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_DATA);
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index c245c9a..5aa1bcc 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -123,6 +123,7 @@ static const struct feat_props {
* ref_lock - A lock to protect cos_ref.
* cos_ref - A reference count array to record how many domains are using the
* COS ID. Every entry of cos_ref corresponds to one COS ID.
+ * dom_set - A bitmap to indicate which domain's cos id has been set.
*/
struct psr_socket_info {
bool feat_init;
@@ -130,6 +131,8 @@ struct psr_socket_info {
struct feat_node *features[FEAT_TYPE_NUM];
spinlock_t ref_lock;
unsigned int cos_ref[MAX_COS_REG_CNT];
+ /* Every bit corresponds to a domain. Index is domain_id. */
+ DECLARE_BITMAP(dom_set, DOMID_IDLE + 1);
};
struct psr_assoc {
@@ -187,6 +190,8 @@ static void free_socket_resources(unsigned int socket)
info->feat_init = false;
memset(info->cos_ref, 0, MAX_COS_REG_CNT * sizeof(unsigned int));
+
+ bitmap_zero(info->dom_set, DOMID_IDLE + 1);
}
static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
@@ -463,13 +468,25 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
- /* If domain's 'psr_cos_ids' is NULL, we set default value for it. */
+ /*
+ * If the domain is not set in 'dom_set' bitmap, that means the domain's
+ * cos id is not valid. So, we have to use default value (0) to set ASSOC
+ * register. Furthermore, if domain's 'psr_cos_ids' is NULL, we need
+ * default value for it too (for case that the domain's psr_cos_ids is not
+ * successfully allocated).
+ */
if ( psra->cos_mask )
- reg = psr_assoc_cos(reg,
- (d->arch.psr_cos_ids ?
- d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
- 0),
- psra->cos_mask);
+ {
+ unsigned int socket = cpu_to_socket(smp_processor_id());
+ struct psr_socket_info *info = socket_info + socket;
+ unsigned int cos = 0;
+
+ if ( likely(test_bit(d->domain_id, info->dom_set)) &&
+ d->arch.psr_cos_ids )
+ cos = d->arch.psr_cos_ids[socket];
+
+ reg = psr_assoc_cos(reg, cos, psra->cos_mask);
+ }
if ( reg != psra->val )
{
@@ -551,7 +568,13 @@ int psr_get_val(struct domain *d, unsigned int socket,
return -ENOENT;
}
+ domain_lock(d);
+ if ( !test_and_set_bit(d->domain_id, socket_info[socket].dom_set) )
+ d->arch.psr_cos_ids[socket] = 0;
+
cos = d->arch.psr_cos_ids[socket];
+ domain_unlock(d);
+
/*
* If input cos exceeds current feature's cos_max, we should return its
* default value which is stored in cos 0. This case only happens
@@ -575,15 +598,214 @@ int psr_get_val(struct domain *d, unsigned int socket,
return -EINVAL;
}
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type)
+/* Set value functions */
+static unsigned int get_cos_num(void)
{
return 0;
}
-/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
+static int gather_val_array(uint32_t val[],
+ unsigned int array_len,
+ const struct psr_socket_info *info,
+ unsigned int old_cos)
+{
+ return -EINVAL;
+}
+
+static int insert_val_into_array(uint32_t val[],
+ unsigned int array_len,
+ const struct psr_socket_info *info,
+ enum psr_feat_type feat_type,
+ enum cbm_type type,
+ uint32_t new_val)
+{
+ return -EINVAL;
+}
+
+static int find_cos(const uint32_t val[], unsigned int array_len,
+ enum psr_feat_type feat_type,
+ const struct psr_socket_info *info)
+{
+ return -ENOENT;
+}
+
+static int pick_avail_cos(const struct psr_socket_info *info,
+ const uint32_t val[], unsigned int array_len,
+ unsigned int old_cos,
+ enum psr_feat_type feat_type)
+{
+ return -ENOENT;
+}
+
+static int write_psr_msrs(unsigned int socket, unsigned int cos,
+ uint32_t val[], unsigned int array_len,
+ enum psr_feat_type feat_type)
+{
+ return -ENOENT;
+}
+
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t new_val, enum cbm_type type)
+{
+ unsigned int old_cos, array_len;
+ int cos, ret;
+ unsigned int *ref;
+ uint32_t *val_array, val;
+ struct psr_socket_info *info = get_socket_info(socket);
+ enum psr_feat_type feat_type;
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ val = new_val;
+ if ( new_val != val )
+ return -EINVAL;
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( feat_type >= ARRAY_SIZE(info->features) ||
+ !info->features[feat_type] )
+ return -ENOENT;
+
+ /*
+ * Step 0:
+ * old_cos means the COS ID current domain is using. By default, it is 0.
+ *
+ * For every COS ID, there is a reference count to record how many domains
+ * are using the COS register corresponding to this COS ID.
+ * - If ref[old_cos] is 0, that means this COS is not used by any domain.
+ * - If ref[old_cos] is 1, that means this COS is only used by current
+ * domain.
+ * - If ref[old_cos] is more than 1, that mean multiple domains are using
+ * this COS.
+ */
+ domain_lock(d);
+ if ( !test_and_set_bit(d->domain_id, info->dom_set) )
+ d->arch.psr_cos_ids[socket] = 0;
+
+ old_cos = d->arch.psr_cos_ids[socket];
+ domain_unlock(d);
+
+ ASSERT(old_cos < MAX_COS_REG_CNT);
+
+ ref = info->cos_ref;
+
+ /*
+ * Step 1:
+ * Gather a value array to store all features cos_reg_val[old_cos].
+ * And, set the input new val into array according to the feature's
+ * position in array.
+ */
+ array_len = get_cos_num();
+ val_array = xzalloc_array(uint32_t, array_len);
+ if ( !val_array )
+ return -ENOMEM;
+
+ if ( (ret = gather_val_array(val_array, array_len, info, old_cos)) != 0 )
+ goto free_array;
+
+ if ( (ret = insert_val_into_array(val_array, array_len, info,
+ feat_type, type, val)) != 0 )
+ goto free_array;
+
+ spin_lock(&info->ref_lock);
+
+ /*
+ * Step 2:
+ * Try to find if there is already a COS ID on which all features' values
+ * are same as the array. Then, we can reuse this COS ID.
+ */
+ cos = find_cos(val_array, array_len, feat_type, info);
+ if ( cos == old_cos )
+ {
+ ret = 0;
+ goto unlock_free_array;
+ }
+
+ /*
+ * Step 3:
+ * If fail to find, we need pick an available COS ID.
+ * In fact, only COS ID which ref is 1 or 0 can be picked for current
+ * domain. If old_cos is not 0 and its ref==1, that means only current
+ * domain is using this old_cos ID. So, this old_cos ID certainly can
+ * be reused by current domain. Ref==0 means there is no any domain
+ * using this COS ID. So it can be used for current domain too.
+ */
+ if ( cos < 0 )
+ {
+ cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
+ if ( cos < 0 )
+ {
+ ret = cos;
+ goto unlock_free_array;
+ }
+
+ /*
+ * Step 4:
+ * Write the feature's MSRs according to the COS ID.
+ */
+ ret = write_psr_msrs(socket, cos, val_array, array_len, feat_type);
+ if ( ret )
+ goto unlock_free_array;
+ }
+
+ /*
+ * Step 5:
+ * Find the COS ID (find_cos result is '>= 0' or an available COS ID is
+ * picked, then update ref according to COS ID.
+ */
+ ref[cos]++;
+ ASSERT(!cos || ref[cos]);
+ ASSERT(!old_cos || ref[old_cos]);
+ ref[old_cos]--;
+ spin_unlock(&info->ref_lock);
+
+ /*
+ * Step 6:
+ * Save the COS ID into current domain's psr_cos_ids[] so that we can know
+ * which COS the domain is using on the socket. One domain can only use
+ * one COS ID at same time on each socket.
+ */
+ domain_lock(d);
+ d->arch.psr_cos_ids[socket] = cos;
+ domain_unlock(d);
+
+ goto free_array;
+
+ unlock_free_array:
+ spin_unlock(&info->ref_lock);
+
+ free_array:
+ xfree(val_array);
+ return ret;
+}
+
static void psr_free_cos(struct domain *d)
{
+ unsigned int socket, cos;
+
+ ASSERT(socket_info);
+
+ if ( !d->arch.psr_cos_ids )
+ return;
+
+ /* Domain is destroyed so its cos_ref should be decreased. */
+ for ( socket = 0; socket < nr_sockets; socket++ )
+ {
+ struct psr_socket_info *info = socket_info + socket;
+
+ clear_bit(d->domain_id, info->dom_set);
+
+ /* cos 0 is default one which does not need be handled. */
+ cos = d->arch.psr_cos_ids[socket];
+ if ( cos == 0 )
+ continue;
+
+ spin_lock(&info->ref_lock);
+ ASSERT(info->cos_ref[cos]);
+ info->cos_ref[cos]--;
+ spin_unlock(&info->ref_lock);
+ }
+
xfree(d->arch.psr_cos_ids);
d->arch.psr_cos_ids = NULL;
}
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 0e960bf..50b8757 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -74,8 +74,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
int psr_get_val(struct domain *d, unsigned int socket,
uint32_t *val, enum cbm_type type);
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type);
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t val, enum cbm_type type);
void psr_domain_init(struct domain *d);
void psr_domain_free(struct domain *d);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v13 08/23] x86: refactor psr: L3 CAT: set value: implement framework.
2017-07-06 1:53 ` [PATCH v13 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
@ 2017-07-12 19:14 ` Jan Beulich
0 siblings, 0 replies; 42+ messages in thread
From: Jan Beulich @ 2017-07-12 19:14 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:08 AM >>>
>As set value flow is the most complicated one in psr, it will be
>divided to some patches to make things clearer. This patch
>implements the set value framework to show a whole picture firstly.
>
>It also changes domctl interface to make it more general.
>
>To make the set value flow be general and can support multiple features
>at same time, it includes below steps:
>1. Test and set dom_ids bit corresponding to the domain. If the old bit is 0
>which means the domain's COS ID is invalid, restore COS ID to 0. If the
>COS ID is valid, get the COS ID that current domain is using.
>2. Gather a value array to store all features current value
>into it and replace the current value of the feature which is
>being set to the new input value.
>3. Find if there is already a COS ID on which all features'
>values are same as the array. Then, we can reuse this COS
>ID.
>4. If fail to find, we need pick an available COS ID. Only COS ID which ref
>is 0 or 1 can be picked.
>5. Write the feature's MSRs according to the COS ID.
>6. Update ref according to COS ID.
>7. Save the COS ID into current domain's psr_cos_ids[socket] so that we
>can know which COS the domain is using on the socket.
>
>So, some functions are abstracted and the callback functions will be
>implemented in next patches.
>
>Here is an example to understand the process. The CPU supports
>two featuers, e.g. L3 CAT and L2 CAT. User wants to set L3 CAT
>of Dom1 to 0x1ff.
>1. At the initial time, the old_cos of Dom1 is 0. The COS registers values
>are below at this time.
>-------------------------------
>| COS 0 | COS 1 | COS 2 | ... |
>-------------------------------
>L3 CAT | 0x7ff | 0x7ff | 0x7ff | ... |
>-------------------------------
>L2 CAT | 0xff | 0xff | 0xff | ... |
>-------------------------------
>
>2. Gather the value array and insert new value into it:
>val[0]: 0x1ff
>val[1]: 0xff
>
>3. It cannot find a matching COS.
>
>4. Pick COS 1 to store the value set.
>
>5. Write the L3 CAT COS 1 registers. The COS registers values are
>changed to below now.
>-------------------------------
>| COS 0 | COS 1 | COS 2 | ... |
>-------------------------------
>L3 CAT | 0x7ff | 0x1ff | ... | ... |
>-------------------------------
>L2 CAT | 0xff | 0xff | ... | ... |
>-------------------------------
>
>6. The ref[1] is increased to 1 because Dom1 is using it now.
>
>7. Save 1 to Dom1's psr_cos_ids[socket].
>
>Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
>of Dom2 is 0 too. Repeat above flow.
>
>The val array assembled is:
>val[0]: 0x1ff
>val[1]: 0xff
>
>So, it can find a matching COS, COS 1. Then, it can reuse COS 1
>for Dom2.
>
>The ref[1] is increased to 2 now because both Dom1 and Dom2 are
>using this COS ID. Set 1 to Dom2's psr_cos_ids[socket].
>
>There is one thing need to emphasize that we need restore domain's COS ID to
>0 when socket is offline. Otherwise, a wrong COS ID will be used when the
>socket is online again. That may cause user see the wrong CBM shown. But it
>takes much time to iterate all domains to restore COS ID to 0. So, we define
>a 'dom_ids[]' to represents all domains, one bit corresponds to one domain.
>If the bit is 0 when entering 'psr_ctxt_switch_to', that means this is the
>first time the domain is switched to this socket or domain's COS ID has not
>been set since the socket is online. So, the COS ID set to ASSOC register on
>this socket should be default value, 0. If not, that means the domain's COS
>ID has been set when the socket was online. So, this COS ID is valid and we
>can directly use it. We restore the domain's COS ID to 0 if the bit
>corresponding to the domain is 0 but the domain's COS ID is not 0 when
>'psr_get_val' and 'psr_set_val' is called. This can avoid CPU serialization
>if restoring action is exectued in 'psr_ctxt_switch_to'.
>
>Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v13 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (7 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 08/23] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-12 19:26 ` Jan Beulich
2017-07-06 1:53 ` [PATCH v13 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
` (13 subsequent siblings)
22 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Only can one COS ID be used by one domain at one time. That means all enabled
features' COS registers at this COS ID are valid for this domain at that time.
When user updates a feature's value, we need make sure all other features'
values are not affected. So, we firstly need gather an array which contains
all features current values and replace the setting feature's value in array
to new value.
Then, we can try to find if there is a COS ID on which all features' COS
registers values are same as the array. If we can find, we just use this COS
ID. If fail to find, we need pick a new COS ID.
This patch implements value array assembling flow.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v13:
- remove an unnecessary blank line.
(suggested by Jan Beulich)
- add a new function 'skip_prior_features()' to skip value array according
to feature type. This function will be used in later patches too.
(suggested by Jan Beulich)
v12:
- use 'ARRAY_SIZE' to calculate array boundary.
(suggested by Jan Beulich)
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exist but props does not exist.
(suggested by Jan Beulich)
- return 0 only when value is set in 'insert_val_into_array'.
(suggested by Jan Beulich)
v11:
- changes about 'feat_props'.
(suggested by Jan Beulich)
- changes about 'get_val'. It is replaced by generic codes.
(suggested by Jan Beulich)
v10:
- remove 'get_old_val' to directly call 'get_val' to get needed val.
(suggested by Jan Beulich)
- move 'psr_check_cbm' into 'insert_val_to_array'.
(suggested by Jan Beulich)
- change type of 'cbm' in 'psr_check_cbm' to 'unsigned long'.
(suggested by Jan Beulich)
- remove 'set_new_val' as it can be handled in generic process.
- changes related to 'feat_props'.
(suggested by Jan Beulich)
- adjust flow in 'gather_val_array' to avoid array cross.
(suggested by Jan Beulich)
- adjust flow in 'insert_val_to_array' to avoid array cross.
(suggested by Jan Beulich)
v9:
- add comments about boundary checking.
(suggested by Wei Liu)
- rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
(suggested by Wei Liu)
- rename 'l3_cat_get_cos_num' to 'cat_get_cos_num' to cover all L3/L2 CAT
features.
(suggested by Roger Pau)
- rename 'l3_cat_get_old_val' to 'cat_get_old_val' to cover all L3/L2 CAT
features and reuse cat_get_val in it.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- replace 'm' to 'new_val'.
(suggested by Jan Beulich)
- move cos check outside callback function.
(suggested by Jan Beulich)
- remove 'get_cos_num' callback function.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v6:
- change 'assemble_val_array' to 'combine_val_array'.
(suggested by Konrad Rzeszutek Wilk)
- check return value of 'get_old_val'.
(suggested by Konrad Rzeszutek Wilk)
- replace some 'EINVAL' to 'ENOSPC'.
(suggested by Konrad Rzeszutek Wilk)
v5:
- modify comments according to changes of codes.
(suggested by Jan Beulich)
- change 'bool_t' to 'bool'.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 134 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 132 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5aa1bcc..84a8e30 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -210,6 +210,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
return feat_type;
}
+static bool psr_check_cbm(unsigned int cbm_len, unsigned long cbm)
+{
+ unsigned int first_bit, zero_bit;
+
+ /* Set bits should only in the range of [0, cbm_len]. */
+ if ( cbm & (~0ul << cbm_len) )
+ return false;
+
+ /* At least one bit need to be set. */
+ if ( cbm == 0 )
+ return false;
+
+ first_bit = find_first_bit(&cbm, cbm_len);
+ zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
+
+ /* Set bits should be contiguous. */
+ if ( zero_bit < cbm_len &&
+ find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
+ return false;
+
+ return true;
+}
+
/* CAT common functions implementation. */
static int cat_init_feature(const struct cpuid_leaf *regs,
struct feat_node *feat,
@@ -601,7 +624,14 @@ int psr_get_val(struct domain *d, unsigned int socket,
/* Set value functions */
static unsigned int get_cos_num(void)
{
- return 0;
+ unsigned int num = 0, i;
+
+ /* Get all features total amount. */
+ for ( i = 0; i < ARRAY_SIZE(feat_props); i++ )
+ if ( feat_props[i] )
+ num += feat_props[i]->cos_num;
+
+ return num;
}
static int gather_val_array(uint32_t val[],
@@ -609,7 +639,69 @@ static int gather_val_array(uint32_t val[],
const struct psr_socket_info *info,
unsigned int old_cos)
{
- return -EINVAL;
+ unsigned int i;
+
+ if ( !val )
+ return -EINVAL;
+
+ /* Get all features current values according to old_cos. */
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ unsigned int cos = old_cos, j;
+ const struct feat_node *feat = info->features[i];
+ const struct feat_props *props = feat_props[i];
+
+ if ( !feat )
+ continue;
+
+ if ( !props )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( array_len < props->cos_num )
+ return -ENOSPC;
+
+ /*
+ * If old_cos exceeds current feature's cos_max, we should get
+ * default value. So assign cos to 0 which stores default value.
+ */
+ if ( cos > feat->cos_max )
+ cos = 0;
+
+ /* Value getting order is same as feature array. */
+ for ( j = 0; j < props->cos_num; j++ )
+ val[j] = feat->cos_reg_val[cos * props->cos_num + j];
+
+ array_len -= props->cos_num;
+ val += props->cos_num;
+ }
+
+ return 0;
+}
+
+static int skip_prior_features(uint32_t **val,
+ unsigned int *array_len,
+ enum psr_feat_type feat_type)
+{
+ unsigned int i;
+
+ for ( i = 0; i < feat_type; i++ )
+ {
+ const struct feat_props * props = feat_props[i];
+
+ if ( !props )
+ continue;
+
+ if ( *array_len <= props->cos_num )
+ return -ENOSPC;
+
+ *array_len -= props->cos_num;
+ *val += props->cos_num;
+ }
+
+ return 0;
}
static int insert_val_into_array(uint32_t val[],
@@ -619,6 +711,44 @@ static int insert_val_into_array(uint32_t val[],
enum cbm_type type,
uint32_t new_val)
{
+ const struct feat_node *feat;
+ const struct feat_props *props;
+ unsigned int i;
+ int ret;
+
+ ASSERT(feat_type < FEAT_TYPE_NUM);
+
+ ret = skip_prior_features(&val, &array_len, feat_type);
+ if ( ret )
+ return ret;
+
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ props = feat_props[feat_type];
+ if ( !props )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( array_len < props->cos_num )
+ return -ENOSPC;
+
+ if ( !psr_check_cbm(feat->cbm_len, new_val) )
+ return -EINVAL;
+
+ /* Value setting position is same as feature array. */
+ for ( i = 0; i < props->cos_num; i++ )
+ {
+ if ( type == props->type[i] )
+ {
+ val[i] = new_val;
+ return 0;
+ }
+ }
+
return -EINVAL;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v13 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array.
2017-07-06 1:53 ` [PATCH v13 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
@ 2017-07-12 19:26 ` Jan Beulich
0 siblings, 0 replies; 42+ messages in thread
From: Jan Beulich @ 2017-07-12 19:26 UTC (permalink / raw)
To: yi.y.sun, xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
>Only can one COS ID be used by one domain at one time. That means all enabled
>features' COS registers at this COS ID are valid for this domain at that time.
>
>When user updates a feature's value, we need make sure all other features'
>values are not affected. So, we firstly need gather an array which contains
>all features current values and replace the setting feature's value in array
>to new value.
>
>Then, we can try to find if there is a COS ID on which all features' COS
>registers values are same as the array. If we can find, we just use this COS
>ID. If fail to find, we need pick a new COS ID.
>
>This patch implements value array assembling flow.
>
>Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
>Reviewed-by: Jan Beulich <jbeulich@suse.com>
This was perhaps premature because of ...
>v13:
>- remove an unnecessary blank line.
>(suggested by Jan Beulich)
>- add a new function 'skip_prior_features()' to skip value array according
>to feature type. This function will be used in later patches too.
... this.
>+static int skip_prior_features(uint32_t **val,
>+ unsigned int *array_len,
>+ enum psr_feat_type feat_type)
>+{
>+ unsigned int i;
>+
>+ for ( i = 0; i < feat_type; i++ )
>+ {
>+ const struct feat_props * props = feat_props[i];
Stray blank after *. Other than that you can retain the R-b, and the cosmetic
issue here could be taken care of while committing if no other need arises for
sending another version.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v13 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (8 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 09/23] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
` (12 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Continue from patch:
'x86: refactor psr: L3 CAT: set value: assemble features value array'
We can try to find if there is a COS ID on which all features' COS registers
values are same as the array assembled before.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v13:
- remove unnecessary blank line.
(suggested by Jan Beulich)
v12:
- use 'ARRAY_SIZE' to calculate array boundary.
(suggested by Jan Beulich)
- move same type variables declaration to one line.
(suggested by Jan Beulich)
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exists but props does not exist.
(suggested by Jan Beulich)
- move 'i' definition into the loop to limit its scope.
(suggested by Jan Beulich)
v11:
- move 'compare_val' implementation from CDP patch to this.
(suggested by Jan Beulich)
- remove 'rc' in 'compare_val' to make codes clearer.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- fix bug.
(suggested by Jan Beulich)
- remove unnecessary variable initialization.
(suggested by Jan Beulich)
- remove unnecessary comment.
(suggested by Jan Beulich)
- adjust array check place prior to using the array.
(suggested by Jan Beulich)
- add comment.
v10:
- remove 'compare_val' hook and its CAT implementation. Make its
functionality be generic in 'find_cos' flow.
(suggested by Jan Beulich)
- changes related to 'props'.
(suggested by Jan Beulich)
- rename 'val_array' to 'val_ptr'.
(suggested by Jan Beulich)
- rename 'find' to 'found'.
(suggested by Jan Beulich)
- move some variables declaration and initialization into loop.
(suggested by Jan Beulich)
- adjust codes positions.
(suggested by Jan Beulich)
v9:
- modify comments of 'compare_val' to be same as current implementation.
(suggested by Wei Liu)
- fix indentation issue.
(suggested by Wei Liu)
- rename 'l3_cat_compare_val' to 'cat_compare_val' to cover all L3/L2 CAT
features.
(suggested by Roger Pau)
- remove parameter 'found' from 'cat_compare_val' and modify the return
values to let caller know if the id is found or not.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- replace 'get_cos_num' to 'feat->cos_num'.
(suggested by Jan Beulich)
- directly use 'cos_reg_val[0]' as default value.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v5:
- modify commit message to provide exact patch name to continue from.
(suggested by Jan Beulich)
- remove 'get_cos_max_from_type' because it can be replaced by
'get_cos_max'.
- move type check out from callback functions to caller.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- modify comments according to changes of codes.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 100 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 84a8e30..306b02b 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -752,10 +752,110 @@ static int insert_val_into_array(uint32_t val[],
return -EINVAL;
}
+static int compare_val(const uint32_t val[],
+ const struct feat_node *feat,
+ const struct feat_props *props,
+ unsigned int cos)
+{
+ unsigned int i;
+
+ for ( i = 0; i < props->cos_num; i++ )
+ {
+ uint32_t feat_val;
+
+ /* If cos is bigger than cos_max, we need compare default value. */
+ if ( cos > feat->cos_max )
+ {
+ /*
+ * COS ID 0 always stores the default value.
+ * For CDP:
+ * - DATA default value stored in cos_reg_val[0];
+ * - CODE default value stored in cos_reg_val[1].
+ */
+ feat_val = feat->cos_reg_val[i];
+
+ /*
+ * If cos is bigger than feature's cos_max, the val should be
+ * default value. Otherwise, it fails to find a COS ID. So we
+ * have to exit find flow.
+ */
+ if ( val[i] != feat_val )
+ return -EINVAL;
+ }
+ else
+ {
+ feat_val = feat->cos_reg_val[cos * props->cos_num + i];
+ if ( val[i] != feat_val )
+ return 0;
+ }
+ }
+
+ return 1;
+}
+
static int find_cos(const uint32_t val[], unsigned int array_len,
enum psr_feat_type feat_type,
const struct psr_socket_info *info)
{
+ unsigned int cos, cos_max;
+ const unsigned int *ref = info->cos_ref;
+ const struct feat_node *feat;
+
+ /* cos_max is the one of the feature which is being set. */
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ cos_max = feat->cos_max;
+
+ for ( cos = 0; cos <= cos_max; cos++ )
+ {
+ const uint32_t *val_ptr = val;
+ unsigned int len = array_len, i;
+ int rc = 0;
+
+ if ( cos && !ref[cos] )
+ continue;
+
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ const struct feat_props *props = feat_props[i];
+
+ feat = info->features[i];
+ if ( !feat )
+ continue;
+
+ if ( !props )
+ {
+ ASSERT_UNREACHABLE();
+ return -ENOENT;
+ }
+
+ if ( len < props->cos_num )
+ return -ENOSPC;
+
+ /*
+ * Compare value according to feature array order.
+ * We must follow this order because value array is assembled
+ * as this order.
+ */
+ rc = compare_val(val_ptr, feat, props, cos);
+ if ( rc < 0 )
+ return rc;
+
+ /* If fail to match, go to next cos to compare. */
+ if ( !rc )
+ break;
+
+ len -= props->cos_num;
+ val_ptr += props->cos_num;
+ }
+
+ /* For this COS ID all entries in the values array do match. Use it. */
+ if ( rc )
+ return cos;
+ }
+
return -ENOENT;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (9 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 10/23] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
` (11 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Continue from previous patch:
'x86: refactor psr: L3 CAT: set value: implement cos finding flow.'
If fail to find a COS ID, we need pick a new COS ID for domain. Only COS ID
that ref[COS_ID] is 1 or 0 can be picked to input a new set feature values.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v13:
- remove unnecessary blank line.
(suggested by Jan Beulich)
- remove unnecessary comment and fix comment wording.
(suggested by Jan Beulich)
v12:
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exists but props does not exist.
(suggested by Jan Beulich)
- declare same type vairables in one line.
(suggested by Jan Beulich)
- use 'ARRAY_SIZE' to calculate array boundary.
(suggested by Jan Beulich)
v11:
- remove unnecessary variable initialization.
(suggested by Jan Beulich)
- changes about 'feat_props'.
(suggested by Jan Beulich)
- replace 'get_val' calling to generic codes.
(suggested by Jan Beulich)
v10:
- remove 'fits_cos_max' hook and CAT implementation. Move the process into
generic flow.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- adjust codes positions.
(suggested by Jan Beulich)
v9:
- modify return value of 'pick_avail_cos' to make it more accurate.
- rename 'l3_cat_fits_cos_max' to 'cat_fits_cos_max' to cover L3/L2 CAT
features.
(suggested by Roger Pau)
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- fix comment.
(suggested by Wei Liu)
- directly use 'cos_reg_val[0]' as default value.
(suggested by Jan Beulich)
- replace 'get_cos_num' to 'feat->cos_num'.
(suggested by Jan Beulich)
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v5:
- modify commit message to provide exact patch name to continue from.
(suggested by Jan Beulich)
- change 'exceeds_cos_max' to 'fits_cos_max' to be accurate.
(suggested by Jan Beulich)
- modify comments according to changes of codes.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- move type check out from callback functions to caller.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- modify code format.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 80 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 306b02b..cbe08ce 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -859,12 +859,91 @@ static int find_cos(const uint32_t val[], unsigned int array_len,
return -ENOENT;
}
+static bool fits_cos_max(const uint32_t val[],
+ uint32_t array_len,
+ const struct psr_socket_info *info,
+ unsigned int cos)
+{
+ unsigned int i;
+
+ for ( i = 0; i < ARRAY_SIZE(info->features); i++ )
+ {
+ const struct feat_node *feat = info->features[i];
+ const struct feat_props *props = feat_props[i];
+
+ if ( !feat )
+ continue;
+
+ if ( !props )
+ {
+ ASSERT_UNREACHABLE();
+ return false;
+ }
+
+ if ( array_len < props->cos_num )
+ return false;
+
+ if ( cos > feat->cos_max )
+ {
+ unsigned int j;
+
+ for ( j = 0; j < props->cos_num; j++ )
+ {
+ /* Get default value, the COS ID of which is zero. */
+ uint32_t default_val = feat->cos_reg_val[j];
+
+ if ( val[j] != default_val )
+ return false;
+ }
+ }
+
+ array_len -= props->cos_num;
+ val += props->cos_num;
+ }
+
+ return true;
+}
+
static int pick_avail_cos(const struct psr_socket_info *info,
const uint32_t val[], unsigned int array_len,
unsigned int old_cos,
enum psr_feat_type feat_type)
{
- return -ENOENT;
+ unsigned int cos, cos_max = 0;
+ const struct feat_node *feat;
+ const unsigned int *ref = info->cos_ref;
+
+ /* cos_max is the one of the feature which is being set. */
+ feat = info->features[feat_type];
+ if ( !feat )
+ return -ENOENT;
+
+ cos_max = feat->cos_max;
+ if ( !cos_max )
+ return -ENOENT;
+
+ /* We cannot use id 0 because it stores the default values. */
+ if ( old_cos && ref[old_cos] == 1 &&
+ fits_cos_max(val, array_len, info, old_cos) )
+ return old_cos;
+
+ /* Find an unused one other than cos0. */
+ for ( cos = 1; cos <= cos_max; cos++ )
+ {
+ /*
+ * ref is 0 means this COS is not used by other domain and
+ * can be used for current setting.
+ */
+ if ( !ref[cos] )
+ {
+ if ( !fits_cos_max(val, array_len, info, cos) )
+ break;
+
+ return cos;
+ }
+ }
+
+ return -EOVERFLOW;
}
static int write_psr_msrs(unsigned int socket, unsigned int cos,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (10 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 11/23] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-12 19:37 ` Jan Beulich
2017-07-06 1:53 ` [PATCH v13 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
` (10 subsequent siblings)
22 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
Continue from previous patch:
'x86: refactor psr: L3 CAT: set value: implement cos id picking flow.'
We have got the feature value and COS ID to set. Then, we write MSRs of the
designated feature.
Till now, set value process is completed.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v13:
- use 'skip_prior_features'.
(suggested by Jan Beulich)
- add 'const' for some variables.
(suggested by Jan Beulich)
v12:
- declare same type varaibles in one line.
(suggested by Jan Beulich)
- replace 'feat_type' to 'props' in 'struct cos_write_info'.
(suggested by Jan Beulich)
- assign the 'cos_num' to a local variable.
(suggested by Jan Beulich)
- use 'ASSERT_UNREACHABLE()' to record bug and return error code if feat
exists but props does not exist.
(suggested by Jan Beulich)
v11:
- rename 'write_psr_msr' to 'write_psr_msrs'.
- rename 'do_write_psr_msr' to 'do_write_psr_msrs'.
- change parameters and codes of 'write_psr_msrs' to handle value array.
- add 'feat_type' in 'struct cos_write_info' to handle props array.
- in 'do_write_psr_msrs', write value array into msrs according to
'props->type[i]'.
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
- move check from 'do_write_psr_msrs' to 'write_psr_msrs'.
(suggested by Jan Beulich)
- change about 'cos_max'.
(suggested by Jan Beulich)
- change about 'feat_props'.
(suggested by Jan Beulich)
v10:
- remove 'type' from 'write_msr' parameter list. Will add it back when
implementing CDP.
(suggested by Jan Beulich)
- remove unnecessary casts.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
v9:
- replace feature list handling to feature array handling.
(suggested by Roger Pau)
- add 'array_len' in 'struct cos_write_info' and check if val array
exceeds it.
- modify 'write_psr_msr' flow only to set one value a time. No need to
set whole feature array values.
- modify patch title to indicate 'L3 CAT'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- modify 'write_msr' callback function to 'void' because we have to set
all features' cbm. When input cos exceeds some features' cos_max, just
skip them but not break the iteration.
v5:
- modify commit message to provide exact patch name to continue from.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- move type check out from callback functions to caller.
(suggested by Jan Beulich)
- modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
(suggested by Jan Beulich)
- correct code format.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 76 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index cbe08ce..48dab60 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -113,6 +113,9 @@ static const struct feat_props {
/* get_feat_info is used to return feature HW info through sysctl. */
bool (*get_feat_info)(const struct feat_node *feat,
uint32_t data[], unsigned int array_len);
+
+ /* write_msr is used to write out feature MSR register. */
+ void (*write_msr)(unsigned int cos, uint32_t val, enum cbm_type type);
} *feat_props[FEAT_TYPE_NUM];
/*
@@ -289,11 +292,17 @@ static bool cat_get_feat_info(const struct feat_node *feat,
}
/* L3 CAT props */
+static void l3_cat_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
+{
+ wrmsrl(MSR_IA32_PSR_L3_MASK(cos), val);
+}
+
static const struct feat_props l3_cat_props = {
.cos_num = 1,
.type[0] = PSR_CBM_TYPE_L3,
.alt_type = PSR_CBM_TYPE_UNKNOWN,
.get_feat_info = cat_get_feat_info,
+ .write_msr = l3_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -946,11 +955,77 @@ static int pick_avail_cos(const struct psr_socket_info *info,
return -EOVERFLOW;
}
+static unsigned int get_socket_cpu(unsigned int socket)
+{
+ if ( likely(socket < nr_sockets) )
+ return cpumask_any(socket_cpumask[socket]);
+
+ return nr_cpu_ids;
+}
+
+struct cos_write_info
+{
+ unsigned int cos;
+ struct feat_node *feature;
+ const uint32_t *val;
+ const struct feat_props *props;
+};
+
+static void do_write_psr_msrs(void *data)
+{
+ const struct cos_write_info *info = data;
+ struct feat_node *feat = info->feature;
+ const struct feat_props *props = info->props;
+ unsigned int i, cos = info->cos, cos_num = props->cos_num;
+
+ for ( i = 0; i < cos_num; i++ )
+ {
+ if ( feat->cos_reg_val[cos * cos_num + i] != info->val[i] )
+ {
+ feat->cos_reg_val[cos * cos_num + i] = info->val[i];
+ props->write_msr(cos, info->val[i], props->type[i]);
+ }
+ }
+}
+
static int write_psr_msrs(unsigned int socket, unsigned int cos,
uint32_t val[], unsigned int array_len,
enum psr_feat_type feat_type)
{
- return -ENOENT;
+ int ret;
+ struct psr_socket_info *info = get_socket_info(socket);
+ struct cos_write_info data =
+ {
+ .cos = cos,
+ .feature = info->features[feat_type],
+ .props = feat_props[feat_type],
+ };
+
+ if ( cos > info->features[feat_type]->cos_max )
+ return -EINVAL;
+
+ /* Skip to the feature's value head. */
+ ret = skip_prior_features(&val, &array_len, feat_type);
+ if ( ret )
+ return ret;
+
+ if ( array_len < feat_props[feat_type]->cos_num )
+ return -ENOSPC;
+
+ data.val = val;
+
+ if ( socket == cpu_to_socket(smp_processor_id()) )
+ do_write_psr_msrs(&data);
+ else
+ {
+ unsigned int cpu = get_socket_cpu(socket);
+
+ if ( cpu >= nr_cpu_ids )
+ return -ENOTSOCK;
+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msrs, &data, 1);
+ }
+
+ return 0;
}
int psr_set_val(struct domain *d, unsigned int socket,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-07-06 1:53 ` [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
@ 2017-07-12 19:37 ` Jan Beulich
2017-07-13 2:59 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2017-07-12 19:37 UTC (permalink / raw)
To: yi.y.sun, xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
>v13:
>- use 'skip_prior_features'.
>- add 'const' for some variables.
You didn't go quite far enough with this:
>+struct cos_write_info
>+{
>+ unsigned int cos;
>+ struct feat_node *feature;
>+ const uint32_t *val;
With this, ...
>static int write_psr_msrs(unsigned int socket, unsigned int cos,
>uint32_t val[], unsigned int array_len,
... I can't see why this can't be const too. Of course that would then affect an
earlier patch.
>enum psr_feat_type feat_type)
>{
>- return -ENOENT;
>+ int ret;
>+ struct psr_socket_info *info = get_socket_info(socket);
>+ struct cos_write_info data =
>+ {
>+ .cos = cos,
>+ .feature = info->features[feat_type],
>+ .props = feat_props[feat_type],
>+ };
>+
>+ if ( cos > info->features[feat_type]->cos_max )
>+ return -EINVAL;
>+
>+ /* Skip to the feature's value head. */
>+ ret = skip_prior_features(&val, &array_len, feat_type);
>+ if ( ret )
>+ return ret;
>+
>+ if ( array_len < feat_props[feat_type]->cos_num )
>+ return -ENOSPC;
>+
>+ data.val = val;
>+
>+ if ( socket == cpu_to_socket(smp_processor_id()) )
>+ do_write_psr_msrs(&data);
>+ else
>+ {
>+ unsigned int cpu = get_socket_cpu(socket);
>+
>+ if ( cpu >= nr_cpu_ids )
>+ return -ENOTSOCK;
>+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msrs, &data, 1);
How frequent an operation can this be? Considering that the actual MSR write(s)
in the handler is (are) conditional I wonder whether it wouldn't be worthwhile
trying to avoid the IPI altogether, by pre-checking whether any write actually
needs doing.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-07-12 19:37 ` Jan Beulich
@ 2017-07-13 2:59 ` Yi Sun
2017-07-13 5:20 ` Jan Beulich
0 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-13 2:59 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau
On 17-07-12 13:37:02, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
> >v13:
> >- use 'skip_prior_features'.
> >- add 'const' for some variables.
>
> You didn't go quite far enough with this:
>
> >+struct cos_write_info
> >+{
> >+ unsigned int cos;
> >+ struct feat_node *feature;
> >+ const uint32_t *val;
>
> With this, ...
>
> >static int write_psr_msrs(unsigned int socket, unsigned int cos,
> >uint32_t val[], unsigned int array_len,
>
> ... I can't see why this can't be const too. Of course that would then affect an
> earlier patch.
>
The 'val' is input into 'skip_prior_features'. In 'skip_prior_features', there
is '*val += props->cos_num;' to change the value. So, I do not add 'const' here.
Of course, I can change the way to skip value array, e.g. using a variable as
index. Which one do you like?
> >enum psr_feat_type feat_type)
> >{
> >- return -ENOENT;
> >+ int ret;
> >+ struct psr_socket_info *info = get_socket_info(socket);
> >+ struct cos_write_info data =
> >+ {
> >+ .cos = cos,
> >+ .feature = info->features[feat_type],
> >+ .props = feat_props[feat_type],
> >+ };
> >+
> >+ if ( cos > info->features[feat_type]->cos_max )
> >+ return -EINVAL;
> >+
> >+ /* Skip to the feature's value head. */
> >+ ret = skip_prior_features(&val, &array_len, feat_type);
> >+ if ( ret )
> >+ return ret;
> >+
> >+ if ( array_len < feat_props[feat_type]->cos_num )
> >+ return -ENOSPC;
> >+
> >+ data.val = val;
> >+
> >+ if ( socket == cpu_to_socket(smp_processor_id()) )
> >+ do_write_psr_msrs(&data);
> >+ else
> >+ {
> >+ unsigned int cpu = get_socket_cpu(socket);
> >+
> >+ if ( cpu >= nr_cpu_ids )
> >+ return -ENOTSOCK;
> >+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msrs, &data, 1);
>
> How frequent an operation can this be? Considering that the actual MSR write(s)
> in the handler is (are) conditional I wonder whether it wouldn't be worthwhile
> trying to avoid the IPI altogether, by pre-checking whether any write actually
> needs doing.
>
Yes, I think I can check if the value to set is same as 'feat->cos_reg_val[cos]'
before calling IPI.
There is one more thing. During implementing MBA, I find there is an issue here.
The current codes in 'struct cos_write_info' and 'write_psr_msrs' only consider
one feature's value setting. In fact, we should consider to set all values in
'val' array to the MSRs with new cos id for all features.
So, the 'cos_write_info' should be something like below to input feature array
and props array to handle all features. Of course, we do not need skip value
array anymore.
struct cos_write_info
{
unsigned int cos;
struct feat_node **features;
uint32_t *val;
unsigned int array_len;
const struct feat_props **props;
};
> Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-07-13 2:59 ` Yi Sun
@ 2017-07-13 5:20 ` Jan Beulich
2017-07-13 7:32 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2017-07-13 5:20 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 07/13/17 5:00 AM >>>
>On 17-07-12 13:37:02, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
>> >v13:
>> >- use 'skip_prior_features'.
>> >- add 'const' for some variables.
>>
>> You didn't go quite far enough with this:
>>
>> >+struct cos_write_info
>> >+{
>> >+ unsigned int cos;
>> >+ struct feat_node *feature;
>> >+ const uint32_t *val;
>>
>> With this, ...
>>
>> >static int write_psr_msrs(unsigned int socket, unsigned int cos,
>> >uint32_t val[], unsigned int array_len,
>>
>> ... I can't see why this can't be const too. Of course that would then affect an
>> earlier patch.
>>
>The 'val' is input into 'skip_prior_features'. In 'skip_prior_features', there
>is '*val += props->cos_num;' to change the value. So, I do not add 'const' here.
>Of course, I can change the way to skip value array, e.g. using a variable as
>index. Which one do you like?
Oh, I see. But yes, I still think it would be nice for const-ness to be
expressible irrespective of this helper function, so making it e.g. just update
"len" without passing in the array pointer at all (leaving that part to the caller)
would seem desirable. Or possibly not even pass "array_len" via indirection,
instead making the function return a non-negative increment value for the
caller to apply to both (keeping negative value to indicate errors). But if you
think it's better the way it is, I could also live with it.
>> >+ if ( socket == cpu_to_socket(smp_processor_id()) )
>> >+ do_write_psr_msrs(&data);
>> >+ else
>> >+ {
>> >+ unsigned int cpu = get_socket_cpu(socket);
>> >+
>> >+ if ( cpu >= nr_cpu_ids )
>> >+ return -ENOTSOCK;
>> >+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msrs, &data, 1);
>>
>> How frequent an operation can this be? Considering that the actual MSR write(s)
>> in the handler is (are) conditional I wonder whether it wouldn't be worthwhile
>> trying to avoid the IPI altogether, by pre-checking whether any write actually
>> needs doing.
>>
>Yes, I think I can check if the value to set is same as 'feat->cos_reg_val[cos]'
>before calling IPI.
Well, as said - whether it's worth the extra effort depends on whether there is
a (reasonable) scenario where this function may be executed frequently.
>There is one more thing. During implementing MBA, I find there is an issue here.
>The current codes in 'struct cos_write_info' and 'write_psr_msrs' only consider
>one feature's value setting. In fact, we should consider to set all values in
>'val' array to the MSRs with new cos id for all features.
>
>So, the 'cos_write_info' should be something like below to input feature array
>and props array to handle all features. Of course, we do not need skip value
>array anymore.
>
>struct cos_write_info
>{
>unsigned int cos;
>struct feat_node **features;
>uint32_t *val;
>unsigned int array_len;
>const struct feat_props **props;
>};
As you can likely understand, I can't really judge on this without seeing what
you need this for. So I'd suggest to keep things the way they are in this series
and discuss changes to it in the context of that other series of yours.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-07-13 5:20 ` Jan Beulich
@ 2017-07-13 7:32 ` Yi Sun
2017-07-13 19:21 ` Jan Beulich
0 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-13 7:32 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau
On 17-07-12 23:20:24, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/13/17 5:00 AM >>>
> >On 17-07-12 13:37:02, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
> >> >v13:
> >> >- use 'skip_prior_features'.
> >> >- add 'const' for some variables.
> >>
> >> You didn't go quite far enough with this:
> >>
> >> >+struct cos_write_info
> >> >+{
> >> >+ unsigned int cos;
> >> >+ struct feat_node *feature;
> >> >+ const uint32_t *val;
> >>
> >> With this, ...
> >>
> >> >static int write_psr_msrs(unsigned int socket, unsigned int cos,
> >> >uint32_t val[], unsigned int array_len,
> >>
> >> ... I can't see why this can't be const too. Of course that would then affect an
> >> earlier patch.
> >>
> >The 'val' is input into 'skip_prior_features'. In 'skip_prior_features', there
> >is '*val += props->cos_num;' to change the value. So, I do not add 'const' here.
> >Of course, I can change the way to skip value array, e.g. using a variable as
> >index. Which one do you like?
>
> Oh, I see. But yes, I still think it would be nice for const-ness to be
> expressible irrespective of this helper function, so making it e.g. just update
> "len" without passing in the array pointer at all (leaving that part to the caller)
> would seem desirable. Or possibly not even pass "array_len" via indirection,
> instead making the function return a non-negative increment value for the
> caller to apply to both (keeping negative value to indicate errors). But if you
> think it's better the way it is, I could also live with it.
>
Thank you! I will try to implement a version out according to your comments.
> >> >+ if ( socket == cpu_to_socket(smp_processor_id()) )
> >> >+ do_write_psr_msrs(&data);
> >> >+ else
> >> >+ {
> >> >+ unsigned int cpu = get_socket_cpu(socket);
> >> >+
> >> >+ if ( cpu >= nr_cpu_ids )
> >> >+ return -ENOTSOCK;
> >> >+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msrs, &data, 1);
> >>
> >> How frequent an operation can this be? Considering that the actual MSR write(s)
> >> in the handler is (are) conditional I wonder whether it wouldn't be worthwhile
> >> trying to avoid the IPI altogether, by pre-checking whether any write actually
> >> needs doing.
> >>
> >Yes, I think I can check if the value to set is same as 'feat->cos_reg_val[cos]'
> >before calling IPI.
>
> Well, as said - whether it's worth the extra effort depends on whether there is
> a (reasonable) scenario where this function may be executed frequently.
>
This function is executed when 'psr-cat-set' command is executed. I consult
the libvirt guy, this command may be executed frequently under some scenarios.
E.g. user may dynamically adjust the cache allocation for VMs according to CMT
result.
> >There is one more thing. During implementing MBA, I find there is an issue here.
> >The current codes in 'struct cos_write_info' and 'write_psr_msrs' only consider
> >one feature's value setting. In fact, we should consider to set all values in
> >'val' array to the MSRs with new cos id for all features.
> >
> >So, the 'cos_write_info' should be something like below to input feature array
> >and props array to handle all features. Of course, we do not need skip value
> >array anymore.
> >
> >struct cos_write_info
> >{
> >unsigned int cos;
> >struct feat_node **features;
> >uint32_t *val;
> >unsigned int array_len;
> >const struct feat_props **props;
> >};
>
> As you can likely understand, I can't really judge on this without seeing what
> you need this for. So I'd suggest to keep things the way they are in this series
> and discuss changes to it in the context of that other series of yours.
>
Ok, I will keep the codes in current series. Will modify them in MBA patch set
for review.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-07-13 7:32 ` Yi Sun
@ 2017-07-13 19:21 ` Jan Beulich
2017-07-14 1:38 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2017-07-13 19:21 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 07/13/17 9:34 AM >>>
>On 17-07-12 23:20:24, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/13/17 5:00 AM >>>
>> >On 17-07-12 13:37:02, Jan Beulich wrote:
>> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
>> >> >+ if ( socket == cpu_to_socket(smp_processor_id()) )
>> >> >+ do_write_psr_msrs(&data);
>> >> >+ else
>> >> >+ {
>> >> >+ unsigned int cpu = get_socket_cpu(socket);
>> >> >+
>> >> >+ if ( cpu >= nr_cpu_ids )
>> >> >+ return -ENOTSOCK;
>> >> >+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msrs, &data, 1);
>> >>
>> >> How frequent an operation can this be? Considering that the actual MSR write(s)
>> >> in the handler is (are) conditional I wonder whether it wouldn't be worthwhile
>> >> trying to avoid the IPI altogether, by pre-checking whether any write actually
>> >> needs doing.
>> >>
>> >Yes, I think I can check if the value to set is same as 'feat->cos_reg_val[cos]'
>> >before calling IPI.
>>
>> Well, as said - whether it's worth the extra effort depends on whether there is
>> a (reasonable) scenario where this function may be executed frequently.
>>
>This function is executed when 'psr-cat-set' command is executed. I consult
>the libvirt guy, this command may be executed frequently under some scenarios.
>E.g. user may dynamically adjust the cache allocation for VMs according to CMT
>result.
Hmm, that's not something I would call frequent - in the whole invocation of the
user mode process the IPI will be lost in the noise. "Frequent" would be something
the kernel does without direct user mode triggering, like on the context switch
path, in code running from a timer, or some such.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow.
2017-07-13 19:21 ` Jan Beulich
@ 2017-07-14 1:38 ` Yi Sun
0 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-14 1:38 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau
On 17-07-13 13:21:46, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/13/17 9:34 AM >>>
> >On 17-07-12 23:20:24, Jan Beulich wrote:
> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/13/17 5:00 AM >>>
> >> >On 17-07-12 13:37:02, Jan Beulich wrote:
> >> >> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
> >> >> >+ if ( socket == cpu_to_socket(smp_processor_id()) )
> >> >> >+ do_write_psr_msrs(&data);
> >> >> >+ else
> >> >> >+ {
> >> >> >+ unsigned int cpu = get_socket_cpu(socket);
> >> >> >+
> >> >> >+ if ( cpu >= nr_cpu_ids )
> >> >> >+ return -ENOTSOCK;
> >> >> >+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msrs, &data, 1);
> >> >>
> >> >> How frequent an operation can this be? Considering that the actual MSR write(s)
> >> >> in the handler is (are) conditional I wonder whether it wouldn't be worthwhile
> >> >> trying to avoid the IPI altogether, by pre-checking whether any write actually
> >> >> needs doing.
> >> >>
> >> >Yes, I think I can check if the value to set is same as 'feat->cos_reg_val[cos]'
> >> >before calling IPI.
> >>
> >> Well, as said - whether it's worth the extra effort depends on whether there is
> >> a (reasonable) scenario where this function may be executed frequently.
> >>
> >This function is executed when 'psr-cat-set' command is executed. I consult
> >the libvirt guy, this command may be executed frequently under some scenarios.
> >E.g. user may dynamically adjust the cache allocation for VMs according to CMT
> >result.
>
> Hmm, that's not something I would call frequent - in the whole invocation of the
> user mode process the IPI will be lost in the noise. "Frequent" would be something
> the kernel does without direct user mode triggering, like on the context switch
> path, in code running from a timer, or some such.
>
Then, it is not 'Frequent'. This function is only trigger by user. So, I
will keep current codes. Thanks!
> Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v13 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (11 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 12/23] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-12 19:52 ` Jan Beulich
2017-07-06 1:53 ` [PATCH v13 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
` (9 subsequent siblings)
22 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the CPU init flow for CDP. The flow is almost
same as L3 CAT.
Note: CDP does NOT work until you apply the later patches of CDP.
"x86: refactor psr: CDP: implement get hw info flow."
"x86: refactor psr: CDP: implement set value callback function."
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v13:
- add commit message.
(suggested by Jan Beulich)
- fix comment issue.
(suggested by Jan Beulich)
- set CDP default value before enabling it.
(suggested by Jan Beulich)
- remove unnecessary check.
(suggested by Jan Beulich)
- set 'alt_type' for CDP.
(suggested by Jan Beulich)
- check 'cos_max' and substract 1 before right shift it to get correct
value.
(suggested by Jan Beulich)
v12:
- move 'type[]' assignment into l3_cdp_props declaration to make it be
'const'.
(suggested by Jan Beulich)
- remove "L2 CAT" indication in printk.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- change 'val' type to uint64_t.
(suggested by Jan Beulich)
- use 1ull.
(suggested by Jan Beulich)
- restore mask(0) MSR to default value.
(suggested by Jan Beulich)
v11:
- changes about 'feat_props'.
(suggested by Jan Beulich)
- remove MSR restore action which is unnecessary.
(suggested by Jan Beulich)
- modify commit message.
v10:
- fix comment.
(suggested by Jan Beulich)
- use swith in 'cat_init_feature' to handle different feature types.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- restore MSRs to default value when cpu online.
(suggested by Jan Beulich)
- remove feat_mask.
(suggested by Jan Beulich)
v9:
- modify commit message to describe flow clearer.
- handle cpu offline and online again case to read MSRs registers values
back and save them into cos array to make user can get real data.
- modify error handling process in 'psr_cpu_prepare' to reduce redundant
codes.
- modify 'get_cdp_data' and 'get_cdp_code' to make them standard.
(suggested by Roger Pau and Jan Beulich)
- encapsulate CDP operations into 'cat_init_feature' to reduce redundant
codes.
(suggested by Roger Pau)
- reuse 'cat_get_cos_max' for CDP.
(suggested by Roger Pau)
- handle 'PSR_CDP' in psr_presmp_init to make init work can be done when
there is only 'psr=cdp' in cmdline.
- remove unnecessary comment.
(suggested by Jan Beulich)
- move CDP related codes in 'cpu_init_work' into 'psr_cpu_init'.
(suggested by Jan Beulich)
- add codes to handle CDP's 'cos_num'.
(suggested by Jan Beulich)
- fix coding style issue.
(suggested by Jan Beulich)
- do not free resources when allocation fails in 'psr_cpu_prepare'.
(suggested by Jan Beulich)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v7:
- initialize 'l3_cdp'.
(suggested by Konrad Rzeszutek Wilk)
v6:
- use 'cpuid_leaf'.
(suggested by Konrad Rzeszutek Wilk and Jan Beulich)
v5:
- remove codes to free 'feat_l3_cdp' in 'free_feature'.
(suggested by Jan Beulich)
- encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
(suggested by Jan Beulich)
- print socket info when 'opt_cpu_info' is true.
(suggested by Jan Beulich)
- rename 'l3_cdp_get_max_cos_max' to 'l3_cdp_get_cos_max'.
(suggested by Jan Beulich)
- rename 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
- move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 70 insertions(+), 7 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 48dab60..13d5c2f 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -62,6 +62,7 @@
enum psr_feat_type {
FEAT_TYPE_L3_CAT,
+ FEAT_TYPE_L3_CDP,
FEAT_TYPE_NUM,
FEAT_TYPE_UNKNOWN,
};
@@ -158,11 +159,28 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
* array creation. It is used to transiently store a spare node.
*/
static struct feat_node *feat_l3_cat;
+static struct feat_node *feat_l3_cdp;
/* Common functions */
#define cat_default_val(len) (0xffffffff >> (32 - (len)))
/*
+ * get_cdp_data - get DATA COS register value from input COS ID.
+ * @feat: the feature node.
+ * @cos: the COS ID.
+ */
+#define get_cdp_data(feat, cos) \
+ ((feat)->cos_reg_val[(cos) * 2])
+
+/*
+ * get_cdp_code - get CODE COS register value from input COS ID.
+ * @feat: the feature node.
+ * @cos: the COS ID.
+ */
+#define get_cdp_code(feat, cos) \
+ ((feat)->cos_reg_val[(cos) * 2 + 1])
+
+/*
* Use this function to check if any allocation feature has been enabled
* in cmdline.
*/
@@ -262,6 +280,29 @@ static int cat_init_feature(const struct cpuid_leaf *regs,
break;
+ case FEAT_TYPE_L3_CDP:
+ {
+ uint64_t val;
+
+ if ( feat->cos_max < 3 )
+ return -ENOENT;
+
+ /* Cut half of cos_max when CDP is enabled. */
+ feat->cos_max = (feat->cos_max - 1) >> 1;
+
+ /* We reserve cos=0 as default cbm (all bits within cbm_len are 1). */
+ get_cdp_code(feat, 0) = cat_default_val(feat->cbm_len);
+ get_cdp_data(feat, 0) = cat_default_val(feat->cbm_len);
+
+ wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
+ wrmsrl(MSR_IA32_PSR_L3_MASK(1), cat_default_val(feat->cbm_len));
+ rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
+ wrmsrl(MSR_IA32_PSR_L3_QOS_CFG,
+ val | (1ull << PSR_L3_QOS_CDP_ENABLE_BIT));
+
+ break;
+ }
+
default:
return -ENOENT;
}
@@ -272,7 +313,8 @@ static int cat_init_feature(const struct cpuid_leaf *regs,
if ( !opt_cpu_info )
return 0;
- printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ ((type == FEAT_TYPE_L3_CDP) ? "CDP" : "L3 CAT"),
cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
return 0;
@@ -305,6 +347,14 @@ static const struct feat_props l3_cat_props = {
.write_msr = l3_cat_write_msr,
};
+/* L3 CDP props */
+static const struct feat_props l3_cdp_props = {
+ .cos_num = 2,
+ .type[0] = PSR_CBM_TYPE_L3_DATA,
+ .type[1] = PSR_CBM_TYPE_L3_CODE,
+ .alt_type = FEAT_TYPE_L3_CAT,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1246,6 +1296,10 @@ static int psr_cpu_prepare(void)
(feat_l3_cat = xzalloc(struct feat_node)) == NULL )
return -ENOMEM;
+ if ( feat_l3_cdp == NULL &&
+ (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
@@ -1277,11 +1331,20 @@ static void psr_cpu_init(void)
{
cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
- feat = feat_l3_cat;
- feat_l3_cat = NULL;
-
- if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CAT) )
- feat_props[FEAT_TYPE_L3_CAT] = &l3_cat_props;
+ if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) )
+ {
+ feat = feat_l3_cdp;
+ feat_l3_cdp = NULL;
+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CDP) )
+ feat_props[FEAT_TYPE_L3_CDP] = &l3_cdp_props;
+ }
+ else
+ {
+ feat = feat_l3_cat;
+ feat_l3_cat = NULL;
+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CAT) )
+ feat_props[FEAT_TYPE_L3_CAT] = &l3_cat_props;
+ }
}
info->feat_init = true;
@@ -1343,7 +1406,7 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- if ( opt_psr & PSR_CAT )
+ if ( opt_psr & (PSR_CAT | PSR_CDP) )
init_psr();
if ( psr_cpu_prepare() )
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v13 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-07-06 1:53 ` [PATCH v13 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
@ 2017-07-12 19:52 ` Jan Beulich
2017-07-13 3:02 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2017-07-12 19:52 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
>This patch implements the CPU init flow for CDP. The flow is almost
>same as L3 CAT.
>
>Note: CDP does NOT work until you apply the later patches of CDP.
>"x86: refactor psr: CDP: implement get hw info flow."
>"x86: refactor psr: CDP: implement set value callback function."
This is _not_ what I did ask for in what I assume ...
>v13:
>- add commit message.
... this refers to: The problem isn't that it won't work, but that it'll crash the
hypervisor. So what I had expected you to add is a note _after_ the first
--- separator that this patch should not be applied without the other two
ones. Or alternatively for this one to add stubs which the subsequent
patches would then fill.
>@@ -262,6 +280,29 @@ static int cat_init_feature(const struct cpuid_leaf *regs,
>
>break;
>
>+ case FEAT_TYPE_L3_CDP:
>+ {
>+ uint64_t val;
>+
>+ if ( feat->cos_max < 3 )
>+ return -ENOENT;
In the admittedly unlikely event that this return path gets taken (or any other
current or future one), ...
>@@ -1277,11 +1331,20 @@ static void psr_cpu_init(void)
>{
>cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
>
>- feat = feat_l3_cat;
>- feat_l3_cat = NULL;
>-
>- if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CAT) )
>- feat_props[FEAT_TYPE_L3_CAT] = &l3_cat_props;
>+ if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) )
>+ {
>+ feat = feat_l3_cdp;
>+ feat_l3_cdp = NULL;
>+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CDP) )
>+ feat_props[FEAT_TYPE_L3_CDP] = &l3_cdp_props;
>+ }
>+ else
>+ {
>+ feat = feat_l3_cat;
>+ feat_l3_cat = NULL;
>+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CAT) )
>+ feat_props[FEAT_TYPE_L3_CAT] = &l3_cat_props;
>+ }
... wouldn't it be a good idea to then try to setup plain L3 CAT here?
I also notice that (already in patch 4) you leak feat here in case
cat_init_feature() fails. You should put it back into the static helper variable
you've taken it from. Furthermore, since you use at most one of feat_l3_cat
and feat_l3_cdp, so I don't see why you need to allocate two of them.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-07-12 19:52 ` Jan Beulich
@ 2017-07-13 3:02 ` Yi Sun
2017-07-13 5:24 ` Jan Beulich
0 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-13 3:02 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
On 17-07-12 13:52:35, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
> >This patch implements the CPU init flow for CDP. The flow is almost
> >same as L3 CAT.
> >
> >Note: CDP does NOT work until you apply the later patches of CDP.
> >"x86: refactor psr: CDP: implement get hw info flow."
> >"x86: refactor psr: CDP: implement set value callback function."
>
> This is _not_ what I did ask for in what I assume ...
>
> >v13:
> >- add commit message.
>
> ... this refers to: The problem isn't that it won't work, but that it'll crash the
> hypervisor. So what I had expected you to add is a note _after_ the first
> --- separator that this patch should not be applied without the other two
> ones. Or alternatively for this one to add stubs which the subsequent
> patches would then fill.
>
Oh, ok. I will move the comments under the first '---'. Furthermore, I'd like
to add a 'stub_write_msr()' here to avoid crash. Will replace the stub function
to real function when implementing the write flow.
> >@@ -262,6 +280,29 @@ static int cat_init_feature(const struct cpuid_leaf *regs,
> >
> >break;
> >
> >+ case FEAT_TYPE_L3_CDP:
> >+ {
> >+ uint64_t val;
> >+
> >+ if ( feat->cos_max < 3 )
> >+ return -ENOENT;
>
> In the admittedly unlikely event that this return path gets taken (or any other
> current or future one), ...
>
> >@@ -1277,11 +1331,20 @@ static void psr_cpu_init(void)
> >{
> >cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> >
> >- feat = feat_l3_cat;
> >- feat_l3_cat = NULL;
> >-
> >- if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CAT) )
> >- feat_props[FEAT_TYPE_L3_CAT] = &l3_cat_props;
> >+ if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) )
> >+ {
> >+ feat = feat_l3_cdp;
> >+ feat_l3_cdp = NULL;
> >+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CDP) )
> >+ feat_props[FEAT_TYPE_L3_CDP] = &l3_cdp_props;
> >+ }
> >+ else
> >+ {
> >+ feat = feat_l3_cat;
> >+ feat_l3_cat = NULL;
> >+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CAT) )
> >+ feat_props[FEAT_TYPE_L3_CAT] = &l3_cat_props;
> >+ }
>
> ... wouldn't it be a good idea to then try to setup plain L3 CAT here?
>
DYM if CDP init fails, enter CAT init flow to make CAT work at least?
> I also notice that (already in patch 4) you leak feat here in case
> cat_init_feature() fails. You should put it back into the static helper variable
> you've taken it from. Furthermore, since you use at most one of feat_l3_cat
> and feat_l3_cdp, so I don't see why you need to allocate two of them.
>
Ok, I will keep the helper variable if cat_init_feature() fails.
Will converge the 'feat_l3_cat' and 'feat_l3_cdp' to one.
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-07-13 3:02 ` Yi Sun
@ 2017-07-13 5:24 ` Jan Beulich
2017-07-13 5:28 ` Jan Beulich
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2017-07-13 5:24 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 07/13/17 5:04 AM >>>
>On 17-07-12 13:52:35, Jan Beulich wrote:
>> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
>> >This patch implements the CPU init flow for CDP. The flow is almost
>> >same as L3 CAT.
>> >
>> >Note: CDP does NOT work until you apply the later patches of CDP.
>> >"x86: refactor psr: CDP: implement get hw info flow."
>> >"x86: refactor psr: CDP: implement set value callback function."
>>
>> This is _not_ what I did ask for in what I assume ...
>>
>> >v13:
>> >- add commit message.
>>
>> ... this refers to: The problem isn't that it won't work, but that it'll crash the
>> hypervisor. So what I had expected you to add is a note _after_ the first
>> --- separator that this patch should not be applied without the other two
>> ones. Or alternatively for this one to add stubs which the subsequent
>> patches would then fill.
>>
>Oh, ok. I will move the comments under the first '---'. Furthermore, I'd like
>to add a 'stub_write_msr()' here to avoid crash. Will replace the stub function
>to real function when implementing the write flow.
Just to avoid any misunderstanding: I hope you don't mean to indeed
introduce and then replace a function by that name. Just like you've done
in earlier patches, introduce the function with it final name right away, and
just fill its body in the later patch.
>> >+ if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) )
>> >+ {
>> >+ feat = feat_l3_cdp;
>> >+ feat_l3_cdp = NULL;
>> >+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CDP) )
>> >+ feat_props[FEAT_TYPE_L3_CDP] = &l3_cdp_props;
>> >+ }
>> >+ else
>> >+ {
>> >+ feat = feat_l3_cat;
>> >+ feat_l3_cat = NULL;
>> >+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L3_CAT) )
>> >+ feat_props[FEAT_TYPE_L3_CAT] = &l3_cat_props;
>> >+ }
>>
>> ... wouldn't it be a good idea to then try to setup plain L3 CAT here?
>>
>DYM if CDP init fails, enter CAT init flow to make CAT work at least?
Yes.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 13/23] x86: refactor psr: CDP: implement CPU init flow.
2017-07-13 5:24 ` Jan Beulich
@ 2017-07-13 5:28 ` Jan Beulich
0 siblings, 0 replies; 42+ messages in thread
From: Jan Beulich @ 2017-07-13 5:28 UTC (permalink / raw)
To: yi.y.sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau
>>> "Jan Beulich" <jbeulich@suse.com> 07/13/17 7:24 AM >>>
>>>> Yi Sun <yi.y.sun@linux.intel.com> 07/13/17 5:04 AM >>>
>>On 17-07-12 13:52:35, Jan Beulich wrote:
>>> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
>>> >This patch implements the CPU init flow for CDP. The flow is almost
>>> >same as L3 CAT.
>>> >
>>> >Note: CDP does NOT work until you apply the later patches of CDP.
>>> >"x86: refactor psr: CDP: implement get hw info flow."
>>> >"x86: refactor psr: CDP: implement set value callback function."
>>>
>>> This is _not_ what I did ask for in what I assume ...
>>>
>>> >v13:
>>> >- add commit message.
>>>
>>> ... this refers to: The problem isn't that it won't work, but that it'll crash the
>>> hypervisor. So what I had expected you to add is a note _after_ the first
>>> --- separator that this patch should not be applied without the other two
>>> ones. Or alternatively for this one to add stubs which the subsequent
>>> patches would then fill.
>>>
>>Oh, ok. I will move the comments under the first '---'. Furthermore, I'd like
>>to add a 'stub_write_msr()' here to avoid crash. Will replace the stub function
>>to real function when implementing the write flow.
>
>Just to avoid any misunderstanding: I hope you don't mean to indeed
>introduce and then replace a function by that name. Just like you've done
>in earlier patches, introduce the function with it final name right away, and
>just fill its body in the later patch.
Oh, and - when you go this route, the must-be-committed-together requirement
would go away altogether, so no need for such a remark anymore.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v13 14/23] x86: refactor psr: CDP: implement get hw info flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (12 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 13/23] x86: refactor psr: CDP: implement CPU init flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
` (8 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements get HW info flow for CDP including L3 CDP callback
function. The flow is almost same as L3 CAT.
With this patch, 'psr-hwinfo' can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v12:
- remove input parameter 'info' in 'psr_cbm_type_to_feat_type'. Use
'feat_props' array to check.
(suggested by Jan Beulich)
v11:
- modify 'psr_get_info' flow to make it simple to cover CDP case.
v10:
- update renamed macros used by psr_get_info.
(suggested by Jan Beulich)
- change 'psr_get_info' flow to cover CDP case to make codes in sysctl
more simple.
(suggested by Jan Beulich)
- remove sysctl redundant codes after applying above changes.
(suggested by Jan Beulich)
v9:
- modify commit message to explain flow more clearly.
- reuse 'cat_get_feat_info' for CDP to reduce redundant codes.
(suggested by Roger Pau)
- fix coding style issues.
(suggested by Wei Liu and Roger Pau)
- rename macros used by psr_get_info to make them meaningful.
(suggested by Jan Beulich)
v5:
- rename 'dat[]' to 'data[]'.
(suggested by Jan Beulich)
- remove type check in callback function.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 13d5c2f..46b8d32 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -223,7 +223,21 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
{
case PSR_CBM_TYPE_L3:
feat_type = FEAT_TYPE_L3_CAT;
+
+ /*
+ * If type is L3 CAT but we cannot find it in feat_props array,
+ * try CDP.
+ */
+ if ( !feat_props[feat_type] )
+ feat_type = FEAT_TYPE_L3_CDP;
+
break;
+
+ case PSR_CBM_TYPE_L3_DATA:
+ case PSR_CBM_TYPE_L3_CODE:
+ feat_type = FEAT_TYPE_L3_CDP;
+ break;
+
default:
ASSERT_UNREACHABLE();
}
@@ -348,11 +362,23 @@ static const struct feat_props l3_cat_props = {
};
/* L3 CDP props */
+static bool l3_cdp_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], uint32_t array_len)
+{
+ if ( !cat_get_feat_info(feat, data, array_len) )
+ return false;
+
+ data[PSR_INFO_IDX_CAT_FLAG] |= XEN_SYSCTL_PSR_CAT_L3_CDP;
+
+ return true;
+}
+
static const struct feat_props l3_cdp_props = {
.cos_num = 2,
.type[0] = PSR_CBM_TYPE_L3_DATA,
.type[1] = PSR_CBM_TYPE_L3_CODE,
.alt_type = FEAT_TYPE_L3_CAT,
+ .get_feat_info = l3_cdp_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (13 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 14/23] x86: refactor psr: CDP: implement get hw info flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-12 20:01 ` Jan Beulich
2017-07-06 1:53 ` [PATCH v13 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
` (7 subsequent siblings)
22 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L3 CDP set value related callback function.
With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v13:
- remove 'strict' from 'psr_cbm_type_to_feat_type' because we use 'alt_type'
to check if we need set both DATA and CODE now.
(suggested by Jan Beulich)
- adjust the implementation in circle in 'insert_val_into_array' to make
codes clearer to show how to decide the return value.
(suggested by Jan Beulich)
v12:
- add comment to explain how to deal with the case that user set new val
for both DATA and CODE at same time.
- add parameter for 'psr_cbm_type_to_feat_type' to return the feature type
according to it.
- use the feature type returned by 'psr_cbm_type_to_feat_type' to check
if we need insert the new value into all items of the feature value array.
- use conditional expression for wrmsrl.
(suggested by Jan Beulich)
v11:
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
- changes about setting both CDP DATA and CODE at same time.
- move 'type[]' declaration into previous patch which introduced 'cos_num'.
(suggested by Jan Beulich)
- changes about 'type[]'.
(suggested by Jan Beulich)
- move 'compare_val' to previous patch.
(suggested by Jan Beulich)
- changes about 'get_val' which has been replace by generic codes.
(suggested by Jan Beulich)
- remove 'restore_default_val' which is unnecessary now.
(suggested by Jan Beulich)
v10:
- remove 'l3_cdp_get_old_val' and use 'l3_cdp_get_val' to replace it.
(suggested by Jan Beulich)
- remove 'l3_cdp_set_new_val'.
- modify 'insert_val_to_array' flow to handle multiple COSs case.
(suggested by Jan Beulich)
- remove 'l3_cdp_compare_val' and implement a generic function
'comapre_val'.
(suggested by Jan Beulich)
- remove 'l3_cdp_fits_cos_max'.
(suggested by Jan Beulich)
- introduce macro 'PSR_MAX_COS_NUM'.
- introduce a new member in 'feat_props', 'type[PSR_MAX_COS_NUM]' to record
all 'cbm_type' the feature has.
(suggested by Jan Beulich)
- modify 'gather_val_array' flow to handle multiple COSs case.
(suggested by Jan Beulich)
- modify 'find_cos' flow and implement 'compare_val' to handle multiple
COSs case.
(suggested by Jan Beulich)
- modify 'fits_cos_max' flow to handle multiple COSs case.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- remove cast in 'l3_cdp_write_msr'.
(suggested by Jan Beulich)
- implement 'compare_val' function to compare if feature values are what
we expect in finding flow.
- implement 'restore_default_val' function to restore feature's COS values
to default if the feature has multiple COSs. It is called when the COS
ID is reduced to 0.
v9:
- add comment to explain why CDP uses 2 COSs.
(suggested by Wei Liu)
- use 'cat_default_val'.
(suggested by Wei Liu)
- remove 'l3_cdp_get_cos_num' because we can directly get cos_num from
feat_node now.
(suggested by Jan Beulich)
- remove cos checking because it has been moved to common function.
(suggested by Jan Beulich)
- l3_cdp_set_new_val parameter 'm' is changed to 'new_val'.
(suggested by Jan Beulich)
- directly use get_cdp_data(feat, 0) and get_cdp_code(feat, 0) to get
default value.
(suggested by Jan Beulich)
- modify 'l3_cdp_write_msr' flow to write value into register according
to input type.
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- modify 'l3_cdp_write_msr' type to 'void'.
v5:
- remove type check in callback function.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- remove 'l3_cdp_get_cos_max_from_type'.
- rename 'l3_cdp_exceeds_cos_max' to 'l3_cdp_fits_cos_max'.
(suggested by Jan Beulich)
v4:
- create this patch to make codes easier to understand.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 27 +++++++++++++++++++++++----
1 file changed, 23 insertions(+), 4 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 46b8d32..e831165 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -373,12 +373,21 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
return true;
}
+static void l3_cdp_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
+{
+ wrmsrl(((type == PSR_CBM_TYPE_L3_DATA) ?
+ MSR_IA32_PSR_L3_MASK_DATA(cos) :
+ MSR_IA32_PSR_L3_MASK_CODE(cos)),
+ val);
+}
+
static const struct feat_props l3_cdp_props = {
.cos_num = 2,
.type[0] = PSR_CBM_TYPE_L3_DATA,
.type[1] = PSR_CBM_TYPE_L3_CODE,
- .alt_type = FEAT_TYPE_L3_CAT,
+ .alt_type = PSR_CBM_TYPE_L3,
.get_feat_info = l3_cdp_get_feat_info,
+ .write_msr = l3_cdp_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -824,17 +833,27 @@ static int insert_val_into_array(uint32_t val[],
if ( !psr_check_cbm(feat->cbm_len, new_val) )
return -EINVAL;
- /* Value setting position is same as feature array. */
+ /*
+ * Value setting position is same as feature array.
+ * For CDP, user may set both DATA and CODE to same value. For such case,
+ * user input 'PSR_CBM_TYPE_L3' as type. The alternative type of CDP is same
+ * as it. So we should set new_val to both of DATA and CODE under such case.
+ */
for ( i = 0; i < props->cos_num; i++ )
{
if ( type == props->type[i] )
{
val[i] = new_val;
- return 0;
+ ret = 0;
+ break;
}
+ else if ( type == props->alt_type )
+ val[i] = new_val;
+ else
+ ret = -EINVAL;
}
- return -EINVAL;
+ return ret;
}
static int compare_val(const uint32_t val[],
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v13 15/23] x86: refactor psr: CDP: implement set value callback function.
2017-07-06 1:53 ` [PATCH v13 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
@ 2017-07-12 20:01 ` Jan Beulich
0 siblings, 0 replies; 42+ messages in thread
From: Jan Beulich @ 2017-07-12 20:01 UTC (permalink / raw)
To: yi.y.sun, xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
>--- a/xen/arch/x86/psr.c
>+++ b/xen/arch/x86/psr.c
>@@ -373,12 +373,21 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
>return true;
>}
>
>+static void l3_cdp_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
>+{
>+ wrmsrl(((type == PSR_CBM_TYPE_L3_DATA) ?
>+ MSR_IA32_PSR_L3_MASK_DATA(cos) :
>+ MSR_IA32_PSR_L3_MASK_CODE(cos)),
>+ val);
>+}
>+
>static const struct feat_props l3_cdp_props = {
>.cos_num = 2,
>.type[0] = PSR_CBM_TYPE_L3_DATA,
>.type[1] = PSR_CBM_TYPE_L3_CODE,
>- .alt_type = FEAT_TYPE_L3_CAT,
>+ .alt_type = PSR_CBM_TYPE_L3,
I did wonder about the value set in the earlier patch, but if you change it here
this is a pretty clear sign that you really should set it to the intended value
right away. With this taken care of (or a good reason provided why it needs
to be the way it is)
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v13 16/23] x86: L2 CAT: implement CPU init flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (14 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 15/23] x86: refactor psr: CDP: implement set value callback function Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-12 20:09 ` Jan Beulich
2017-07-06 1:53 ` [PATCH v13 17/23] x86: L2 CAT: implement get hw info flow Yi Sun
` (6 subsequent siblings)
22 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the CPU init flow for L2 CAT.
Note: L2 CAT does NOT work until you apply the later patches of L2 CAT.
"x86: L2 CAT: implement get hw info flow."
"x86: L2 CAT: implement get value flow."
"x86: L2 CAT: implement set value flow."
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v13:
- add commit message.
(suggested by Jan Beulich)
- set 'alt_type' for L2 CAT.
(suggested by Jan Beulich)
- define a static string array to show which feature's info is printing.
(suggested by Jan Beulich)
v12:
- move 'type[]' assignment into l2_cat_props declaration to make it be
'const'.
(suggested by Jan Beulich)
- add "L2 CAT" indicator in printk.
(suggested by Jan Beulich)
- restore mask(0) MSR to default value.
(suggested by Jan Beulich)
v11:
- move l2 cat 'type[]' assignement into 'psr_cpu_init'.
- remove COS MSR restore action in 'cpu_init_feature'.
- set 'feat_init' to true after CPU init.
- modify commit message.
v10:
- implement L2 CAT case in 'cat_init_feature'.
(suggested by Jan Beulich)
- changes about 'props'.
(suggested by Jan Beulich)
- introduce 'PSR_CBM_TYPE_L2'.
v9:
- modify error handling process in 'psr_cpu_prepare' to reduce redundant
codes.
- reuse 'cat_init_feature' and 'cat_get_cos_max' for L2 CAT to reduce
redundant codes.
(suggested by Roger Pau)
- remove unnecessary comment.
(suggested by Jan Beulich)
- move L2 CAT related codes from 'cpu_init_work' into 'psr_cpu_init'.
(suggested by Jan Beulich)
- do not free resource when allocation fails in 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v7:
- initialize 'l2_cat'.
(suggested by Konrad Rzeszutek Wilk)
v6:
- use 'struct cpuid_leaf'.
(suggested by Konrad Rzeszutek Wilk and Jan Beulich)
v5:
- remove 'feat_l2_cat' free in 'free_feature'.
(suggested by Jan Beulich)
- encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
(suggested by Jan Beulich)
- print socket info when 'opt_cpu_info' is true.
(suggested by Jan Beulich)
- rename 'l2_cat_get_max_cos_max' to 'l2_cat_get_cos_max'.
(suggested by Jan Beulich)
- rename 'dat[]' to 'data[]'
(suggested by Jan Beulich)
- move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 40 +++++++++++++++++++++++++++++++++++++---
xen/include/asm-x86/msr-index.h | 1 +
xen/include/asm-x86/psr.h | 2 ++
3 files changed, 40 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index e831165..81c9454 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -63,6 +63,7 @@
enum psr_feat_type {
FEAT_TYPE_L3_CAT,
FEAT_TYPE_L3_CDP,
+ FEAT_TYPE_L2_CAT,
FEAT_TYPE_NUM,
FEAT_TYPE_UNKNOWN,
};
@@ -160,6 +161,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
*/
static struct feat_node *feat_l3_cat;
static struct feat_node *feat_l3_cdp;
+static struct feat_node *feat_l2_cat;
/* Common functions */
#define cat_default_val(len) (0xffffffff >> (32 - (len)))
@@ -269,6 +271,12 @@ static bool psr_check_cbm(unsigned int cbm_len, unsigned long cbm)
}
/* CAT common functions implementation. */
+static char *feat_name[FEAT_TYPE_NUM] = {
+ "L3 CAT",
+ "CDP",
+ "L2 CAT",
+};
+
static int cat_init_feature(const struct cpuid_leaf *regs,
struct feat_node *feat,
struct psr_socket_info *info,
@@ -284,13 +292,17 @@ static int cat_init_feature(const struct cpuid_leaf *regs,
switch ( type )
{
case FEAT_TYPE_L3_CAT:
+ case FEAT_TYPE_L2_CAT:
if ( feat->cos_max < 1 )
return -ENOENT;
/* We reserve cos=0 as default cbm (all bits within cbm_len are 1). */
feat->cos_reg_val[0] = cat_default_val(feat->cbm_len);
- wrmsrl(MSR_IA32_PSR_L3_MASK(0), cat_default_val(feat->cbm_len));
+ wrmsrl((type == FEAT_TYPE_L3_CAT ?
+ MSR_IA32_PSR_L3_MASK(0) :
+ MSR_IA32_PSR_L2_MASK(0)),
+ cat_default_val(feat->cbm_len));
break;
@@ -328,8 +340,8 @@ static int cat_init_feature(const struct cpuid_leaf *regs,
return 0;
printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
- ((type == FEAT_TYPE_L3_CDP) ? "CDP" : "L3 CAT"),
- cpu_to_socket(smp_processor_id()), feat->cos_max, feat->cbm_len);
+ feat_name[type], cpu_to_socket(smp_processor_id()),
+ feat->cos_max, feat->cbm_len);
return 0;
}
@@ -390,6 +402,13 @@ static const struct feat_props l3_cdp_props = {
.write_msr = l3_cdp_write_msr,
};
+/* L2 CAT props */
+static const struct feat_props l2_cat_props = {
+ .cos_num = 1,
+ .type[0] = PSR_CBM_TYPE_L2,
+ .alt_type = PSR_CBM_TYPE_UNKNOWN,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1345,6 +1364,10 @@ static int psr_cpu_prepare(void)
(feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
return -ENOMEM;
+ if ( feat_l2_cat == NULL &&
+ (feat_l2_cat = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
@@ -1392,6 +1415,17 @@ static void psr_cpu_init(void)
}
}
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L2 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, ®s);
+
+ feat = feat_l2_cat;
+ feat_l2_cat = NULL;
+ if ( !cat_init_feature(®s, feat, info, FEAT_TYPE_L2_CAT) )
+ feat_props[FEAT_TYPE_L2_CAT] = &l2_cat_props;
+ }
+
info->feat_init = true;
assoc_init:
diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h
index 771e750..6c49c6d 100644
--- a/xen/include/asm-x86/msr-index.h
+++ b/xen/include/asm-x86/msr-index.h
@@ -345,6 +345,7 @@
#define MSR_IA32_PSR_L3_MASK(n) (0x00000c90 + (n))
#define MSR_IA32_PSR_L3_MASK_CODE(n) (0x00000c90 + (n) * 2 + 1)
#define MSR_IA32_PSR_L3_MASK_DATA(n) (0x00000c90 + (n) * 2)
+#define MSR_IA32_PSR_L2_MASK(n) (0x00000d10 + (n))
/* Intel Model 6 */
#define MSR_P6_PERFCTR(n) (0x000000c1 + (n))
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 50b8757..18a42f3 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -23,6 +23,7 @@
/* Resource Type Enumeration */
#define PSR_RESOURCE_TYPE_L3 0x2
+#define PSR_RESOURCE_TYPE_L2 0x4
/* L3 Monitoring Features */
#define PSR_CMT_L3_OCCUPANCY 0x1
@@ -56,6 +57,7 @@ enum cbm_type {
PSR_CBM_TYPE_L3,
PSR_CBM_TYPE_L3_CODE,
PSR_CBM_TYPE_L3_DATA,
+ PSR_CBM_TYPE_L2,
PSR_CBM_TYPE_UNKNOWN,
};
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v13 16/23] x86: L2 CAT: implement CPU init flow.
2017-07-06 1:53 ` [PATCH v13 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
@ 2017-07-12 20:09 ` Jan Beulich
2017-07-13 3:03 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2017-07-12 20:09 UTC (permalink / raw)
To: yi.y.sun, xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, roger.pau
>>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
>This patch implements the CPU init flow for L2 CAT.
>
>Note: L2 CAT does NOT work until you apply the later patches of L2 CAT.
>"x86: L2 CAT: implement get hw info flow."
>"x86: L2 CAT: implement get value flow."
>"x86: L2 CAT: implement set value flow."
Same comment as on the respective CDP one.
>@@ -269,6 +271,12 @@ static bool psr_check_cbm(unsigned int cbm_len, unsigned long cbm)
>}
>
>/* CAT common functions implementation. */
>+static char *feat_name[FEAT_TYPE_NUM] = {
const char * const
Additionally - do you need or plan to use this in more than one function? If
not, it should be made local to its only user. If so, the variable name should
include "cat", as these appear to be CAT-specific feature names only.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v13 16/23] x86: L2 CAT: implement CPU init flow.
2017-07-12 20:09 ` Jan Beulich
@ 2017-07-13 3:03 ` Yi Sun
0 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-13 3:03 UTC (permalink / raw)
To: Jan Beulich
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau
On 17-07-12 14:09:40, Jan Beulich wrote:
> >>> Yi Sun <yi.y.sun@linux.intel.com> 07/06/17 4:07 AM >>>
> >This patch implements the CPU init flow for L2 CAT.
> >
> >Note: L2 CAT does NOT work until you apply the later patches of L2 CAT.
> >"x86: L2 CAT: implement get hw info flow."
> >"x86: L2 CAT: implement get value flow."
> >"x86: L2 CAT: implement set value flow."
>
> Same comment as on the respective CDP one.
>
Will change it.
> >@@ -269,6 +271,12 @@ static bool psr_check_cbm(unsigned int cbm_len, unsigned long cbm)
> >}
> >
> >/* CAT common functions implementation. */
> >+static char *feat_name[FEAT_TYPE_NUM] = {
>
> const char * const
>
> Additionally - do you need or plan to use this in more than one function? If
> not, it should be made local to its only user. If so, the variable name should
> include "cat", as these appear to be CAT-specific feature names only.
>
This should be moved into cat_init_feature() as a local array. Thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v13 17/23] x86: L2 CAT: implement get hw info flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (15 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 16/23] x86: L2 CAT: implement CPU init flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 18/23] x86: L2 CAT: implement get value flow Yi Sun
` (5 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L2 CAT get HW info flow and interface in sysctl.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v10:
- modify macro name according to previous patch change.
(suggested by Jan Beulich)
- modify commit message.
v9:
- reuse 'cat_get_feat_info' for L2 CAT to reduce redundant codes.
(suggested by Roger Pau)
- modify sysctl implementation of L2 CAT to input data[3] to use
'cat_get_feat_info'.
(suggested by Roger Pau)
- modify macros names to newly defined ones.
(suggested by Jan Beulich)
- remove 'l2_info' to reuse 'l3_info'.
(suggested by Jan Beulich)
- modify macro name according to previous patch change.
(suggested by Jan Beulich)
v5:
- rename 'dat[]' to 'data[]'
(suggested by Jan Beulich)
- remove type check in callback function.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
xen/arch/x86/psr.c | 5 +++++
xen/arch/x86/sysctl.c | 23 +++++++++++++++++++++--
xen/include/public/sysctl.h | 1 +
3 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 81c9454..610a55c 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -240,6 +240,10 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
feat_type = FEAT_TYPE_L3_CDP;
break;
+ case PSR_CBM_TYPE_L2:
+ feat_type = FEAT_TYPE_L2_CAT;
+ break;
+
default:
ASSERT_UNREACHABLE();
}
@@ -407,6 +411,7 @@ static const struct feat_props l2_cat_props = {
.cos_num = 1,
.type[0] = PSR_CBM_TYPE_L2,
.alt_type = PSR_CBM_TYPE_UNKNOWN,
+ .get_feat_info = cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index c23270d..5e91755 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -174,10 +174,10 @@ long arch_do_sysctl(
case XEN_SYSCTL_psr_cat_op:
switch ( sysctl->u.psr_cat_op.cmd )
{
- case XEN_SYSCTL_PSR_CAT_get_l3_info:
- {
uint32_t data[PSR_INFO_ARRAY_SIZE];
+ case XEN_SYSCTL_PSR_CAT_get_l3_info:
+ {
ret = psr_get_info(sysctl->u.psr_cat_op.target,
PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
if ( ret )
@@ -195,6 +195,25 @@ long arch_do_sysctl(
break;
}
+ case XEN_SYSCTL_PSR_CAT_get_l2_info:
+ {
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L2, data, ARRAY_SIZE(data));
+ if ( ret )
+ break;
+
+ sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ data[PSR_INFO_IDX_COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ data[PSR_INFO_IDX_CAT_CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.flags =
+ data[PSR_INFO_IDX_CAT_FLAG];
+
+ if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
+ ret = -EFAULT;
+ break;
+ }
+
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index ee76a66..4c76d3a 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
#define XEN_SYSCTL_PSR_CAT_get_l3_info 0
+#define XEN_SYSCTL_PSR_CAT_get_l2_info 1
struct xen_sysctl_psr_cat_op {
uint32_t cmd; /* IN: XEN_SYSCTL_PSR_CAT_* */
uint32_t target; /* IN */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 18/23] x86: L2 CAT: implement get value flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (16 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 17/23] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 19/23] x86: L2 CAT: implement set " Yi Sun
` (4 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L2 CAT get value interface in domctl.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v11:
- remove "get_val' assignment because it has been replaced by generic
codes.
(suggested by Jan Beulich)
v10:
- remove cast in domctl.
(suggested by Jan Beulich)
v9:
- reuse 'cat_get_val' for L2 CAT to reduce redundant codes
(suggested by Roger Pau)
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v5:
- remove type check in callback function.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
xen/arch/x86/domctl.c | 7 +++++++
xen/include/public/domctl.h | 1 +
2 files changed, 8 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index cea643c..d637bca 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1460,6 +1460,13 @@ long arch_do_domctl(
copyback = 1;
break;
+ case XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM:
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &val32, PSR_CBM_TYPE_L2);
+ domctl->u.psr_cat_op.data = val32;
+ copyback = 1;
+ break;
+
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index f7cbc0a..61a9450 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1134,6 +1134,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
uint64_t data; /* IN/OUT */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 19/23] x86: L2 CAT: implement set value flow.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (17 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 18/23] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 20/23] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
` (3 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements L2 CAT set value related callback function
and domctl interface.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
v11:
- remove 'domctl->u.psr_cat_op.data' check because it has been moved into
'psr_set_val'.
(suggested by Jan Beulich)
- move 'feat->cos_reg_val' assignment and value comparison in 'write_msr'
callback function out as generic codes.
(suggested by Jan Beulich)
v10:
- check input data and remove cast in domctl.
(suggested by Jan Beulich)
- remove some hooks assignment due to previous patches changes.
(suggested by Jan Beulich)
- remove cast in 'l2_cat_write_msr'.
(suggested by Jan Beulich)
- remove 'return in 'l2_cat_write_msr'.
(suggested by Jan Beulich)
v9:
- reuse some CAT common functions for L2 CAT to reduce redundant codes.
(suggested by Roger Pau)
- remove parameter 'found' from 'cat_compare_val' and modify the return
values to let caller know if the id is found or not. These things are
done in patch "x86: refactor psr: set value: implement cos finding flow."
(suggested by Roger Pau and Dario Faggioli)
- remove 'get_cos_num' related codes.
(suggested by Jan Beulich)
- modify 'l2_cat_write_msr' according to previous patch change.
- changes about 'uint64_t' to 'uint32_t'.
(suggested by Jan Beulich)
v8:
- modify 'l2_cat_write_msr' to 'void'.
v5:
- remove type check in callback function.
(suggested by Jan Beulich)
- modify return value of callback functions because we do not need them
to return number of entries the feature uses. In caller, we call
'get_cos_num' to get the number of entries the feature uses.
(suggested by Jan Beulich)
- remove 'l2_cat_get_cos_max_from_type'.
(suggested by Jan Beulich)
- rename 'l2_cat_exceeds_cos_max' to 'l2_cat_fits_cos_max'.
(suggested by Jan Beulich)
v4:
- create this patch because of codes architecture change.
(suggested by Jan Beulich)
---
xen/arch/x86/domctl.c | 6 ++++++
xen/arch/x86/psr.c | 6 ++++++
xen/include/public/domctl.h | 1 +
3 files changed, 13 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index d637bca..a0892d6 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1439,6 +1439,12 @@ long arch_do_domctl(
PSR_CBM_TYPE_L3_DATA);
break;
+ case XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM:
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L2);
+ break;
+
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
ret = psr_get_val(d, domctl->u.psr_cat_op.target,
&val32, PSR_CBM_TYPE_L3);
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 610a55c..06f60f1 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -407,11 +407,17 @@ static const struct feat_props l3_cdp_props = {
};
/* L2 CAT props */
+static void l2_cat_write_msr(unsigned int cos, uint32_t val, enum cbm_type type)
+{
+ wrmsrl(MSR_IA32_PSR_L2_MASK(cos), val);
+}
+
static const struct feat_props l2_cat_props = {
.cos_num = 1,
.type[0] = PSR_CBM_TYPE_L2,
.alt_type = PSR_CBM_TYPE_UNKNOWN,
.get_feat_info = cat_get_feat_info,
+ .write_msr = l2_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 61a9450..5197c34 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1134,6 +1134,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM 6
#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 20/23] tools: L2 CAT: support get HW info for L2 CAT.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (18 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 19/23] x86: L2 CAT: implement set " Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 21/23] tools: L2 CAT: support show cbm " Yi Sun
` (2 subsequent siblings)
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, He Chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements xl/xc changes to support get HW info
for L2 CAT.
'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
info.
Example(on machine which only supports L2 CAT):
Cache Monitoring Technology (CMT):
Enabled : 0
Cache Allocation Technology (CAT): L2
Socket ID : 0
Maximum COS : 3
CBM length : 8
Default CBM : 0xff
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v12:
- acked by Wei for tools side changes.
v11:
- acked by Jan for Hypervisor side changes.
v10:
- change macros names according to previous changes.
(suggested by Jan Beulich)
v9:
- add some cases to handle return error no.
- move xl_cmdimpl.c codes into xl/xl_psr.c.
- change 'l3_info' to 'cat_info' to cover both L3 and L2 CAT.
v6:
- adjust '{' position for 'switch'.
(suggested by Wei Liu)
- modify commit message to remove error log.
(suggested by Dario Faggioli)
v5:
- modify commit message to remove error log.
(suggested by Wei Liu and Jan Beulich)
- replace unnecessary 'return' to 'break'.
(suggested by Wei Liu)
- restore 'libxl_psr_cat_get_l3_info' to keep interface backward compatible
but change codes in it to call new function to get hw info.
(suggested by Wei Liu)
- add 'L2_CBM' into 'psr_cbm_type' because it is interface change which
should be in same patch with new 'LIBXL_HAVE_' macro.
(suggested by Wei Liu)
- addjust logs sentence to make unnecessary error logs not show.
(suggested by Wei Liu and Jan Beulich)
v4:
- create this patch to help reviewers better understand the codes.
---
tools/libxc/include/xenctrl.h | 6 ++---
tools/libxc/xc_psr.c | 39 +++++++++++++++++++++++---------
tools/libxl/libxl.h | 9 ++++++++
tools/libxl/libxl_psr.c | 28 ++++++++++++++++++-----
tools/libxl/libxl_types.idl | 1 +
tools/xl/xl_psr.c | 52 +++++++++++++++++++++++++++++++++----------
xen/arch/x86/sysctl.c | 12 +++++-----
xen/include/public/sysctl.h | 2 +-
8 files changed, 111 insertions(+), 38 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 1629f41..16ed45b 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2470,9 +2470,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
xc_psr_cat_type type, uint32_t target,
uint64_t *data);
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled);
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len,
+ bool *cdp_enabled);
int xc_get_cpu_levelling_caps(xc_interface *xch, uint32_t *caps);
int xc_get_cpu_featureset(xc_interface *xch, uint32_t index,
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 43b3286..84a08c4 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -317,24 +317,41 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
return rc;
}
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled)
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len, bool *cdp_enabled)
{
- int rc;
+ int rc = -1;
DECLARE_SYSCTL;
sysctl.cmd = XEN_SYSCTL_psr_cat_op;
- sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
sysctl.u.psr_cat_op.target = socket;
- rc = xc_sysctl(xch, &sysctl);
- if ( !rc )
+ switch ( lvl )
{
- *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
- *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
- *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
- XEN_SYSCTL_PSR_CAT_L3_CDP;
+ case 2:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l2_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.cat_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.cat_info.cbm_len;
+ *cdp_enabled = false;
+ }
+ break;
+ case 3:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.cat_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.cat_info.cbm_len;
+ *cdp_enabled = sysctl.u.psr_cat_op.u.cat_info.flags &
+ XEN_SYSCTL_PSR_CAT_L3_CDP;
+ }
+ break;
+ default:
+ errno = EOPNOTSUPP;
+ break;
}
return rc;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index cf8687a..504d359 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -922,6 +922,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, const libxl_mac *src);
* If this is defined, the Code and Data Prioritization feature is supported.
*/
#define LIBXL_HAVE_PSR_CDP 1
+
+/*
+ * LIBXL_HAVE_PSR_L2_CAT
+ *
+ * If this is defined, the L2 Cache Allocation Technology feature is supported.
+ */
+#define LIBXL_HAVE_PSR_L2_CAT 1
#endif
/*
@@ -2200,6 +2207,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
* On success, the function returns an array of elements in 'info',
* and the length in 'nr'.
*/
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl);
int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
int *nr);
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr);
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index ec5c79d..f55ba1e 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -91,6 +91,15 @@ static void libxl__psr_cat_log_err_msg(libxl__gc *gc, int err)
case ENXIO:
msg = "Unable to set code or data CBM when CDP is disabled";
break;
+ case EINVAL:
+ msg = "Invalid input or some internal values are not expected";
+ break;
+ case ERANGE:
+ msg = "Socket number is wrong";
+ break;
+ case ENOSPC:
+ msg = "Value array exceeds the range";
+ break;
default:
libxl__psr_log_err_msg(gc, err);
@@ -352,8 +361,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
return rc;
}
-int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
- int *nr)
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl)
{
GC_INIT(ctx);
int rc;
@@ -380,9 +389,8 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
libxl_for_each_set_bit(socketid, socketmap) {
ptr[i].id = socketid;
- if (xc_psr_cat_get_l3_info(ctx->xch, socketid, &ptr[i].cos_max,
- &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
- libxl__psr_cat_log_err_msg(gc, errno);
+ if (xc_psr_cat_get_info(ctx->xch, socketid, lvl, &ptr[i].cos_max,
+ &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
rc = ERROR_FAIL;
free(ptr);
goto out;
@@ -398,6 +406,16 @@ out:
return rc;
}
+int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr)
+{
+ int rc;
+
+ rc = libxl_psr_cat_get_info(ctx, info, nr, 3);
+
+ return rc;
+}
+
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr)
{
int i;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 2204425..68339a1 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -975,6 +975,7 @@ libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
(1, "L3_CBM"),
(2, "L3_CBM_CODE"),
(3, "L3_CBM_DATA"),
+ (4, "L2_CBM"),
])
libxl_psr_cat_info = Struct("psr_cat_info", [
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index c061b29..271b88f 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -294,21 +294,19 @@ int main_psr_cmt_show(int argc, char **argv)
}
#endif
-#ifdef LIBXL_HAVE_PSR_CAT
-static int psr_cat_hwinfo(void)
+#if defined(LIBXL_HAVE_PSR_CAT) || defined(LIBXL_HAVE_PSR_L2_CAT)
+static int psr_l3_cat_hwinfo(void)
{
- int rc;
- int i, nr;
+ int rc, nr;
+ unsigned int i;
uint32_t l3_cache_size;
libxl_psr_cat_info *info;
- printf("Cache Allocation Technology (CAT):\n");
-
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
- if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (rc)
return rc;
- }
+
+ printf("Cache Allocation Technology (CAT):\n");
for (i = 0; i < nr; i++) {
rc = libxl_psr_cmt_get_l3_cache_size(ctx, info[i].id, &l3_cache_size);
@@ -417,7 +415,7 @@ static int psr_cat_show(uint32_t domid)
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
if (rc) {
fprintf(stderr, "Failed to get cat info\n");
return rc;
@@ -434,6 +432,32 @@ out:
return rc;
}
+static int psr_l2_cat_hwinfo(void)
+{
+ int rc;
+ unsigned int i;
+ int nr;
+ libxl_psr_cat_info *info;
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 2);
+ if (rc)
+ return rc;
+
+ printf("Cache Allocation Technology (CAT): L2\n");
+
+ for (i = 0; i < nr; i++) {
+ /* There is no CMT on L2 cache so far. */
+ printf("%-16s: %u\n", "Socket ID", info[i].id);
+ printf("%-16s: %u\n", "Maximum COS", info[i].cos_max);
+ printf("%-16s: %u\n", "CBM length", info[i].cbm_len);
+ printf("%-16s: %#llx\n", "Default CBM",
+ (1ull << info[i].cbm_len) - 1);
+ }
+
+ libxl_psr_cat_info_list_free(info, nr);
+ return rc;
+}
+
int main_psr_cat_cbm_set(int argc, char **argv)
{
uint32_t domid;
@@ -551,7 +575,11 @@ int main_psr_hwinfo(int argc, char **argv)
ret = psr_cmt_hwinfo();
if (!ret && (all || cat))
- ret = psr_cat_hwinfo();
+ ret = psr_l3_cat_hwinfo();
+
+ /* L2 CAT is independent of CMT and L3 CAT */
+ if (all || cat)
+ ret = psr_l2_cat_hwinfo();
return ret;
}
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 5e91755..14c12b7 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -183,11 +183,11 @@ long arch_do_sysctl(
if ( ret )
break;
- sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ sysctl->u.psr_cat_op.u.cat_info.cos_max =
data[PSR_INFO_IDX_COS_MAX];
- sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ sysctl->u.psr_cat_op.u.cat_info.cbm_len =
data[PSR_INFO_IDX_CAT_CBM_LEN];
- sysctl->u.psr_cat_op.u.l3_info.flags =
+ sysctl->u.psr_cat_op.u.cat_info.flags =
data[PSR_INFO_IDX_CAT_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
@@ -202,11 +202,11 @@ long arch_do_sysctl(
if ( ret )
break;
- sysctl->u.psr_cat_op.u.l3_info.cos_max =
+ sysctl->u.psr_cat_op.u.cat_info.cos_max =
data[PSR_INFO_IDX_COS_MAX];
- sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+ sysctl->u.psr_cat_op.u.cat_info.cbm_len =
data[PSR_INFO_IDX_CAT_CBM_LEN];
- sysctl->u.psr_cat_op.u.l3_info.flags =
+ sysctl->u.psr_cat_op.u.cat_info.flags =
data[PSR_INFO_IDX_CAT_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 4c76d3a..9e51af6 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -754,7 +754,7 @@ struct xen_sysctl_psr_cat_op {
uint32_t cos_max; /* OUT: Maximum COS */
#define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
uint32_t flags; /* OUT: CAT flags */
- } l3_info;
+ } cat_info;
} u;
};
typedef struct xen_sysctl_psr_cat_op xen_sysctl_psr_cat_op_t;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 21/23] tools: L2 CAT: support show cbm for L2 CAT.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (19 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 20/23] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 22/23] tools: L2 CAT: support set " Yi Sun
2017-07-06 1:53 ` [PATCH v13 23/23] docs: add L2 CAT description in docs Yi Sun
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, He Chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements changes in xl/xc changes to support
showing CBM of L2 CAT.
The new level option is introduced to original CAT showing
command in order to show CBM for specified level CAT.
- 'xl psr-cat-show' is updated to show CBM of a domain
according to input cache level.
Examples:
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v9:
- move xl_cmdimpl.c changes into xl/xl_psr.c.
- move xl_cmdtable.c changes into xl/xl_cmdtable.c.
v6:
- check if input level is correct.
- adjust '{' postion for 'if'.
(suggested by Wei Liu)
v5:
- remove 'L2_CBM' in idl because it has been moved to patch 21:
"tools: L2 CAT: support get HW info for L2 CAT".
(suggested by Wei Liu)
v4:
- create this patch because of codes architecture change.
---
tools/libxc/include/xenctrl.h | 1 +
tools/libxc/xc_psr.c | 3 ++
tools/xl/xl_cmdtable.c | 3 +-
tools/xl/xl_psr.c | 85 +++++++++++++++++++++++++++++--------------
4 files changed, 63 insertions(+), 29 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 16ed45b..552f63b 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2446,6 +2446,7 @@ enum xc_psr_cat_type {
XC_PSR_CAT_L3_CBM = 1,
XC_PSR_CAT_L3_CBM_CODE = 2,
XC_PSR_CAT_L3_CBM_DATA = 3,
+ XC_PSR_CAT_L2_CBM = 4,
};
typedef enum xc_psr_cat_type xc_psr_cat_type;
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 84a08c4..04f5927 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -299,6 +299,9 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 30eb93c..539cb64 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -557,7 +557,8 @@ struct cmd_spec cmd_table[] = {
{ "psr-cat-show",
&main_psr_cat_show, 0, 1,
"Show Cache Allocation Technology information",
- "<Domain>",
+ "[options] <Domain>",
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
},
#endif
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index 271b88f..575f4a0 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -342,7 +342,7 @@ static void psr_cat_print_one_domain_cbm_type(uint32_t domid, uint32_t socketid,
}
static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
char *domain_name;
@@ -350,27 +350,38 @@ static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
printf("%5d%25s", domid, domain_name);
free(domain_name);
- if (!cdp_enabled) {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM);
- } else {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ switch (lvl) {
+ case 3:
+ if (!cdp_enabled) {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM);
+ } else {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ }
+ break;
+ case 2:
psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ LIBXL_PSR_CBM_TYPE_L2_CBM);
+ break;
+ default:
+ printf("Input lvl %d is wrong!", lvl);
+ break;
}
printf("\n");
}
static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
int i, nr_domains;
libxl_dominfo *list;
if (domid != INVALID_DOMID) {
- psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled, lvl);
return 0;
}
@@ -380,49 +391,59 @@ static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
}
for (i = 0; i < nr_domains; i++)
- psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled, lvl);
libxl_dominfo_list_free(list, nr_domains);
return 0;
}
-static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
+static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info,
+ unsigned int lvl)
{
int rc;
uint32_t l3_cache_size;
- rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
- if (rc) {
- fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
- info->id);
- return -1;
+ printf("%-16s: %u\n", "Socket ID", info->id);
+
+ /* So far, CMT only supports L3 cache. */
+ if (lvl == 3) {
+ rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
+ if (rc) {
+ fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
+ info->id);
+ return -1;
+ }
+ printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
}
- printf("%-16s: %u\n", "Socket ID", info->id);
- printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
printf("%-16s: %#llx\n", "Default CBM", (1ull << info->cbm_len) - 1);
if (info->cdp_enabled)
printf("%5s%25s%16s%16s\n", "ID", "NAME", "CBM (code)", "CBM (data)");
else
printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
- return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled);
+ return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled, lvl);
}
-static int psr_cat_show(uint32_t domid)
+static int psr_cat_show(uint32_t domid, unsigned int lvl)
{
int i, nr;
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (lvl != 2 && lvl != 3) {
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
+ }
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, lvl);
if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ fprintf(stderr, "Failed to get %s cat info\n", (lvl == 3)?"L3":"L2");
return rc;
}
for (i = 0; i < nr; i++) {
- rc = psr_cat_print_socket(domid, info + i);
+ rc = psr_cat_print_socket(domid, info + i, lvl);
if (rc)
goto out;
}
@@ -533,11 +554,19 @@ int main_psr_cat_cbm_set(int argc, char **argv)
int main_psr_cat_show(int argc, char **argv)
{
- int opt;
+ int opt = 0;
uint32_t domid;
+ unsigned int lvl = 3;
- SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cat-show", 0) {
- /* No options */
+ static struct option opts[] = {
+ {"level", 1, 0, 'l'},
+ COMMON_LONG_OPTS
+ };
+
+ SWITCH_FOREACH_OPT(opt, "l:", opts, "psr-cat-show", 0) {
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
if (optind >= argc)
@@ -549,7 +578,7 @@ int main_psr_cat_show(int argc, char **argv)
return 2;
}
- return psr_cat_show(domid);
+ return psr_cat_show(domid, lvl);
}
int main_psr_hwinfo(int argc, char **argv)
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 22/23] tools: L2 CAT: support set cbm for L2 CAT.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (20 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 21/23] tools: L2 CAT: support show cbm " Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
2017-07-06 1:53 ` [PATCH v13 23/23] docs: add L2 CAT description in docs Yi Sun
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, He Chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch implements the xl/xc changes to support set CBM
for L2 CAT.
The new level option is introduced to original CAT setting
command in order to set CBM for specified level CAT.
- 'xl psr-cat-set' is updated to set cache capacity bitmasks(CBM)
for a domain according to input cache level.
root@:~$ xl psr-cat-set -l2 1 0x7f
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v11:
- revert change in v9 to handle the case to set both CODE and DATA for CDP
at same time. This must be handled in hypervisor.
v10:
- fix comments.
(suggested by Wei Liu)
v9:
- handle the case to set both CODE and DATA for CDP at same time.
For such case, user does not input '-c' or '-d' to set CDP cbm.
- move xl_cmdimpl.c changes into xl/xl_psr.c.
- move xl_cmdtable.c changes into xl/xl_cmdtable.c.
v6:
- rename 'psr-cat-cbm-set' to 'psr-cat-set'.
(suggested by Kevin Tian)
- return 'EXIT_FAILURE' for error case.
(suggested by Dario Faggioli)
- print error info when input level is wrong.
v4:
- create this patch because of codes architecture change.
---
tools/libxc/xc_psr.c | 3 +++
tools/xl/xl_cmdtable.c | 3 ++-
tools/xl/xl_psr.c | 33 +++++++++++++++++++++++----------
3 files changed, 28 insertions(+), 11 deletions(-)
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 04f5927..039b920 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -266,6 +266,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 539cb64..2c71a9f 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -546,11 +546,12 @@ struct cmd_spec cmd_table[] = {
},
#endif
#ifdef LIBXL_HAVE_PSR_CAT
- { "psr-cat-cbm-set",
+ { "psr-cat-set",
&main_psr_cat_cbm_set, 0, 1,
"Set cache capacity bitmasks(CBM) for a domain",
"[options] <Domain> <CBM>",
"-s <socket> Specify the socket to process, otherwise all sockets are processed\n"
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
"-c Set code CBM if CDP is supported\n"
"-d Set data CBM if CDP is supported\n"
},
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index 575f4a0..7309d4f 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -490,19 +490,21 @@ int main_psr_cat_cbm_set(int argc, char **argv)
char *value;
libxl_string_list socket_list;
unsigned long start, end;
- int i, j, len;
+ unsigned int i, j, len;
+ unsigned int lvl = 3;
static struct option opts[] = {
{"socket", 1, 0, 's'},
{"data", 0, 0, 'd'},
{"code", 0, 0, 'c'},
+ {"level", 1, 0, 'l'},
COMMON_LONG_OPTS
};
libxl_socket_bitmap_alloc(ctx, &target_map, 0);
libxl_bitmap_set_none(&target_map);
- SWITCH_FOREACH_OPT(opt, "s:cd", opts, "psr-cat-cbm-set", 2) {
+ SWITCH_FOREACH_OPT(opt, "s:l:cd", opts, "psr-cat-set", 2) {
case 's':
trim(isspace, optarg, &value);
split_string_into_string_list(value, ",", &socket_list);
@@ -522,24 +524,35 @@ int main_psr_cat_cbm_set(int argc, char **argv)
case 'c':
opt_code = 1;
break;
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
- if (opt_data && opt_code) {
- fprintf(stderr, "Cannot handle -c and -d at the same time\n");
- return -1;
- } else if (opt_data) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
- } else if (opt_code) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ if (lvl == 2)
+ type = LIBXL_PSR_CBM_TYPE_L2_CBM;
+ else if (lvl == 3) {
+ if (opt_data && opt_code) {
+ fprintf(stderr, "Cannot handle -c and -d at the same time\n");
+ return EXIT_FAILURE;
+ } else if (opt_data) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
+ } else if (opt_code) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ } else {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ }
} else {
type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
}
if (libxl_bitmap_is_empty(&target_map))
libxl_bitmap_set_any(&target_map);
if (argc != optind + 2) {
- help("psr-cat-cbm-set");
+ help("psr-cat-set");
return 2;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v13 23/23] docs: add L2 CAT description in docs.
2017-07-06 1:52 [PATCH v13 00/23] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (21 preceding siblings ...)
2017-07-06 1:53 ` [PATCH v13 22/23] tools: L2 CAT: support set " Yi Sun
@ 2017-07-06 1:53 ` Yi Sun
22 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2017-07-06 1:53 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, He Chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau
This patch adds L2 CAT description in related documents.
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v13:
- rebase the patch on latest code.
---
docs/man/xl.pod.1.in | 27 +++++++++++++++++++++++----
docs/misc/xl-psr.markdown | 18 ++++++++++++------
2 files changed, 35 insertions(+), 10 deletions(-)
diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index d87fe16..16c8306 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -1712,6 +1712,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
a domain is attached to the monitoring service, monitoring data can be shown
for any of these monitoring types.
+There is no cache monitoring and memory bandwidth monitoring on L2 cache so
+far.
+
=over 4
=item B<psr-cmt-attach> I<domain-id>
@@ -1736,7 +1739,7 @@ monitor types are:
Intel Broadwell and later server platforms offer capabilities to configure and
make use of the Cache Allocation Technology (CAT) mechanisms, which enable more
-cache resources (i.e. L3 cache) to be made available for high priority
+cache resources (i.e. L3/L2 cache) to be made available for high priority
applications. In the Xen implementation, CAT is used to control cache allocation
on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
(CBM) for the domain.
@@ -1746,11 +1749,11 @@ Intel Broadwell and later server platforms also offer Code/Data Prioritization
applications. CDP is used on a per VM basis in the Xen implementation. To
specify code or data CBM for the domain, CDP feature must be enabled and CBM
type options need to be specified when setting CBM, and the type options (code
-and data) are mutually exclusive.
+and data) are mutually exclusive. There is no CDP support on L2 so far.
=over 4
-=item B<psr-cat-cbm-set> [I<OPTIONS>] I<domain-id> I<cbm>
+=item B<psr-cat-set> [I<OPTIONS>] I<domain-id> I<cbm>
Set cache capacity bitmasks(CBM) for a domain. For how to specify I<cbm>
please refer to L<http://xenbits.xen.org/docs/unstable/misc/xl-psr.html>.
@@ -1763,6 +1766,11 @@ B<OPTIONS>
Specify the socket to process, otherwise all sockets are processed.
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
=item B<-c>, B<--code>
Set code CBM when CDP is enabled.
@@ -1773,10 +1781,21 @@ Set data CBM when CDP is enabled.
=back
-=item B<psr-cat-show> [I<domain-id>]
+=item B<psr-cat-show> [I<OPTIONS>] [I<domain-id>]
Show CAT settings for a certain domain or all domains.
+B<OPTIONS>
+
+=over 4
+
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
+=back
+
=back
=head1 IGNORED FOR COMPATIBILITY WITH XM
diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
index c3c1e8e..04dd957 100644
--- a/docs/misc/xl-psr.markdown
+++ b/docs/misc/xl-psr.markdown
@@ -70,7 +70,7 @@ total-mem-bandwidth instead of cache-occupancy). E.g. after a `xl psr-cmt-attach
Cache Allocation Technology (CAT) is a new feature available on Intel
Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
-partition cache allocation (i.e. L3 cache) based on application priority or
+partition cache allocation (i.e. L3/L2 cache) based on application priority or
Class of Service (COS). Each COS is configured using capacity bitmasks (CBM)
which represent cache capacity and indicate the degree of overlap and
isolation between classes. System cache resource is divided into numbers of
@@ -107,7 +107,7 @@ System CAT information such as maximum COS and CBM length can be obtained by:
The simplest way to change a domain's CBM from its default is running:
-`xl psr-cat-cbm-set [OPTIONS] <domid> <cbm>`
+`xl psr-cat-set [OPTIONS] <domid> <cbm>`
where cbm is a number to represent the corresponding cache subset can be used.
A cbm is valid only when:
@@ -119,13 +119,19 @@ A cbm is valid only when:
In a multi-socket system, the same cbm will be set on each socket by default.
Per socket cbm can be specified with the `--socket SOCKET` option.
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
+
Setting the CBM may not be successful if insufficient COS is available. In
such case unused COS(es) may be freed by setting CBM of all related domains to
its default value(all-ones).
Per domain CBM settings can be shown by:
-`xl psr-cat-show`
+`xl psr-cat-show [OPTIONS] <domid>`
+
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
## Code and Data Prioritization (CDP)
@@ -172,13 +178,13 @@ options is invalid.
Example:
Setting code CBM for a domain:
-`xl psr-cat-cbm-set -c <domid> <cbm>`
+`xl psr-cat-set -c <domid> <cbm>`
Setting data CBM for a domain:
-`xl psr-cat-cbm-set -d <domid> <cbm>`
+`xl psr-cat-set -d <domid> <cbm>`
Setting the same code and data CBM for a domain:
-`xl psr-cat-cbm-set <domid> <cbm>`
+`xl psr-cat-set <domid> <cbm>`
## Reference
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread