* [PATCH v6 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
@ 2017-02-08 8:15 ` Yi Sun
2017-02-08 15:56 ` Konrad Rzeszutek Wilk
2017-02-08 8:15 ` [PATCH v6 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
` (22 subsequent siblings)
23 siblings, 1 reply; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:15 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch creates CAT and CDP feature document in doc/features/. It describes
key points to implement L3 CAT/CDP and L2 CAT which is described in details in
Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES".
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v6:
- write a new feature document to cover L3 CAT/CDP and L2 CAT.
- adjust 'Terminology' position in document.
- fix wordings.
- add SDM chapter title in commit message.
- add more explanations.
---
docs/features/intel_psr_cat_cdp.pandoc | 453 +++++++++++++++++++++++++++++++++
1 file changed, 453 insertions(+)
create mode 100644 docs/features/intel_psr_cat_cdp.pandoc
diff --git a/docs/features/intel_psr_cat_cdp.pandoc b/docs/features/intel_psr_cat_cdp.pandoc
new file mode 100644
index 0000000..ebce2bd
--- /dev/null
+++ b/docs/features/intel_psr_cat_cdp.pandoc
@@ -0,0 +1,453 @@
+% Intel Cache Allocation Technology and Code and Data Prioritization Features
+% Revision 1.0
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+ Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+ Component(s): Hypervisor, toolstack
+
+ Hardware: L3 CAT: Haswell and beyond CPUs
+ CDP : Broadwell and beyond CPUs
+ L2 CAT: Atom codename Goldmont and beyond CPUs
+---------------- ----------------------------------------------------
+
+# Terminology
+
+* CAT Cache Allocation Technology
+* CBM Capacity BitMasks
+* CDP Code and Data Prioritization
+* COS/CLOS Class of Service
+* MSRs Machine Specific Registers
+* PSR Intel Platform Shared Resource
+
+# Overview
+
+Intel provides a set of allocation capabilities including Cache Allocatation
+Technology (CAT) and Code and Data Prioritization (CDP).
+
+CAT allows an OS or hypervisor to control allocation of a CPU's shared cache
+based on application priority or Class of Service (COS). Each COS is configured
+using capacity bitmasks (CBMs) which represent cache capacity and indicate the
+degree of overlap and isolation between classes. Once CAT is configured, the pr-
+ocessor allows access to portions of cache according to the established COS.
+Intel Xeon processor E5 v4 family (and some others) introduce capabilities to
+configure and make use of the CAT mechanism on the L3 cache. Intel Goldmont pro-
+cessor provides support for control over the L2 cache.
+
+Code and Data Prioritization (CDP) Technology is an extension of CAT. CDP
+enables isoloation and separate prioritization of code and data fetches to
+the L3 cahce in a SW configurable manner, which can enable workload priorit-
+ization and tuning of cache capacity to the characteristics of the workload.
+CDP extends CAT by providing separate code and data masks per Class of Service
+(COS). When SW configures to enable CDP, L3 CAT is disabled.
+
+# User details
+
+* Feature Enabling:
+
+ Add "psr=cat" to boot line parameter to enable all supported level CAT featu-
+ res. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW.
+
+* xl interfaces:
+
+ 1. `psr-cat-show [OPTIONS] domain-id`:
+
+ Show L2 CAT or L3 CAT/CDP CBM of the domain designated by Xen domain-id.
+
+ Option `-l`:
+ `-l2`: Show cbm for L2 cache.
+ `-l3`: Show cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ 2. `psr-cat-set [OPTIONS] domain-id cbm`:
+
+ Set L2 CAT or L3 CAT/CDP CBM to the domain designated by Xen domain-id.
+
+ Option `-s`: Specify the socket to process, otherwise all sockets are
+ processed.
+
+ Option `-l`:
+ `-l2`: Specify cbm for L2 cache.
+ `-l3`: Specify cbm for L3 cache.
+
+ If `-lX` is specified and LX is not supported, print error.
+ If no `-l` is specified, level 3 is the default option.
+
+ Option `-c` or `-d`:
+ `-c`: Set L3 CDP code cbm.
+ `-d`: Set L3 CDP data cbm.
+
+ 3. `psr-hwinfo [OPTIONS]`:
+
+ Show CMT & L2 CAT & L3 CAT/CDP HW information on every socket.
+
+ Option `-m, --cmt`: Show Cache Monitoring Technology (CMT) hardware info.
+
+ Option `-a, --cat`: Show CAT/CDP hardware info.
+
+# Technical details
+
+L3 CAT/CDP and L2 CAT are all members of Intel PSR features, they share the base
+PSR infrastructure in Xen.
+
+## Hardware perspective
+
+ CAT/CDP defines a range of MSRs to assign different cache access patterns
+ which are known as CBMs, each CBM is associated with a COS.
+
+ ```
+ E.g. L2 CAT:
+ +----------------------------+----------------+
+ IA32_PQR_ASSOC | MSR (per socket) | Address |
+ +----+---+-------+ +----------------------------+----------------+
+ | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
+ +----+---+-------+ +----------------------------+----------------+
+ └-------------> | ... | ... |
+ +----------------------------+----------------+
+ | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
+ +----------------------------+----------------+
+ ```
+
+ L3 CAT/CDP uses a range of MSRs from 0xC90 ~ 0xC90+n (n<128).
+
+ L2 CAT uses a range of MSRs from 0xD10 ~ 0xD10+n (n<64), following the L3
+ CAT/CDP MSRs, setting different L2 cache accessing patterns from L3 cache is
+ supported.
+
+ Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the
+ hardware indicating the cache space an application should be limited to as
+ well as providing an indication of overlap and isolation in the CAT-capable
+ cache from other applications contending for the cache.
+
+ Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
+ note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
+ 0FF0H, 003CH, etc.).
+
+ ```
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Default Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | A | A | A | A | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | A | A | A | A | Overlapped Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | A | A |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+
+ +----+----+----+----+----+----+----+----+
+ | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+ +----+----+----+----+----+----+----+----+
+ COS0 | A | A | A | A | | | | | Isolated Bitmask
+ +----+----+----+----+----+----+----+----+
+ COS1 | | | | | A | A | | |
+ +----+----+----+----+----+----+----+----+
+ COS2 | | | | | | | A | A |
+ +----+----+----+----+----+----+----+----+
+ ```
+
+ We can get the CBM length through CPUID. The default value of CBM is calcul-
+ ated by `(1ull << cbm_len) - 1`. That is a fully open bitmask, all ones bitm-
+ ask. The COS[0] always stores the default value without change.
+
+ There is a `IA32_PQR_ASSOC` register which stores the COS ID of the VCPU. HW
+ enforces cache allocation according to the corresponding CBM.
+
+## The relationship between L3 CAT/CDP and L2 CAT
+
+ HW may support all features. By default, CDP is disabled on the processor.
+ If the L3 CAT MSRs are used without enabling CDP, the processor operates in
+ a traditional CAT-only mode. When CDP is enabled,
+ * the CAT mask MSRs are re-mapped into interleaved pairs of mask MSRs for
+ data or code fetches.
+ * the range of COS for CAT is re-indexed, with the lower-half of the COS
+ range available for CDP.
+
+ L2 CAT is independent of L3 CAT/CDP, which means L2 CAT can be enabled while
+ L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are both enabled.
+
+ As a requirement, the bits of CBM of CAT/CDP must be continuous.
+
+ N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same associate
+ register `IA32_PQR_ASSOC`, which means one COS is associated with a pair of
+ L2 CAT CBM and L3 CAT/CDP CBM.
+
+ Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or other
+ PSR features in future). In some cases, a VM is permitted to have a COS
+ that is beyond one (or more) of PSR features but within the others. For
+ instance, let's assume the max COS of L2 CAT is 8 but the max COS of L3
+ CAT is 16, when a VM is assigned 9 as COS, the L3 CAT CBM associated to
+ COS 9 would be enforced, but for L2 CAT, the HW works as default value is
+ set since COS 9 is beyond the max COS (8) of L2 CAT.
+
+## Design Overview
+
+* Core COS/CBM association
+
+ When enforcing CAT/CDP, all cores of domains have the same default COS (COS0)
+ which is associated with the fully open CBM (all ones bitmask) to access all
+ cache. The default COS is used only in hypervisor and is transparent to tool
+ stack and user.
+
+ System administrator can change PSR allocation policy at runtime by tool stack.
+ Since L2 CAT shares COS with L3 CAT/CDP, a COS corresponds to a 2-tuple, like
+ [L2 CBM, L3 CBM] with only-CAT enabled, when CDP is enabled, one COS correspo-
+ nds to a 3-tuple, like [L2 CBM, L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT
+ nor L3 CDP is enabled, things would be easier, one COS corresponds to one L2
+ CBM.
+
+* VCPU schedule
+
+ When context switch happens, the COS of VCPU is written to per-thread MSR
+ `IA32_PQR_ASSOC`, and then hardware enforces cache allocation according to
+ the corresponding CBM.
+
+* Multi-sockets
+
+ Different sockets may have different CAT/CDP capability (e.g. max COS) alth-
+ ough it is consistent on the same socket. So the capability of per-socket CAT/
+ CDP is specified.
+
+ 'psr-cat-set' can set CBM for one domain per socket. On each socket, we main-
+ tain a COS array for all domains. One domain uses one COS at one time. One COS
+ stores the CBM of the domain to work. So, when a VCPU of the domain is migrat-
+ ed from socket 1 to socket 2, it follows configuration on socket 2.
+
+ E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets do-
+ main 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
+ is migrated from socket 1 to 2, the COS ID used is 7, that means 0x3f is the
+ CBM to work for this domain 1 now.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+ 1. Boot line parameter "psr=cat" enables L2 CAT and L3 CAT if hardware suppo-
+ rted. "psr=cdp" enables CDP if hardware supported.
+
+ 2. SYSCTL:
+ - XEN_SYSCTL_PSR_CAT_get_l3_info: Get L3 CAT/CDP information.
+ - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
+
+ 3. DOMCTL:
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: Get L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: Set L3 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE: Get CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: Set CDP Code CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA: Get CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: Set CDP Data CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
+
+* xl interfaces:
+
+ 1. psr-cat-show -lX domain-id
+ Show LX cbm for a domain.
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
+
+ 2. psr-cat-set -lX domain-id cbm
+ Set LX cbm for a domain.
+ => XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA /
+ XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
+
+ 3. psr-hwinfo
+ Show PSR HW information, including L3 CAT/CDP/L2 CAT
+ => XEN_SYSCTL_PSR_CAT_get_l3_info /
+ XEN_SYSCTL_PSR_CAT_get_l2_info
+
+* Key data structure:
+
+ 1. Feature HW info
+
+ ```
+ struct psr_cat_hw_info {
+ unsigned int cbm_len;
+ unsigned int cos_max;
+ };
+ ```
+
+ - Member `cbm_len`
+
+ `cbm_len` is one of the hardware info of CAT. It means the max number
+ of bits to set.
+
+ - Member `cos_max`
+
+ `cos_max` is one of the hardware info of CAT. It means the max number
+ of COS registers.
+
+ 2. Feature list node
+
+ ```
+ struct feat_node {
+ enum psr_feat_type feature;
+ struct feat_ops ops;
+ struct psr_cat_hw_info info;
+ uint64_t cos_reg_val[MAX_COS_REG_NUM];
+ struct list_head list;
+ };
+ ```
+
+ When a PSR enforcement feature is enabled, it will be added into a
+ feature list. The head of the list is created in psr initialization.
+
+ - Member `feature`
+
+ `feature` is an integer number, to indicate which feature the list entry
+ corresponds to.
+
+ - Member `ops`
+
+ `ops` maintains a callback function list of the feature. It will be introduced
+ in details later at `4. Feature operation functions structure`.
+
+ - Member `info`
+
+ `info` maintains the feature HW information which are provided to psr_hwinfo
+ command.
+
+ - Member `cos_reg_val`
+
+ `cos_reg_val` is an array to maintain the value set in all COS registers of
+ the feature. The array is indexed by COS ID.
+
+ 3. Per-socket PSR features information structure
+
+ ```
+ struct psr_socket_info {
+ unsigned int feat_mask;
+ unsigned int nr_feat;
+ struct list_head feat_list;
+ unsigned int cos_ref[MAX_COS_REG_NUM];
+ spinlock_t ref_lock;
+ };
+ ```
+
+ We collect all PSR allocation features information of a socket in this
+ `struct psr_socket_info`.
+
+ - Member `feat_mask`
+
+ `feat_mask` is a bitmap, to indicate which feature is enabled on current
+ socket. We define `feat_mask` bitmap as:
+
+ bit 0: L3 CAT status.
+ bit 1: L3 CDP status.
+ bit 2: L2 CAT status.
+
+ - Member `nr_feat`
+
+ `nr_feat` means the number of PSR features enabled.
+
+ - Member `cos_ref`
+
+ `cos_ref` is an array which maintains the reference of one COS. It maps
+ to cos_reg_val[MAX_COS_REG_NUM] in `struct feat_node`. If one COS is
+ used by one domain, the corresponding reference will increase by one. If
+ a domain releases the COS, the reference will decrease by one. The array
+ is indexed by COS ID.
+
+ 4. Feature operation functions structure
+
+ ```
+ struct feat_ops {
+ unsigned int (*get_cos_max)(const struct feat_node *feat);
+ int (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], uint32_t array_len);
+ int (*get_val)(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val);
+ unsigned int (*get_cos_num)(const struct feat_node *feat);
+ int (*get_old_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos);
+ int (*set_new_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos,
+ enum cbm_type type,
+ uint64_t m);
+ int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
+ unsigned int cos, bool *found);
+ unsigned int (*fits_cos_max)(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos);
+ int (*write_msr)(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat);
+ };
+ ```
+
+ We abstract above callback functions to encapsulate the feature specific
+ behaviors into them. Then, it is easy to add a new feature. We just need:
+ 1) Implement such ops and callback functions for every feature.
+ 2) Register the ops into `struct feat_node`.
+ 3) Add the feature into feature list during CPU initialization.
+
+# Limitations
+
+CAT/CDP can only work on HW which enables it(check by CPUID). So far, there is
+no HW which enables both L2 CAT and L3 CAT/CDP. But SW implementation has cons-
+idered such scenario to enable both L2 CAT and L3 CAT/CDP.
+
+# Testing
+
+We can execute above xl commands to verify L2 CAT and L3 CAT/CDP on different
+HWs support them.
+
+For example:
+ root@:~$ xl psr-hwinfo --cat
+ Cache Allocation Technology (CAT): L2
+ Socket ID : 0
+ Maximum COS : 3
+ CBM length : 8
+ Default CBM : 0xff
+
+ root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
+
+ root@:~$ xl psr-cat-show -l2 1
+ Socket ID : 0
+ Default CBM : 0xff
+ ID NAME CBM
+ 1 ubuntu14 0x7f
+
+# Areas for improvement
+
+N/A
+
+# Known issues
+
+N/A
+
+# References
+
+"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date Revision Version Notes
+---------- -------- -------- -------------------------------------------
+2016-08-12 1.0 Xen 4.9 Design document written
+---------- -------- -------- -------------------------------------------
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v6 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-02-08 8:15 ` [PATCH v6 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
@ 2017-02-08 15:56 ` Konrad Rzeszutek Wilk
2017-02-09 6:38 ` Yi Sun
0 siblings, 1 reply; 37+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-08 15:56 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 08, 2017 at 04:15:53PM +0800, Yi Sun wrote:
> This patch creates CAT and CDP feature document in doc/features/. It describes
> key points to implement L3 CAT/CDP and L2 CAT which is described in details in
> Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES".
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> v6:
> - write a new feature document to cover L3 CAT/CDP and L2 CAT.
> - adjust 'Terminology' position in document.
> - fix wordings.
> - add SDM chapter title in commit message.
> - add more explanations.
> ---
> docs/features/intel_psr_cat_cdp.pandoc | 453 +++++++++++++++++++++++++++++++++
> 1 file changed, 453 insertions(+)
> create mode 100644 docs/features/intel_psr_cat_cdp.pandoc
>
> diff --git a/docs/features/intel_psr_cat_cdp.pandoc b/docs/features/intel_psr_cat_cdp.pandoc
> new file mode 100644
> index 0000000..ebce2bd
> --- /dev/null
> +++ b/docs/features/intel_psr_cat_cdp.pandoc
> @@ -0,0 +1,453 @@
> +% Intel Cache Allocation Technology and Code and Data Prioritization Features
> +% Revision 1.0
> +
> +\clearpage
> +
> +# Basics
> +
> +---------------- ----------------------------------------------------
> + Status: **Tech Preview**
> +
> +Architecture(s): Intel x86
> +
> + Component(s): Hypervisor, toolstack
> +
> + Hardware: L3 CAT: Haswell and beyond CPUs
> + CDP : Broadwell and beyond CPUs
> + L2 CAT: Atom codename Goldmont and beyond CPUs
> +---------------- ----------------------------------------------------
> +
> +# Terminology
> +
> +* CAT Cache Allocation Technology
> +* CBM Capacity BitMasks
> +* CDP Code and Data Prioritization
> +* COS/CLOS Class of Service
> +* MSRs Machine Specific Registers
> +* PSR Intel Platform Shared Resource
> +
> +# Overview
> +
> +Intel provides a set of allocation capabilities including Cache Allocatation
> +Technology (CAT) and Code and Data Prioritization (CDP).
> +
> +CAT allows an OS or hypervisor to control allocation of a CPU's shared cache
> +based on application priority or Class of Service (COS). Each COS is configured
> +using capacity bitmasks (CBMs) which represent cache capacity and indicate the
> +degree of overlap and isolation between classes. Once CAT is configured, the pr-
> +ocessor allows access to portions of cache according to the established COS.
> +Intel Xeon processor E5 v4 family (and some others) introduce capabilities to
> +configure and make use of the CAT mechanism on the L3 cache. Intel Goldmont pro-
> +cessor provides support for control over the L2 cache.
> +
> +Code and Data Prioritization (CDP) Technology is an extension of CAT. CDP
> +enables isoloation and separate prioritization of code and data fetches to
^^^^^^^^^^
isolation
> +the L3 cahce in a SW configurable manner, which can enable workload priorit-
^^^^^
cache
> +ization and tuning of cache capacity to the characteristics of the workload.
> +CDP extends CAT by providing separate code and data masks per Class of Service
> +(COS). When SW configures to enable CDP, L3 CAT is disabled.
> +
> +# User details
> +
> +* Feature Enabling:
> +
> + Add "psr=cat" to boot line parameter to enable all supported level CAT featu-
> + res. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW.
> +
> +* xl interfaces:
> +
> + 1. `psr-cat-show [OPTIONS] domain-id`:
> +
> + Show L2 CAT or L3 CAT/CDP CBM of the domain designated by Xen domain-id.
> +
> + Option `-l`:
> + `-l2`: Show cbm for L2 cache.
> + `-l3`: Show cbm for L3 cache.
> +
> + If `-lX` is specified and LX is not supported, print error.
> + If no `-l` is specified, level 3 is the default option.
> +
> + 2. `psr-cat-set [OPTIONS] domain-id cbm`:
> +
> + Set L2 CAT or L3 CAT/CDP CBM to the domain designated by Xen domain-id.
> +
> + Option `-s`: Specify the socket to process, otherwise all sockets are
> + processed.
> +
> + Option `-l`:
> + `-l2`: Specify cbm for L2 cache.
> + `-l3`: Specify cbm for L3 cache.
> +
> + If `-lX` is specified and LX is not supported, print error.
> + If no `-l` is specified, level 3 is the default option.
> +
> + Option `-c` or `-d`:
> + `-c`: Set L3 CDP code cbm.
> + `-d`: Set L3 CDP data cbm.
> +
> + 3. `psr-hwinfo [OPTIONS]`:
> +
> + Show CMT & L2 CAT & L3 CAT/CDP HW information on every socket.
> +
> + Option `-m, --cmt`: Show Cache Monitoring Technology (CMT) hardware info.
> +
> + Option `-a, --cat`: Show CAT/CDP hardware info.
> +
> +# Technical details
> +
> +L3 CAT/CDP and L2 CAT are all members of Intel PSR features, they share the base
> +PSR infrastructure in Xen.
> +
> +## Hardware perspective
> +
> + CAT/CDP defines a range of MSRs to assign different cache access patterns
> + which are known as CBMs, each CBM is associated with a COS.
> +
> + ```
> + E.g. L2 CAT:
> + +----------------------------+----------------+
> + IA32_PQR_ASSOC | MSR (per socket) | Address |
> + +----+---+-------+ +----------------------------+----------------+
> + | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
> + +----+---+-------+ +----------------------------+----------------+
> + └-------------> | ... | ... |
> + +----------------------------+----------------+
> + | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
> + +----------------------------+----------------+
> + ```
> +
> + L3 CAT/CDP uses a range of MSRs from 0xC90 ~ 0xC90+n (n<128).
> +
> + L2 CAT uses a range of MSRs from 0xD10 ~ 0xD10+n (n<64), following the L3
> + CAT/CDP MSRs, setting different L2 cache accessing patterns from L3 cache is
> + supported.
> +
> + Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the
> + hardware indicating the cache space an application should be limited to as
s/application/VM/ ?
> + well as providing an indication of overlap and isolation in the CAT-capable
> + cache from other applications contending for the cache.
s/application/VM/ ?
Perhaps 'domain' as you use that later in the document?
> +
> + Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
> + note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
> + 0FF0H, 003CH, etc.).
> +
> + ```
> + +----+----+----+----+----+----+----+----+
> + | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
> + +----+----+----+----+----+----+----+----+
> + COS0 | A | A | A | A | A | A | A | A | Default Bitmask
> + +----+----+----+----+----+----+----+----+
> + COS1 | A | A | A | A | A | A | A | A |
> + +----+----+----+----+----+----+----+----+
> + COS2 | A | A | A | A | A | A | A | A |
> + +----+----+----+----+----+----+----+----+
> +
> + +----+----+----+----+----+----+----+----+
> + | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
> + +----+----+----+----+----+----+----+----+
> + COS0 | A | A | A | A | A | A | A | A | Overlapped Bitmask
> + +----+----+----+----+----+----+----+----+
> + COS1 | | | | | A | A | A | A |
> + +----+----+----+----+----+----+----+----+
> + COS2 | | | | | | | A | A |
> + +----+----+----+----+----+----+----+----+
> +
> + +----+----+----+----+----+----+----+----+
> + | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
> + +----+----+----+----+----+----+----+----+
> + COS0 | A | A | A | A | | | | | Isolated Bitmask
> + +----+----+----+----+----+----+----+----+
> + COS1 | | | | | A | A | | |
> + +----+----+----+----+----+----+----+----+
> + COS2 | | | | | | | A | A |
> + +----+----+----+----+----+----+----+----+
> + ```
> +
> + We can get the CBM length through CPUID. The default value of CBM is calcul-
> + ated by `(1ull << cbm_len) - 1`. That is a fully open bitmask, all ones bitm-
> + ask. The COS[0] always stores the default value without change.
> +
> + There is a `IA32_PQR_ASSOC` register which stores the COS ID of the VCPU. HW
> + enforces cache allocation according to the corresponding CBM.
> +
> +## The relationship between L3 CAT/CDP and L2 CAT
> +
> + HW may support all features. By default, CDP is disabled on the processor.
> + If the L3 CAT MSRs are used without enabling CDP, the processor operates in
> + a traditional CAT-only mode. When CDP is enabled,
s/,/:/
> + * the CAT mask MSRs are re-mapped into interleaved pairs of mask MSRs for
> + data or code fetches.
> + * the range of COS for CAT is re-indexed, with the lower-half of the COS
> + range available for CDP.
> +
> + L2 CAT is independent of L3 CAT/CDP, which means L2 CAT can be enabled while
> + L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are both enabled.
> +
> + As a requirement, the bits of CBM of CAT/CDP must be continuous.
> +
> + N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same associate
> + register `IA32_PQR_ASSOC`, which means one COS is associated with a pair of
> + L2 CAT CBM and L3 CAT/CDP CBM.
> +
> + Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or other
> + PSR features in future). In some cases, a VM is permitted to have a COS
I noticed you say 'domain' later on in the document. Would it make
sense to replace s/VM/domain/ to be same in this design?
> + that is beyond one (or more) of PSR features but within the others. For
> + instance, let's assume the max COS of L2 CAT is 8 but the max COS of L3
> + CAT is 16, when a VM is assigned 9 as COS, the L3 CAT CBM associated to
> + COS 9 would be enforced, but for L2 CAT, the HW works as default value is
> + set since COS 9 is beyond the max COS (8) of L2 CAT.
> +
> +## Design Overview
> +
> +* Core COS/CBM association
> +
> + When enforcing CAT/CDP, all cores of domains have the same default COS (COS0)
> + which is associated with the fully open CBM (all ones bitmask) to access all
> + cache. The default COS is used only in hypervisor and is transparent to tool
> + stack and user.
> +
> + System administrator can change PSR allocation policy at runtime by tool stack.
> + Since L2 CAT shares COS with L3 CAT/CDP, a COS corresponds to a 2-tuple, like
> + [L2 CBM, L3 CBM] with only-CAT enabled, when CDP is enabled, one COS correspo-
> + nds to a 3-tuple, like [L2 CBM, L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT
> + nor L3 CDP is enabled, things would be easier, one COS corresponds to one L2
> + CBM.
> +
> +* VCPU schedule
> +
> + When context switch happens, the COS of VCPU is written to per-thread MSR
> + `IA32_PQR_ASSOC`, and then hardware enforces cache allocation according to
> + the corresponding CBM.
> +
> +* Multi-sockets
> +
> + Different sockets may have different CAT/CDP capability (e.g. max COS) alth-
> + ough it is consistent on the same socket. So the capability of per-socket CAT/
> + CDP is specified.
> +
> + 'psr-cat-set' can set CBM for one domain per socket. On each socket, we main-
> + tain a COS array for all domains. One domain uses one COS at one time. One COS
> + stores the CBM of the domain to work. So, when a VCPU of the domain is migrat-
> + ed from socket 1 to socket 2, it follows configuration on socket 2.
> +
> + E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets do-
> + main 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
> + is migrated from socket 1 to 2, the COS ID used is 7, that means 0x3f is the
> + CBM to work for this domain 1 now.
> +
> +## Implementation Description
> +
> +* Hypervisor interfaces:
> +
> + 1. Boot line parameter "psr=cat" enables L2 CAT and L3 CAT if hardware suppo-
> + rted. "psr=cdp" enables CDP if hardware supported.
> +
> + 2. SYSCTL:
> + - XEN_SYSCTL_PSR_CAT_get_l3_info: Get L3 CAT/CDP information.
> + - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
> +
> + 3. DOMCTL:
> + - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: Get L3 CBM for a domain.
> + - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: Set L3 CBM for a domain.
> + - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE: Get CDP Code CBM for a domain.
> + - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: Set CDP Code CBM for a domain.
> + - XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA: Get CDP Data CBM for a domain.
> + - XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: Set CDP Data CBM for a domain.
> + - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
> + - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
> +
> +* xl interfaces:
> +
> + 1. psr-cat-show -lX domain-id
> + Show LX cbm for a domain.
> + => XEN_SYSCTL_PSR_CAT_get_l3_info /
> + XEN_SYSCTL_PSR_CAT_get_l2_info /
> + XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM /
> + XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE /
> + XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA /
> + XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
> +
> + 2. psr-cat-set -lX domain-id cbm
> + Set LX cbm for a domain.
> + => XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM /
> + XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE /
> + XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA /
> + XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
> +
> + 3. psr-hwinfo
> + Show PSR HW information, including L3 CAT/CDP/L2 CAT
> + => XEN_SYSCTL_PSR_CAT_get_l3_info /
> + XEN_SYSCTL_PSR_CAT_get_l2_info
> +
> +* Key data structure:
> +
> + 1. Feature HW info
> +
> + ```
> + struct psr_cat_hw_info {
> + unsigned int cbm_len;
> + unsigned int cos_max;
> + };
> + ```
> +
> + - Member `cbm_len`
> +
> + `cbm_len` is one of the hardware info of CAT. It means the max number
> + of bits to set.
> +
> + - Member `cos_max`
> +
> + `cos_max` is one of the hardware info of CAT. It means the max number
> + of COS registers.
> +
> + 2. Feature list node
> +
> + ```
> + struct feat_node {
> + enum psr_feat_type feature;
> + struct feat_ops ops;
> + struct psr_cat_hw_info info;
> + uint64_t cos_reg_val[MAX_COS_REG_NUM];
> + struct list_head list;
> + };
> + ```
> +
> + When a PSR enforcement feature is enabled, it will be added into a
> + feature list. The head of the list is created in psr initialization.
> +
> + - Member `feature`
> +
> + `feature` is an integer number, to indicate which feature the list entry
> + corresponds to.
> +
> + - Member `ops`
> +
> + `ops` maintains a callback function list of the feature. It will be introduced
> + in details later at `4. Feature operation functions structure`.
I think you can just do:
[Feature operation functions structure]
And when you run `pandoc -toc -o intel_psr_cat_cdp.pdf
intel_psr_cat_cdp.pandoc`
it will provide the right link (which you can follow) to the proper
section.
> +
> + - Member `info`
> +
> + `info` maintains the feature HW information which are provided to psr_hwinfo
> + command.
> +
> + - Member `cos_reg_val`
> +
> + `cos_reg_val` is an array to maintain the value set in all COS registers of
> + the feature. The array is indexed by COS ID.
> +
> + 3. Per-socket PSR features information structure
> +
> + ```
> + struct psr_socket_info {
> + unsigned int feat_mask;
> + unsigned int nr_feat;
> + struct list_head feat_list;
> + unsigned int cos_ref[MAX_COS_REG_NUM];
> + spinlock_t ref_lock;
> + };
> + ```
> +
> + We collect all PSR allocation features information of a socket in this
> + `struct psr_socket_info`.
> +
> + - Member `feat_mask`
> +
> + `feat_mask` is a bitmap, to indicate which feature is enabled on current
> + socket. We define `feat_mask` bitmap as:
> +
> + bit 0: L3 CAT status.
> + bit 1: L3 CDP status.
> + bit 2: L2 CAT status.
Just in case if you change the code and there are more bit positions - I
would recommend you replace the 'We define 'feat_mask' bitmap as:
.. bit 0 .."
with:
"See values defined in 'enum psr_feat_type'"
As that will make it easier in case the code is changed but the doc
becomes out-dated.
> +
> + - Member `nr_feat`
> +
> + `nr_feat` means the number of PSR features enabled.
> +
> + - Member `cos_ref`
> +
> + `cos_ref` is an array which maintains the reference of one COS. It maps
> + to cos_reg_val[MAX_COS_REG_NUM] in `struct feat_node`. If one COS is
> + used by one domain, the corresponding reference will increase by one. If
> + a domain releases the COS, the reference will decrease by one. The array
> + is indexed by COS ID.
> +
> + 4. Feature operation functions structure
> +
> + ```
> + struct feat_ops {
> + unsigned int (*get_cos_max)(const struct feat_node *feat);
> + int (*get_feat_info)(const struct feat_node *feat,
> + uint32_t data[], uint32_t array_len);
> + int (*get_val)(const struct feat_node *feat, unsigned int cos,
> + enum cbm_type type, uint64_t *val);
> + unsigned int (*get_cos_num)(const struct feat_node *feat);
> + int (*get_old_val)(uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int old_cos);
> + int (*set_new_val)(uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int old_cos,
> + enum cbm_type type,
> + uint64_t m);
> + int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
> + unsigned int cos, bool *found);
> + unsigned int (*fits_cos_max)(const uint64_t val[],
> + const struct feat_node *feat,
> + unsigned int cos);
> + int (*write_msr)(unsigned int cos, const uint64_t val[],
> + struct feat_node *feat);
> + };
> + ```
> +
> + We abstract above callback functions to encapsulate the feature specific
> + behaviors into them. Then, it is easy to add a new feature. We just need:
> + 1) Implement such ops and callback functions for every feature.
> + 2) Register the ops into `struct feat_node`.
> + 3) Add the feature into feature list during CPU initialization.
> +
> +# Limitations
> +
> +CAT/CDP can only work on HW which enables it(check by CPUID). So far, there is
> +no HW which enables both L2 CAT and L3 CAT/CDP. But SW implementation has cons-
> +idered such scenario to enable both L2 CAT and L3 CAT/CDP.
> +
> +# Testing
> +
> +We can execute above xl commands to verify L2 CAT and L3 CAT/CDP on different
> +HWs support them.
> +
> +For example:
> + root@:~$ xl psr-hwinfo --cat
> + Cache Allocation Technology (CAT): L2
> + Socket ID : 0
> + Maximum COS : 3
> + CBM length : 8
> + Default CBM : 0xff
> +
> + root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
> +
> + root@:~$ xl psr-cat-show -l2 1
> + Socket ID : 0
> + Default CBM : 0xff
> + ID NAME CBM
> + 1 ubuntu14 0x7f
> +
> +# Areas for improvement
> +
> +N/A
I would say that using '0x7f' is not very user-friendly. It really
would be good if that changed to something easier to grok.
For example if I am system admin and I look at:
+----+----+----+----+----+----+----+----+
| M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+----+----+----+----+----+----+----+----+
COS0 | A | A | A | A | | | | | Isolated Bitmask
+----+----+----+----+----+----+----+----+
COS1 | | | | | A | A | | |
+----+----+----+----+----+----+----+----+
COS2 | | | | | | | A | A |
+----+----+----+----+----+----+----+----+
I would think that giving an guest 'M7->M4' means it has more
cache than M3->M2 or M1->M0.
But that is not spelled in details. Or what happens if I do:
+----+----+----+----+----+----+----+----+
| M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+----+----+----+----+----+----+----+----+
COS0 | | | | | | A | | | Isolated Bitmask
+----+----+----+----+----+----+----+----+
COS1 | | | | | | | A | |
+----+----+----+----+----+----+----+----+
COS2 | | | | | | | | A |
+----+----+----+----+----+----+----+----+
Does that have the same effect as the previous one?
I would think not, but perhaps it is the same (we set
three 'pools').
And does this mean that I've made a grave error
and M7->M3 are in effect only available to the hypervisor (and
dom0?, but only if dom0 is PV, but not for PVH dom0?)
Thanks!
> +
> +# Known issues
> +
> +N/A
> +
> +# References
> +
> +"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
> +
> +# History
> +
> +------------------------------------------------------------------------
> +Date Revision Version Notes
> +---------- -------- -------- -------------------------------------------
> +2016-08-12 1.0 Xen 4.9 Design document written
Perhaps update that a bit? I think we are at 1.6 ?
> +---------- -------- -------- -------------------------------------------
> --
> 1.9.1
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v6 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
2017-02-08 15:56 ` Konrad Rzeszutek Wilk
@ 2017-02-09 6:38 ` Yi Sun
0 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-09 6:38 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-08 10:56:55, Konrad Rzeszutek Wilk wrote:
> On Wed, Feb 08, 2017 at 04:15:53PM +0800, Yi Sun wrote:
> > + Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the
> > + hardware indicating the cache space an application should be limited to as
>
> s/application/VM/ ?
>
> > + well as providing an indication of overlap and isolation in the CAT-capable
> > + cache from other applications contending for the cache.
>
> s/application/VM/ ?
>
> Perhaps 'domain' as you use that later in the document?
>
> > + PSR features in future). In some cases, a VM is permitted to have a COS
>
> I noticed you say 'domain' later on in the document. Would it make
> sense to replace s/VM/domain/ to be same in this design?
>
I think domain should be better.
> > + - Member `ops`
> > +
> > + `ops` maintains a callback function list of the feature. It will be introduced
> > + in details later at `4. Feature operation functions structure`.
>
> I think you can just do:
>
> [Feature operation functions structure]
>
> And when you run `pandoc -toc -o intel_psr_cat_cdp.pdf
> intel_psr_cat_cdp.pandoc`
>
> it will provide the right link (which you can follow) to the proper
> section.
I tried this command but encountered below error.
$ pandoc -toc -o intel_psr_cat_cdp.pdf docs/features/intel_psr_cat_cdp.pandoc
pandoc: cannot produce pdf output with oc writer
> > + root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
> > +
> > + root@:~$ xl psr-cat-show -l2 1
> > + Socket ID : 0
> > + Default CBM : 0xff
> > + ID NAME CBM
> > + 1 ubuntu14 0x7f
> > +
> > +# Areas for improvement
> > +
> > +N/A
>
> I would say that using '0x7f' is not very user-friendly. It really
> would be good if that changed to something easier to grok.
>
I agree that '0x7f' is not user-friendly. This needs the user know some HW
details.
> For example if I am system admin and I look at:
>
> +----+----+----+----+----+----+----+----+
> | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
> +----+----+----+----+----+----+----+----+
> COS0 | A | A | A | A | | | | | Isolated Bitmask
> +----+----+----+----+----+----+----+----+
> COS1 | | | | | A | A | | |
> +----+----+----+----+----+----+----+----+
> COS2 | | | | | | | A | A |
> +----+----+----+----+----+----+----+----+
>
> I would think that giving an guest 'M7->M4' means it has more
> cache than M3->M2 or M1->M0.
>
> But that is not spelled in details. Or what happens if I do:
> +----+----+----+----+----+----+----+----+
> | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
> +----+----+----+----+----+----+----+----+
> COS0 | | | | | | A | | | Isolated Bitmask
> +----+----+----+----+----+----+----+----+
> COS1 | | | | | | | A | |
> +----+----+----+----+----+----+----+----+
> COS2 | | | | | | | | A |
> +----+----+----+----+----+----+----+----+
>
> Does that have the same effect as the previous one?
> I would think not, but perhaps it is the same (we set
> three 'pools').
>
No, this has different effect. One bit means a quantity of cache capacity, e.g.
1 bit equals 1M. For first case, 0 uses 4M cache, 1 uses 2M, 2 uses 2M. For
second case, all three use 1M respectively.
Although '0x7F' is not user friendly, it gives the ability to control the
granularity of cache allocation. Furthermore, the implementation in Xen is low
level interface and we should provide the most flexibility to users. I think
the upper layer, e.g. libvirt, may do further wrapper to provide more user
friendly interfaces.
Thanks,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v6 02/24] x86: refactor psr: remove L3 CAT/CDP codes.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-02-08 8:15 ` [PATCH v6 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
@ 2017-02-08 8:15 ` Yi Sun
2017-02-08 8:15 ` [PATCH v6 03/24] x86: refactor psr: implement main data structures Yi Sun
` (21 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:15 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
The current cache allocation codes in psr.c do not consider
future features addition and are not friendly to extend.
To make psr.c be more flexible to add new features and fulfill
the program principle, open for extension but closed for
modification, we have to refactor the psr.c:
1. Analyze cache allocation features and abstract general data
structures.
2. Analyze the init and all other functions flow, abstract all
steps that different features may have different implementations.
Make these steps be callback functions and register feature
specific fuctions. Then, the main processes will not be changed
when introducing a new feature.
Because the quantity of refactor codes is big and the logics are
changed a lot, it will cause reviewers confused if just change
old codes. Reviewers have to understand both old codes and new
implementations. After review iterations from V1 to V3, Jan has
proposed to remove all old cache allocation codes firstly, then
implement new codes step by step. This will help to make codes
be more easily reviewable.
There is no construction without destruction. So, this patch
removes all current L3 CAT/CDP codes in psr.c. The following
patches will introduce the new mechanism.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
xen/arch/x86/psr.c | 470 +----------------------------------------------------
1 file changed, 5 insertions(+), 465 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0b5073c..96a8589 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -23,24 +23,6 @@
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
-struct psr_cat_cbm {
- union {
- uint64_t cbm;
- struct {
- uint64_t code;
- uint64_t data;
- };
- };
- unsigned int ref;
-};
-
-struct psr_cat_socket_info {
- unsigned int cbm_len;
- unsigned int cos_max;
- struct psr_cat_cbm *cos_to_cbm;
- spinlock_t cbm_lock;
-};
-
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -48,26 +30,11 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
-static unsigned long *__read_mostly cat_socket_enable;
-static struct psr_cat_socket_info *__read_mostly cat_socket_info;
-static unsigned long *__read_mostly cdp_socket_enable;
-
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
-static unsigned int __read_mostly opt_cos_max = 255;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
-static struct psr_cat_cbm *temp_cos_to_cbm;
-
-static unsigned int get_socket_cpu(unsigned int socket)
-{
- if ( likely(socket < nr_sockets) )
- return cpumask_any(socket_cpumask[socket]);
-
- return nr_cpu_ids;
-}
-
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -107,9 +74,6 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
- if ( val_str && !strcmp(s, "cos_max") )
- opt_cos_max = simple_strtoul(val_str, NULL, 0);
-
s = ss + 1;
} while ( ss );
}
@@ -213,16 +177,7 @@ static inline void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( cat_socket_info )
- {
- unsigned int socket = cpu_to_socket(smp_processor_id());
-
- if ( test_bit(socket, cat_socket_enable) )
- psra->cos_mask = ((1ull << get_count_order(
- cat_socket_info[socket].cos_max)) - 1) << 32;
- }
-
- if ( psr_cmt_enabled() || psra->cos_mask )
+ if ( psr_cmt_enabled() )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -231,12 +186,6 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
-static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
- uint64_t cos_mask)
-{
- *reg = (*reg & ~cos_mask) | (((uint64_t)cos << 32) & cos_mask);
-}
-
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -245,459 +194,54 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
- if ( psra->cos_mask )
- psr_assoc_cos(®, d->arch.psr_cos_ids ?
- d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
- 0, psra->cos_mask);
-
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
psra->val = reg;
}
}
-static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
-{
- if ( !cat_socket_info )
- return ERR_PTR(-ENODEV);
-
- if ( socket >= nr_sockets )
- return ERR_PTR(-ENOTSOCK);
-
- if ( !test_bit(socket, cat_socket_enable) )
- return ERR_PTR(-ENOENT);
-
- return cat_socket_info + socket;
-}
-
-static inline bool_t cdp_is_enabled(unsigned int socket)
-{
- return cdp_socket_enable && test_bit(socket, cdp_socket_enable);
-}
int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
uint32_t *cos_max, uint32_t *flags)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- *cbm_len = info->cbm_len;
- *cos_max = info->cos_max;
-
- *flags = 0;
- if ( cdp_is_enabled(socket) )
- *flags |= XEN_SYSCTL_PSR_CAT_L3_CDP;
-
return 0;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
- bool_t cdp_enabled = cdp_is_enabled(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- if ( cdp_enabled )
- return -EXDEV;
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].code;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
- else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].data;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- }
-
- return 0;
-}
-
-static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
-{
- unsigned int first_bit, zero_bit;
-
- /* Set bits should only in the range of [0, cbm_len). */
- if ( cbm & (~0ull << cbm_len) )
- return 0;
-
- /* At least one bit need to be set. */
- if ( cbm == 0 )
- return 0;
-
- first_bit = find_first_bit(&cbm, cbm_len);
- zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
-
- /* Set bits should be contiguous. */
- if ( zero_bit < cbm_len &&
- find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
- return 0;
-
- return 1;
-}
-
-struct cos_cbm_info
-{
- unsigned int cos;
- bool_t cdp;
- uint64_t cbm_code;
- uint64_t cbm_data;
-};
-
-static void do_write_l3_cbm(void *data)
-{
- struct cos_cbm_info *info = data;
-
- if ( info->cdp )
- {
- wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(info->cos), info->cbm_code);
- wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(info->cos), info->cbm_data);
- }
- else
- wrmsrl(MSR_IA32_PSR_L3_MASK(info->cos), info->cbm_code);
-}
-
-static int write_l3_cbm(unsigned int socket, unsigned int cos,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp)
-{
- struct cos_cbm_info info =
- {
- .cos = cos,
- .cbm_code = cbm_code,
- .cbm_data = cbm_data,
- .cdp = cdp,
- };
-
- if ( socket == cpu_to_socket(smp_processor_id()) )
- do_write_l3_cbm(&info);
- else
- {
- unsigned int cpu = get_socket_cpu(socket);
-
- if ( cpu >= nr_cpu_ids )
- return -ENOTSOCK;
- on_selected_cpus(cpumask_of(cpu), do_write_l3_cbm, &info, 1);
- }
-
return 0;
}
-static int find_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
-{
- unsigned int cos;
-
- for ( cos = 0; cos <= cos_max; cos++ )
- {
- if ( (map[cos].ref || cos == 0) &&
- ((!cdp_enabled && map[cos].cbm == cbm_code) ||
- (cdp_enabled && map[cos].code == cbm_code &&
- map[cos].data == cbm_data)) )
- return cos;
- }
-
- return -ENOENT;
-}
-
-static int pick_avail_cos(struct psr_cat_cbm *map, unsigned int cos_max,
- unsigned int old_cos)
-{
- unsigned int cos;
-
- /* If old cos is referred only by the domain, then use it. */
- if ( map[old_cos].ref == 1 && old_cos != 0 )
- return old_cos;
-
- /* Find an unused one other than cos0. */
- for ( cos = 1; cos <= cos_max; cos++ )
- if ( map[cos].ref == 0 )
- return cos;
-
- return -ENOENT;
-}
-
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type)
{
- unsigned int old_cos, cos_max;
- int cos, ret;
- uint64_t cbm_data, cbm_code;
- bool_t cdp_enabled = cdp_is_enabled(socket);
- struct psr_cat_cbm *map;
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
- if ( IS_ERR(info) )
- return PTR_ERR(info);
-
- if ( !psr_check_cbm(info->cbm_len, cbm) )
- return -EINVAL;
-
- if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
- type == PSR_CBM_TYPE_L3_DATA) )
- return -ENXIO;
-
- cos_max = info->cos_max;
- old_cos = d->arch.psr_cos_ids[socket];
- map = info->cos_to_cbm;
-
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- cbm_code = cbm;
- cbm_data = cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- cbm_code = cbm;
- cbm_data = map[old_cos].data;
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- cbm_code = map[old_cos].code;
- cbm_data = cbm;
- break;
-
- default:
- ASSERT_UNREACHABLE();
- return -EINVAL;
- }
-
- spin_lock(&info->cbm_lock);
- cos = find_cos(map, cos_max, cbm_code, cbm_data, cdp_enabled);
- if ( cos >= 0 )
- {
- if ( cos == old_cos )
- {
- spin_unlock(&info->cbm_lock);
- return 0;
- }
- }
- else
- {
- cos = pick_avail_cos(map, cos_max, old_cos);
- if ( cos < 0 )
- {
- spin_unlock(&info->cbm_lock);
- return cos;
- }
-
- /* We try to avoid writing MSR. */
- if ( (cdp_enabled &&
- (map[cos].code != cbm_code || map[cos].data != cbm_data)) ||
- (!cdp_enabled && map[cos].cbm != cbm_code) )
- {
- ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
- if ( ret )
- {
- spin_unlock(&info->cbm_lock);
- return ret;
- }
- map[cos].code = cbm_code;
- map[cos].data = cbm_data;
- }
- }
-
- map[cos].ref++;
- map[old_cos].ref--;
- spin_unlock(&info->cbm_lock);
-
- d->arch.psr_cos_ids[socket] = cos;
-
return 0;
}
-/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
-static void psr_free_cos(struct domain *d)
-{
- unsigned int socket;
- unsigned int cos;
- struct psr_cat_socket_info *info;
-
- if( !d->arch.psr_cos_ids )
- return;
-
- for_each_set_bit(socket, cat_socket_enable, nr_sockets)
- {
- if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
- continue;
-
- info = cat_socket_info + socket;
- spin_lock(&info->cbm_lock);
- info->cos_to_cbm[cos].ref--;
- spin_unlock(&info->cbm_lock);
- }
-
- xfree(d->arch.psr_cos_ids);
- d->arch.psr_cos_ids = NULL;
-}
-
int psr_domain_init(struct domain *d)
{
- if ( cat_socket_info )
- {
- d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
- if ( !d->arch.psr_cos_ids )
- return -ENOMEM;
- }
-
return 0;
}
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
- psr_free_cos(d);
-}
-
-static int cat_cpu_prepare(unsigned int cpu)
-{
- if ( !cat_socket_info )
- return 0;
-
- if ( temp_cos_to_cbm == NULL &&
- (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
- opt_cos_max + 1UL)) == NULL )
- return -ENOMEM;
-
- return 0;
-}
-
-static void cat_cpu_init(void)
-{
- unsigned int eax, ebx, ecx, edx;
- struct psr_cat_socket_info *info;
- unsigned int socket;
- unsigned int cpu = smp_processor_id();
- uint64_t val;
- const struct cpuinfo_x86 *c = cpu_data + cpu;
-
- if ( !cpu_has(c, X86_FEATURE_PQE) || c->cpuid_level < PSR_CPUID_LEVEL_CAT )
- return;
-
- socket = cpu_to_socket(cpu);
- if ( test_bit(socket, cat_socket_enable) )
- return;
-
- cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
- if ( ebx & PSR_RESOURCE_TYPE_L3 )
- {
- cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
- info = cat_socket_info + socket;
- info->cbm_len = (eax & 0x1f) + 1;
- info->cos_max = min(opt_cos_max, edx & 0xffff);
-
- info->cos_to_cbm = temp_cos_to_cbm;
- temp_cos_to_cbm = NULL;
- /* cos=0 is reserved as default cbm(all ones). */
- info->cos_to_cbm[0].cbm = (1ull << info->cbm_len) - 1;
-
- spin_lock_init(&info->cbm_lock);
-
- set_bit(socket, cat_socket_enable);
-
- if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
- cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
- {
- info->cos_to_cbm[0].code = (1ull << info->cbm_len) - 1;
- info->cos_to_cbm[0].data = (1ull << info->cbm_len) - 1;
-
- /* We only write mask1 since mask0 is always all ones by default. */
- wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->cbm_len) - 1);
-
- rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
- wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
-
- /* Cut half of cos_max when CDP is enabled. */
- info->cos_max >>= 1;
-
- set_bit(socket, cdp_socket_enable);
- }
- printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
- socket, info->cos_max, info->cbm_len,
- cdp_is_enabled(socket) ? "on" : "off");
- }
-}
-
-static void cat_cpu_fini(unsigned int cpu)
-{
- unsigned int socket = cpu_to_socket(cpu);
-
- if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
- {
- struct psr_cat_socket_info *info = cat_socket_info + socket;
-
- if ( info->cos_to_cbm )
- {
- xfree(info->cos_to_cbm);
- info->cos_to_cbm = NULL;
- }
-
- if ( cdp_is_enabled(socket) )
- clear_bit(socket, cdp_socket_enable);
-
- clear_bit(socket, cat_socket_enable);
- }
-}
-
-static void __init psr_cat_free(void)
-{
- xfree(cat_socket_enable);
- cat_socket_enable = NULL;
- xfree(cat_socket_info);
- cat_socket_info = NULL;
-}
-
-static void __init init_psr_cat(void)
-{
- if ( opt_cos_max < 1 )
- {
- printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
- return;
- }
-
- cat_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
- cat_socket_info = xzalloc_array(struct psr_cat_socket_info, nr_sockets);
- cdp_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
-
- if ( !cat_socket_enable || !cat_socket_info )
- psr_cat_free();
}
static int psr_cpu_prepare(unsigned int cpu)
{
- return cat_cpu_prepare(cpu);
+ return 0;
}
static void psr_cpu_init(void)
{
- if ( cat_socket_info )
- cat_cpu_init();
-
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
- if ( cat_socket_info )
- cat_cpu_fini(cpu);
+ return;
}
static int cpu_callback(
@@ -738,14 +282,10 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- if ( opt_psr & PSR_CAT )
- init_psr_cat();
-
- if ( psr_cpu_prepare(0) )
- psr_cat_free();
+ psr_cpu_prepare(0);
psr_cpu_init();
- if ( psr_cmt_enabled() || cat_socket_info )
+ if ( psr_cmt_enabled() )
register_cpu_notifier(&cpu_nfb);
return 0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 03/24] x86: refactor psr: implement main data structures.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-02-08 8:15 ` [PATCH v6 01/24] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-02-08 8:15 ` [PATCH v6 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
@ 2017-02-08 8:15 ` Yi Sun
2017-02-08 16:02 ` Konrad Rzeszutek Wilk
2017-02-09 11:05 ` Wei Liu
2017-02-08 8:15 ` [PATCH v6 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
` (20 subsequent siblings)
23 siblings, 2 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:15 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
To construct an extendible framework, we need analyze PSR features
and abstract the common things and feature specific things. Then,
encapsulate them into different data structures.
By analyzing PSR features, we can get below map.
+------+------+------+
--------->| Dom0 | Dom1 | ... |
| +------+------+------+
| |
|Dom ID | cos_id of domain
| V
| +-----------------------------------------------------------------------------+
User --------->| PSR |
Socket ID | +--------------+---------------+---------------+ |
| | Socket0 Info | Socket 1 Info | ... | |
| +--------------+---------------+---------------+ |
| | cos_id=0 cos_id=1 ... |
| | +-----------------------+-----------------------+-----------+ |
| |->Ref : | ref 0 | ref 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L3 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------------------+-----------------------+-----------+ |
| |->L2 CAT: | cos 0 | cos 1 | ... | |
| | +-----------------------+-----------------------+-----------+ |
| | +-----------+-----------+-----------+-----------+-----------+ |
| |->CDP : | cos0 code | cos0 data | cos1 code | cos1 data | ... | |
| +-----------+-----------+-----------+-----------+-----------+ |
+-----------------------------------------------------------------------------+
So, we need define a socket info data structure, 'struct
psr_socket_info' to manage information per socket. It contains a
reference count array according to COS ID and a feature list to
manage all features enabled. Every entry of the reference count
array is used to record how many domains are using the COS registers
according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
Dom1 uses COS_ID=1 registers of both features to save CBM values, like
below.
+-------+-------+-------+-----+
| COS 0 | COS 1 | COS 2 | ... |
+-------+-------+-------+-----+
L3 CAT | 0x7ff | 0x1ff | ... | ... |
+-------+-------+-------+-----+
L2 CAT | 0xff | 0xff | ... | ... |
+-------+-------+-------+-----+
If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
That means, both Dom1 and Dom2 use same COS registers(ID=1) to save same
L3/L2 values. So, the value ref[1] is 2 which means 2 domains are using
COS_ID 1.
To manage a feature, we need define a feature node data structure,
'struct feat_node', to manage feature's specific HW info, its callback
functions (all feature's specific behaviors are encapsulated into these
callback functions), and an array of all COS registers values of this
feature.
CDP is a special feature which uses two entries of the array
for one COS ID. So, the number of CDP COS registers is the half of L3
CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
it is enabled. CDP uses the COS registers array as below.
+-----------+-----------+-----------+-----------+-----------+
CDP cos_reg_val[] index: | 0 | 1 | 2 | 3 | ... |
+-----------+-----------+-----------+-----------+-----------+
value: | cos0 code | cos0 data | cos1 code | cos1 data | ... |
+-----------+-----------+-----------+-----------+-----------+
For more details, please refer SDM and patches to implement 'get value' and
'set value'.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v6:
- modify commit message to point out patches to refer to understand details.
- fix wording.
- add comments to explain 'feat_mask'.
---
xen/arch/x86/psr.c | 106 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 106 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 96a8589..4656936 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -17,12 +17,118 @@
#include <xen/cpu.h>
#include <xen/err.h>
#include <xen/sched.h>
+#include <xen/list.h>
#include <asm/psr.h>
+/*
+ * Terminology:
+ * - CAT Cache Allocation Technology
+ * - CBM Capacity BitMasks
+ * - CDP Code and Data Prioritization
+ * - COS/CLOS Class of Service. Also mean COS registers.
+ * - COS_MAX Max number of COS for the feature (minus 1)
+ * - MSRs Machine Specific Registers
+ * - PSR Intel Platform Shared Resource
+ */
+
#define PSR_CMT (1<<0)
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
+/*
+ * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
+ * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
+ * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
+ *
+ * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for
+ * up to 64 L2 CAT COS. The COS_ID=[0,63].
+ *
+ * So, the maximum COS register count of one feature is 128.
+ */
+#define MAX_COS_REG_CNT 128
+
+/*
+ * PSR features are managed per socket. Below structure defines the members
+ * used to manage these features.
+ * feat_mask - Mask used to record features enabled on socket. There may be
+ * some features enabled at same time.
+ * nr_feat - Record how many features enabled.
+ * feat_list - A list used to manage all features enabled.
+ * cos_ref - A reference count array to record how many domains are using the
+ * COS_ID.
+ * Every entry of cos_ref corresponds to one COS ID.
+ * ref_lock - A lock to protect cos_ref.
+ */
+struct psr_socket_info {
+ /*
+ * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
+ * psr_feat_type' means the bit position.
+ * bit 0: L3 CAT
+ * bit 1: L3 CDP
+ * bit 2: L2 CAT
+ */
+ unsigned int feat_mask;
+ unsigned int nr_feat;
+ struct list_head feat_list;
+ unsigned int cos_ref[MAX_COS_REG_CNT];
+ spinlock_t ref_lock;
+};
+
+enum psr_feat_type {
+ PSR_SOCKET_L3_CAT = 0,
+ PSR_SOCKET_L3_CDP,
+ PSR_SOCKET_L2_CAT,
+};
+
+/* CAT/CDP HW info data structure. */
+struct psr_cat_hw_info {
+ unsigned int cbm_len;
+ unsigned int cos_max;
+};
+
+/* Encapsulate feature specific HW info here. */
+struct feat_hw_info {
+ union {
+ struct psr_cat_hw_info l3_cat_info;
+ };
+};
+
+struct feat_node;
+
+/*
+ * This structure defines feature operation callback functions. Every feature
+ * enabled MUST implement such callback functions and register them to ops.
+ *
+ * Feature specific behaviors will be encapsulated into these callback
+ * functions. Then, the main flows will not be changed when introducing a new
+ * feature.
+ */
+struct feat_ops {
+ /* get_cos_max is used to get feature's cos_max. */
+ unsigned int (*get_cos_max)(const struct feat_node *feat);
+};
+
+/*
+ * This structure represents one feature.
+ * feature - Which feature it is.
+ * feat_ops - Feature operation callback functions.
+ * info - Feature HW info.
+ * cos_reg_val - Array to store the values of COS registers. One entry stores
+ * the value of one COS register.
+ * For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
+ * For CDP, two entries correspond to one COS_ID. E.g.
+ * COS_ID=0 corresponds to cos_reg_val[0] (Data) and
+ * cos_reg_val[1] (Code).
+ * list - Feature list.
+ */
+struct feat_node {
+ enum psr_feat_type feature;
+ struct feat_ops ops;
+ struct feat_hw_info info;
+ uint64_t cos_reg_val[MAX_COS_REG_CNT];
+ struct list_head list;
+};
+
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v6 03/24] x86: refactor psr: implement main data structures.
2017-02-08 8:15 ` [PATCH v6 03/24] x86: refactor psr: implement main data structures Yi Sun
@ 2017-02-08 16:02 ` Konrad Rzeszutek Wilk
2017-02-09 11:05 ` Wei Liu
1 sibling, 0 replies; 37+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-08 16:02 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v6 03/24] x86: refactor psr: implement main data structures.
2017-02-08 8:15 ` [PATCH v6 03/24] x86: refactor psr: implement main data structures Yi Sun
2017-02-08 16:02 ` Konrad Rzeszutek Wilk
@ 2017-02-09 11:05 ` Wei Liu
1 sibling, 0 replies; 37+ messages in thread
From: Wei Liu @ 2017-02-09 11:05 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 08, 2017 at 04:15:55PM +0800, Yi Sun wrote:
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 96a8589..4656936 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -17,12 +17,118 @@
> #include <xen/cpu.h>
> #include <xen/err.h>
> #include <xen/sched.h>
> +#include <xen/list.h>
Please move the inclusion before sched.h -- they should be sorted
alphabetically.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v6 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (2 preceding siblings ...)
2017-02-08 8:15 ` [PATCH v6 03/24] x86: refactor psr: implement main data structures Yi Sun
@ 2017-02-08 8:15 ` Yi Sun
2017-02-08 16:34 ` Konrad Rzeszutek Wilk
2017-02-08 8:15 ` [PATCH v6 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
` (19 subsequent siblings)
23 siblings, 1 reply; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:15 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the CPU init and free flow including L3 CAT
initialization and feature list free.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v6:
- use 'struct cpuid_leaf' in x86_emulate.h.
- move cpuid_{,count}_leaf() helpers from cpuid.c to processor.h for
external user and pass the compilation.
- fix comments to make them clearer.
- use __clear_bit().
- call ASSERT() before __set_bit().
- adjust printk() position.
---
xen/arch/x86/cpuid.c | 6 --
xen/arch/x86/psr.c | 177 +++++++++++++++++++++++++++++++++++++++-
xen/include/asm-x86/processor.h | 7 ++
3 files changed, 182 insertions(+), 8 deletions(-)
diff --git a/xen/arch/x86/cpuid.c b/xen/arch/x86/cpuid.c
index e0a387e..e3e92dd 100644
--- a/xen/arch/x86/cpuid.c
+++ b/xen/arch/x86/cpuid.c
@@ -34,12 +34,6 @@ static void cpuid_leaf(uint32_t leaf, struct cpuid_leaf *data)
cpuid(leaf, &data->a, &data->b, &data->c, &data->d);
}
-static void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
- struct cpuid_leaf *data)
-{
- cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
-}
-
static void sanitise_featureset(uint32_t *fs)
{
/* for_each_set_bit() uses unsigned longs. Extend with zeroes. */
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 4656936..9496c97 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -19,6 +19,7 @@
#include <xen/sched.h>
#include <xen/list.h>
#include <asm/psr.h>
+#include <asm/x86_emulate.h>
/*
* Terminology:
@@ -35,6 +36,9 @@
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
+#define CAT_CBM_LEN_MASK 0x1f
+#define CAT_COS_MAX_MASK 0xffff
+
/*
* Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
* the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
@@ -136,11 +140,84 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
+static struct psr_socket_info *__read_mostly socket_info;
+
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
+static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
+/*
+ * Declare global feature list entry for every feature to facilitate the
+ * feature list creation. It will be allocated in psr_cpu_prepare() and
+ * inserted into feature list in cpu_init_work(). It is protected by
+ * cpu_add_remove_lock spinlock.
+ */
+static struct feat_node *feat_l3_cat;
+
+/* Common functions. */
+static void free_feature(struct psr_socket_info *info)
+{
+ struct feat_node *feat, *next;
+
+ if ( !info )
+ return;
+
+ /*
+ * Free resources of features. But we do not free global feature list
+ * entry, like feat_l3_cat. Although it may cause a few memory leak,
+ * it is OK simplify things.
+ */
+ list_for_each_entry_safe(feat, next, &info->feat_list, list)
+ {
+ __clear_bit(feat->feature, &info->feat_mask);
+ list_del(&feat->list);
+ xfree(feat);
+ }
+}
+
+/* L3 CAT functions implementation. */
+static void l3_cat_init_feature(struct cpuid_leaf regs,
+ struct feat_node *feat,
+ struct psr_socket_info *info)
+{
+ struct psr_cat_hw_info l3_cat;
+ unsigned int socket;
+
+ /* No valid value so do not enable feature. */
+ if ( !regs.a || !regs.b )
+ return;
+
+ l3_cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
+ l3_cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
+
+ /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+ feat->cos_reg_val[0] = (1ull << l3_cat.cbm_len) - 1;
+
+ feat->feature = PSR_SOCKET_L3_CAT;
+ ASSERT(!test_bit(PSR_SOCKET_L3_CAT, &info->feat_mask));
+ __set_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
+
+ feat->info.l3_cat_info = l3_cat;
+
+ info->nr_feat++;
+
+ /* Add this feature into list. */
+ list_add_tail(&feat->list, &info->feat_list);
+
+ socket = cpu_to_socket(smp_processor_id());
+ if ( !opt_cpu_info )
+ return;
+
+ printk(XENLOG_INFO "L3 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ socket, feat->info.l3_cat_info.cos_max,
+ feat->info.l3_cat_info.cbm_len);
+}
+
+static const struct feat_ops l3_cat_ops = {
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -180,6 +257,9 @@ static void __init parse_psr_param(char *s)
if ( val_str && !strcmp(s, "rmid_max") )
opt_rmid_max = simple_strtoul(val_str, NULL, 0);
+ if ( val_str && !strcmp(s, "cos_max") )
+ opt_cos_max = simple_strtoul(val_str, NULL, 0);
+
s = ss + 1;
} while ( ss );
}
@@ -335,18 +415,107 @@ void psr_domain_free(struct domain *d)
psr_free_rmid(d);
}
+static void cpu_init_work(void)
+{
+ struct psr_socket_info *info;
+ unsigned int socket;
+ unsigned int cpu = smp_processor_id();
+ struct feat_node *feat;
+ struct cpuid_leaf regs = {.a = 0, .b = 0, .c = 0, .d = 0};
+
+ if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
+ return;
+ else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
+ {
+ __clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
+ return;
+ }
+
+ socket = cpu_to_socket(cpu);
+ info = socket_info + socket;
+ if ( info->feat_mask )
+ return;
+
+ INIT_LIST_HEAD(&info->feat_list);
+ spin_lock_init(&info->ref_lock);
+
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L3 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
+
+ feat = feat_l3_cat;
+ /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
+ feat_l3_cat = NULL;
+ feat->ops = l3_cat_ops;
+
+ l3_cat_init_feature(regs, feat, info);
+ }
+}
+
+static void cpu_fini_work(unsigned int cpu)
+{
+ unsigned int socket = cpu_to_socket(cpu);
+
+ if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
+ {
+ free_feature(socket_info + socket);
+ }
+}
+
+static void __init init_psr(void)
+{
+ if ( opt_cos_max < 1 )
+ {
+ printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
+ return;
+ }
+
+ socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
+
+ if ( !socket_info )
+ {
+ printk(XENLOG_INFO "Failed to alloc socket_info!\n");
+ return;
+ }
+}
+
+static void __init psr_free(void)
+{
+ unsigned int i;
+
+ for ( i = 0; i < nr_sockets; i++ )
+ free_feature(&socket_info[i]);
+
+ xfree(socket_info);
+ socket_info = NULL;
+}
+
static int psr_cpu_prepare(unsigned int cpu)
{
+ if ( !socket_info )
+ return 0;
+
+ /* Malloc memory for the global feature head here. */
+ if ( feat_l3_cat == NULL &&
+ (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
+
return 0;
}
static void psr_cpu_init(void)
{
+ if ( socket_info )
+ cpu_init_work();
+
psr_assoc_init();
}
static void psr_cpu_fini(unsigned int cpu)
{
+ if ( socket_info )
+ cpu_fini_work(cpu);
return;
}
@@ -388,10 +557,14 @@ static int __init psr_presmp_init(void)
if ( (opt_psr & PSR_CMT) && opt_rmid_max )
init_psr_cmt(opt_rmid_max);
- psr_cpu_prepare(0);
+ if ( opt_psr & PSR_CAT )
+ init_psr();
+
+ if ( psr_cpu_prepare(0) )
+ psr_free();
psr_cpu_init();
- if ( psr_cmt_enabled() )
+ if ( psr_cmt_enabled() || socket_info )
register_cpu_notifier(&cpu_nfb);
return 0;
diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h
index 7735bc2..eb0a1e9 100644
--- a/xen/include/asm-x86/processor.h
+++ b/xen/include/asm-x86/processor.h
@@ -14,6 +14,7 @@
#include <asm/types.h>
#include <asm/cpufeature.h>
#include <asm/desc.h>
+#include <asm/x86_emulate.h>
#endif
#include <asm/x86-defns.h>
@@ -259,6 +260,12 @@ static always_inline unsigned int cpuid_count_ebx(
return ebx;
}
+static always_inline void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
+ struct cpuid_leaf *data)
+{
+ cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
+}
+
static inline unsigned long read_cr0(void)
{
unsigned long cr0;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v6 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-08 8:15 ` [PATCH v6 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
@ 2017-02-08 16:34 ` Konrad Rzeszutek Wilk
2017-02-08 19:03 ` Konrad Rzeszutek Wilk
2017-02-09 11:00 ` Yi Sun
0 siblings, 2 replies; 37+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-08 16:34 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
.snip..
> +static void free_feature(struct psr_socket_info *info)
> +{
> + struct feat_node *feat, *next;
> +
> + if ( !info )
> + return;
> +
> + /*
> + * Free resources of features. But we do not free global feature list
> + * entry, like feat_l3_cat. Although it may cause a few memory leak,
> + * it is OK simplify things.
it is OK as it simplifies things.
> + */
> + list_for_each_entry_safe(feat, next, &info->feat_list, list)
> + {
> + __clear_bit(feat->feature, &info->feat_mask);
> + list_del(&feat->list);
> + xfree(feat);
> + }
> +}
> +
> +/* L3 CAT functions implementation. */
> +static void l3_cat_init_feature(struct cpuid_leaf regs,
> + struct feat_node *feat,
> + struct psr_socket_info *info)
> +{
> + struct psr_cat_hw_info l3_cat;
Having experienced a few times myself forgetting to set
_all_ the entries a structure allocated on the stack and then
debugging for hours - perhaps you could do:
l3_cat = { };
Which forces the compiler to initialize _all_ the values to zero.
> + unsigned int socket;
> +
> + /* No valid value so do not enable feature. */
> + if ( !regs.a || !regs.b )
You use regs.d below. Would it make sense to check for that value as
well?
Or is a value of 0 for cox_max OK? I would think so, but not exactly
sure.
> + return;
> +
> + l3_cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
> + l3_cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
> +
> + /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> + feat->cos_reg_val[0] = (1ull << l3_cat.cbm_len) - 1;
> +
> + feat->feature = PSR_SOCKET_L3_CAT;
> + ASSERT(!test_bit(PSR_SOCKET_L3_CAT, &info->feat_mask));
> + __set_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
> +
> + feat->info.l3_cat_info = l3_cat;
> +
> + info->nr_feat++;
> +
> + /* Add this feature into list. */
> + list_add_tail(&feat->list, &info->feat_list);
> +
> + socket = cpu_to_socket(smp_processor_id());
> + if ( !opt_cpu_info )
> + return;
> +
> + printk(XENLOG_INFO "L3 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
> + socket, feat->info.l3_cat_info.cos_max,
> + feat->info.l3_cat_info.cbm_len);
> +}
> +
> +static const struct feat_ops l3_cat_ops = {
> +};
> +
> static void __init parse_psr_bool(char *s, char *value, char *feature,
> unsigned int mask)
> {
> @@ -180,6 +257,9 @@ static void __init parse_psr_param(char *s)
> if ( val_str && !strcmp(s, "rmid_max") )
> opt_rmid_max = simple_strtoul(val_str, NULL, 0);
>
> + if ( val_str && !strcmp(s, "cos_max") )
> + opt_cos_max = simple_strtoul(val_str, NULL, 0);
> +
> s = ss + 1;
> } while ( ss );
> }
> @@ -335,18 +415,107 @@ void psr_domain_free(struct domain *d)
> psr_free_rmid(d);
> }
>
> +static void cpu_init_work(void)
> +{
> + struct psr_socket_info *info;
> + unsigned int socket;
> + unsigned int cpu = smp_processor_id();
> + struct feat_node *feat;
> + struct cpuid_leaf regs = {.a = 0, .b = 0, .c = 0, .d = 0};
> +
> + if ( !cpu_has(¤t_cpu_data, X86_FEATURE_PQE) )
> + return;
> + else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
> + {
> + __clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
> + return;
> + }
> +
> + socket = cpu_to_socket(cpu);
> + info = socket_info + socket;
> + if ( info->feat_mask )
> + return;
> +
> + INIT_LIST_HEAD(&info->feat_list);
> + spin_lock_init(&info->ref_lock);
> +
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
> + if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> + {
> + cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
> +
> + feat = feat_l3_cat;
> + /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
> + feat_l3_cat = NULL;
> + feat->ops = l3_cat_ops;
> +
> + l3_cat_init_feature(regs, feat, info);
> + }
> +}
> +
> +static void cpu_fini_work(unsigned int cpu)
> +{
> + unsigned int socket = cpu_to_socket(cpu);
> +
> + if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
I fear I don't understand this.
Looking at 65a399ac it says:
+ .notifier_call = cpu_callback,
+ /*
+ * Ensure socket_cpumask is still valid in CPU_DEAD notification
+ * (E.g. our CPU_DEAD notification should be called ahead of
+ * cpu_smpboot_free).
Which means that socket_cpumask[socket] should have an value.
In fact cpumask_any(socket_cpumask[socket]) will return true at this
point.
Which means that code above gets called if this psr_cpu_fini is called
_after_ cpu_smpboot_free (b/c socket_cpumask[socket] will be NULL), see
cpu_smpboot_free.
But with .priority being -1 that won't happen.
But lets ignore that, lets say it does get called _after_
cpu_smpboot_free. In which case the first part of the 'if' condition
is true and we end up doing: cpumask_empty(NULL) which will result
in an page fault.
I think this is a bug.
Perhaps it should just be:
/* .priority is = -1 and we MUST run before cpu_smpboot_free. */
ASSERT(socket_cpumask[socket] && cpumask_any(socket_cpumask[socket]));
But since that is only compiled on debug=y it may be better to just
change that ASSERT to an 'if' with the 'else' hitting an ASSERT(1)?
Like so:
if ( socket_cpumask[socket] && cpumask_any(socket_cpumask[socket] )
free_feature(socket_info + socket);
else
ASSERT(0);
?
But maybe I am reading the code wrong?
> + {
> + free_feature(socket_info + socket);
> + }
> +}
> +
> +static void __init init_psr(void)
Why don't we make this return an value? And then the init
code(psr_presmp_init) can just return an error?
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v6 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-08 16:34 ` Konrad Rzeszutek Wilk
@ 2017-02-08 19:03 ` Konrad Rzeszutek Wilk
2017-02-09 11:10 ` Yi Sun
2017-02-09 11:00 ` Yi Sun
1 sibling, 1 reply; 37+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-08 19:03 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
> > +static void cpu_fini_work(unsigned int cpu)
> > +{
> > + unsigned int socket = cpu_to_socket(cpu);
> > +
> > + if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
>
> I fear I don't understand this.
>
> Looking at 65a399ac it says:
>
> + .notifier_call = cpu_callback,
> + /*
> + * Ensure socket_cpumask is still valid in CPU_DEAD notification
> + * (E.g. our CPU_DEAD notification should be called ahead of
> + * cpu_smpboot_free).
>
> Which means that socket_cpumask[socket] should have an value.
>
> In fact cpumask_any(socket_cpumask[socket]) will return true at this
> point.
>
> Which means that code above gets called if this psr_cpu_fini is called
> _after_ cpu_smpboot_free (b/c socket_cpumask[socket] will be NULL), see
> cpu_smpboot_free.
>
> But with .priority being -1 that won't happen.
>
> But lets ignore that, lets say it does get called _after_
> cpu_smpboot_free. In which case the first part of the 'if' condition
> is true and we end up doing: cpumask_empty(NULL) which will result
> in an page fault.
>
> I think this is a bug.
I missed the fact that we call this function on CPU_UP_CANCELED
_and_ CPU_DEAD and that remove_siblinginfo is called before CPU_DEAD.
So say the socket is being onlined and we are the first CPU on
that (CPU=2).
1) CPU_UP_PREPARE notifier calls psr_cpu_prepare which allocates
'feat_l3_cat'.
2). Some other notifies dies and we end up calling CPU_UP_CANCELED.
psr_cpu_fini -> cpu_fini_work:
Since __cpu_up has not been called that means
socket_cpumask[socket] == NULL
and we enter free_feature.
It does 'if !(socket_info + socket)' effectively. And that is false
as init_psr was the one initializing that array which means the
pointer (info aka socket_info + socket) certainly is not NULL.
Anyhow this means free_feature tries to walk the list - but 'info'
is full of zeros.
Which means list_for_each_entry_safe is going to blow when trying
to set 'n' (as pos is zero aka NULL).
Perhaps you want an INIT_LIST_HEAD in 'init_psr' so that free_feature
can be idempotent?
Or perhaps you need '&&' in the if conditional (and naturally
remove the !)?
I wonder what other CPU notifiers are guilty of this..
Lets try another example:
1) CPU_UP_PREPARE notifier calls psr_cpu_prepare which allocates
'feat_l3_cat'.
2) All notifiers were OK, the __cpu_up has been called. The
socket_cpumask[socket] = <some data>
3) All notifiers are called with CPU_ONLINE.
All good.
..some time later ..
4). CPU is brought down. Notifiers for CPU_DOWN_PREPARE are called
[psr is not part of it]
5) CPU notifiers CPU_DEAD are called. They all have to return
NOTIFY_DONE otherwise we hit 'BUG_ON'
We call psr_cpu_fini -> cpu_fini_work
socket_cpumask[socket] has some data, so the first conditional
is false:
( !socket_cpumask[socket] ||
the second:
cpumask_empty(socket_cpumask[socket]) )
is true as take_cpu_down -> __cpu_disable -> remove_siblinginfo
was called before us.(and this is the first CPU in the socket).
And we call free_feature.
which is correct and we free our data.
And the 'cpumask_empty' guards us so that we only call this
on the very last CPU.
And we would still call this if the conditional was:
if ( socket_cpumask[socket] && cpumask_empty(socket_cpumask[socket]))
I think?
Feel free to poke holes at my analysis - I am sure I missed some edge
case.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v6 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-08 19:03 ` Konrad Rzeszutek Wilk
@ 2017-02-09 11:10 ` Yi Sun
0 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-09 11:10 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-08 14:03:15, Konrad Rzeszutek Wilk wrote:
> > > +static void cpu_fini_work(unsigned int cpu)
> > > +{
> > > + unsigned int socket = cpu_to_socket(cpu);
> > > +
> > > + if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
> >
> > I fear I don't understand this.
> >
> > Looking at 65a399ac it says:
> >
> > + .notifier_call = cpu_callback,
> > + /*
> > + * Ensure socket_cpumask is still valid in CPU_DEAD notification
> > + * (E.g. our CPU_DEAD notification should be called ahead of
> > + * cpu_smpboot_free).
> >
> > Which means that socket_cpumask[socket] should have an value.
> >
> > In fact cpumask_any(socket_cpumask[socket]) will return true at this
> > point.
> >
> > Which means that code above gets called if this psr_cpu_fini is called
> > _after_ cpu_smpboot_free (b/c socket_cpumask[socket] will be NULL), see
> > cpu_smpboot_free.
> >
> > But with .priority being -1 that won't happen.
> >
> > But lets ignore that, lets say it does get called _after_
> > cpu_smpboot_free. In which case the first part of the 'if' condition
> > is true and we end up doing: cpumask_empty(NULL) which will result
> > in an page fault.
> >
> > I think this is a bug.
>
> I missed the fact that we call this function on CPU_UP_CANCELED
> _and_ CPU_DEAD and that remove_siblinginfo is called before CPU_DEAD.
>
> So say the socket is being onlined and we are the first CPU on
> that (CPU=2).
>
>
> 1) CPU_UP_PREPARE notifier calls psr_cpu_prepare which allocates
> 'feat_l3_cat'.
>
> 2). Some other notifies dies and we end up calling CPU_UP_CANCELED.
> psr_cpu_fini -> cpu_fini_work:
>
> Since __cpu_up has not been called that means
> socket_cpumask[socket] == NULL
>
> and we enter free_feature.
>
> It does 'if !(socket_info + socket)' effectively. And that is false
> as init_psr was the one initializing that array which means the
> pointer (info aka socket_info + socket) certainly is not NULL.
>
> Anyhow this means free_feature tries to walk the list - but 'info'
> is full of zeros.
>
> Which means list_for_each_entry_safe is going to blow when trying
> to set 'n' (as pos is zero aka NULL).
>
> Perhaps you want an INIT_LIST_HEAD in 'init_psr' so that free_feature
> can be idempotent?
>
> Or perhaps you need '&&' in the if conditional (and naturally
> remove the !)?
>
> I wonder what other CPU notifiers are guilty of this..
>
You are right. I made a mistake here. Thanks a lot for finding out the issue!
In previous version, I called INIT_LIST_HEAD in init_psr. But I moved it into
cpu_init_work to place it together with spin_lock_init per review comments.
This causes this mis-match issue. I think your suggestion is good to change
check criteria as below.
if ( socket_cpumask[socket] && cpumask_empty(socket_cpumask[socket]) )
Then, free_feature will only be executed when CPU_STARTING has been done.
>
> Lets try another example:
>
> 1) CPU_UP_PREPARE notifier calls psr_cpu_prepare which allocates
> 'feat_l3_cat'.
>
> 2) All notifiers were OK, the __cpu_up has been called. The
> socket_cpumask[socket] = <some data>
>
> 3) All notifiers are called with CPU_ONLINE.
> All good.
>
>
> ..some time later ..
>
> 4). CPU is brought down. Notifiers for CPU_DOWN_PREPARE are called
> [psr is not part of it]
>
> 5) CPU notifiers CPU_DEAD are called. They all have to return
> NOTIFY_DONE otherwise we hit 'BUG_ON'
>
> We call psr_cpu_fini -> cpu_fini_work
>
> socket_cpumask[socket] has some data, so the first conditional
> is false:
>
> ( !socket_cpumask[socket] ||
>
> the second:
>
> cpumask_empty(socket_cpumask[socket]) )
>
> is true as take_cpu_down -> __cpu_disable -> remove_siblinginfo
>
> was called before us.(and this is the first CPU in the socket).
>
> And we call free_feature.
> which is correct and we free our data.
>
> And the 'cpumask_empty' guards us so that we only call this
> on the very last CPU.
>
>
> And we would still call this if the conditional was:
>
> if ( socket_cpumask[socket] && cpumask_empty(socket_cpumask[socket]))
>
> I think?
>
> Feel free to poke holes at my analysis - I am sure I missed some edge
> case.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v6 04/24] x86: refactor psr: implement CPU init and free flow.
2017-02-08 16:34 ` Konrad Rzeszutek Wilk
2017-02-08 19:03 ` Konrad Rzeszutek Wilk
@ 2017-02-09 11:00 ` Yi Sun
1 sibling, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-09 11:00 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On 17-02-08 11:34:16, Konrad Rzeszutek Wilk wrote:
> > + /* No valid value so do not enable feature. */
> > + if ( !regs.a || !regs.b )
>
> You use regs.d below. Would it make sense to check for that value as
> well?
>
> Or is a value of 0 for cox_max OK? I would think so, but not exactly
> sure.
>
Sorry, this is a typo.
> > + {
> > + free_feature(socket_info + socket);
> > + }
> > +}
> > +
> > +static void __init init_psr(void)
>
> Why don't we make this return an value? And then the init
> code(psr_presmp_init) can just return an error?
Because we need init CMT related things besides CAT in psr_presmp_init. So no
matter init_psr return value, we need call psr_cpu_init to call psr_assoc_init.
We will check if socket_info in following functions called to avoid error.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v6 05/24] x86: refactor psr: implement Domain init/free and schedule flows.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (3 preceding siblings ...)
2017-02-08 8:15 ` [PATCH v6 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
@ 2017-02-08 8:15 ` Yi Sun
2017-02-08 16:43 ` Konrad Rzeszutek Wilk
2017-02-08 8:15 ` [PATCH v6 06/24] x86: refactor psr: implement get hw info flow Yi Sun
` (18 subsequent siblings)
23 siblings, 1 reply; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:15 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the Domain init/free and schedule flows.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v6:
- change 'PSR_ASSOC_REG_POS' to 'PSR_ASSOC_REG_SHIFT'.
---
xen/arch/x86/psr.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 61 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 9496c97..89fbaaa 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -51,6 +51,8 @@
*/
#define MAX_COS_REG_CNT 128
+#define PSR_ASSOC_REG_SHIFT 32
+
/*
* PSR features are managed per socket. Below structure defines the members
* used to manage these features.
@@ -215,7 +217,13 @@ static void l3_cat_init_feature(struct cpuid_leaf regs,
feat->info.l3_cat_info.cbm_len);
}
+static unsigned int l3_cat_get_cos_max(const struct feat_node *feat)
+{
+ return feat->info.l3_cat_info.cos_max;
+}
+
static const struct feat_ops l3_cat_ops = {
+ .get_cos_max = l3_cat_get_cos_max,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -359,11 +367,33 @@ void psr_free_rmid(struct domain *d)
d->arch.psr_rmid = 0;
}
+static inline unsigned int get_max_cos_max(const struct psr_socket_info *info)
+{
+ const struct feat_node *feat;
+ unsigned int cos_max = 0;
+
+ list_for_each_entry(feat, &info->feat_list, list)
+ cos_max = max(feat->ops.get_cos_max(feat), cos_max);
+
+ return cos_max;
+}
+
static inline void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
- if ( psr_cmt_enabled() )
+ if ( socket_info )
+ {
+ unsigned int socket = cpu_to_socket(smp_processor_id());
+ const struct psr_socket_info *info = socket_info + socket;
+ unsigned int cos_max = get_max_cos_max(info);
+
+ if ( info->feat_mask )
+ psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
+ PSR_ASSOC_REG_SHIFT;
+ }
+
+ if ( psr_cmt_enabled() || psra->cos_mask )
rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
}
@@ -372,6 +402,13 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
*reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
}
+static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
+ uint64_t cos_mask)
+{
+ *reg = (*reg & ~cos_mask) |
+ (((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
+}
+
void psr_ctxt_switch_to(struct domain *d)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -380,6 +417,11 @@ void psr_ctxt_switch_to(struct domain *d)
if ( psr_cmt_enabled() )
psr_assoc_rmid(®, d->arch.psr_rmid);
+ if ( psra->cos_mask )
+ psr_assoc_cos(®, d->arch.psr_cos_ids ?
+ d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
+ 0, psra->cos_mask);
+
if ( reg != psra->val )
{
wrmsrl(MSR_IA32_PSR_ASSOC, reg);
@@ -405,14 +447,32 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
return 0;
}
+/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
+static void psr_free_cos(struct domain *d)
+{
+ if( !d->arch.psr_cos_ids )
+ return;
+
+ xfree(d->arch.psr_cos_ids);
+ d->arch.psr_cos_ids = NULL;
+}
+
int psr_domain_init(struct domain *d)
{
+ if ( socket_info )
+ {
+ d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
+ if ( !d->arch.psr_cos_ids )
+ return -ENOMEM;
+ }
+
return 0;
}
void psr_domain_free(struct domain *d)
{
psr_free_rmid(d);
+ psr_free_cos(d);
}
static void cpu_init_work(void)
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v6 05/24] x86: refactor psr: implement Domain init/free and schedule flows.
2017-02-08 8:15 ` [PATCH v6 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
@ 2017-02-08 16:43 ` Konrad Rzeszutek Wilk
0 siblings, 0 replies; 37+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-08 16:43 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 08, 2017 at 04:15:57PM +0800, Yi Sun wrote:
> This patch implements the Domain init/free and schedule flows.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v6 06/24] x86: refactor psr: implement get hw info flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (4 preceding siblings ...)
2017-02-08 8:15 ` [PATCH v6 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
@ 2017-02-08 8:15 ` Yi Sun
2017-02-10 21:34 ` Konrad Rzeszutek Wilk
2017-02-08 8:15 ` [PATCH v6 07/24] x86: refactor psr: implement get value flow Yi Sun
` (17 subsequent siblings)
23 siblings, 1 reply; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:15 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements get HW info flow including L3 CAT callback
function.
It also changes sysctl interface to make it more general.
With this patch, 'psr-hwinfo' can work for L3 CAT.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v6:
- adjust '{' position for switch.
- define 'PSR_SOCKET_UNKNOWN = 0xFFFF' in 'enum psr_feat_type'.
- use 'ERANGE' when socket id exceeds the range.
- adjust value postion of macros.
---
xen/arch/x86/psr.c | 75 +++++++++++++++++++++++++++++++++++++++++++++--
xen/arch/x86/sysctl.c | 14 +++++----
xen/include/asm-x86/psr.h | 19 +++++++-----
3 files changed, 93 insertions(+), 15 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 89fbaaa..d4a9003 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -84,6 +84,7 @@ enum psr_feat_type {
PSR_SOCKET_L3_CAT = 0,
PSR_SOCKET_L3_CDP,
PSR_SOCKET_L2_CAT,
+ PSR_SOCKET_UNKNOWN = 0xFFFF,
};
/* CAT/CDP HW info data structure. */
@@ -112,6 +113,9 @@ struct feat_node;
struct feat_ops {
/* get_cos_max is used to get feature's cos_max. */
unsigned int (*get_cos_max)(const struct feat_node *feat);
+ /* get_feat_info is used to get feature HW info. */
+ bool (*get_feat_info)(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len);
};
/*
@@ -179,6 +183,24 @@ static void free_feature(struct psr_socket_info *info)
}
}
+static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
+{
+ enum psr_feat_type feat_type;
+
+ /* Judge if feature is enabled. */
+ switch ( type )
+ {
+ case PSR_CBM_TYPE_L3:
+ feat_type = PSR_SOCKET_L3_CAT;
+ break;
+ default:
+ feat_type = PSR_SOCKET_UNKNOWN;
+ break;
+ }
+
+ return feat_type;
+}
+
/* L3 CAT functions implementation. */
static void l3_cat_init_feature(struct cpuid_leaf regs,
struct feat_node *feat,
@@ -222,8 +244,22 @@ static unsigned int l3_cat_get_cos_max(const struct feat_node *feat)
return feat->info.l3_cat_info.cos_max;
}
+static bool l3_cat_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], unsigned int array_len)
+{
+ if ( !data || 3 > array_len )
+ return false;
+
+ data[CBM_LEN] = feat->info.l3_cat_info.cbm_len;
+ data[COS_MAX] = feat->info.l3_cat_info.cos_max;
+ data[PSR_FLAG] = 0;
+
+ return true;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
+ .get_feat_info = l3_cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -429,10 +465,43 @@ void psr_ctxt_switch_to(struct domain *d)
}
}
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags)
+static struct psr_socket_info *get_socket_info(unsigned int socket)
{
- return 0;
+ if ( !socket_info )
+ return ERR_PTR(-ENODEV);
+
+ if ( socket >= nr_sockets )
+ return ERR_PTR(-ERANGE);
+
+ if ( !socket_info[socket].feat_mask )
+ return ERR_PTR(-ENOENT);
+
+ return socket_info + socket;
+}
+
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len)
+{
+ const struct psr_socket_info *info = get_socket_info(socket);
+ const struct feat_node *feat;
+ enum psr_feat_type feat_type;
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ if ( feat->feature != feat_type )
+ continue;
+
+ if ( feat->ops.get_feat_info(feat, data, array_len) )
+ return 0;
+ else
+ return -EINVAL;
+ }
+
+ return -ENOENT;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index b8c30d4..e340baa 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -176,15 +176,19 @@ long arch_do_sysctl(
switch ( sysctl->u.psr_cat_op.cmd )
{
case XEN_SYSCTL_PSR_CAT_get_l3_info:
- ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
- &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
- &sysctl->u.psr_cat_op.u.l3_info.cos_max,
- &sysctl->u.psr_cat_op.u.l3_info.flags);
+ {
+ uint32_t data[3];
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L3, data, 3);
+
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
ret = -EFAULT;
break;
-
+ }
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 57f47e9..0342a80 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -19,19 +19,24 @@
#include <xen/types.h>
/* CAT cpuid level */
-#define PSR_CPUID_LEVEL_CAT 0x10
+#define PSR_CPUID_LEVEL_CAT 0x10
/* Resource Type Enumeration */
-#define PSR_RESOURCE_TYPE_L3 0x2
+#define PSR_RESOURCE_TYPE_L3 0x2
/* L3 Monitoring Features */
-#define PSR_CMT_L3_OCCUPANCY 0x1
+#define PSR_CMT_L3_OCCUPANCY 0x1
/* CDP Capability */
-#define PSR_CAT_CDP_CAPABILITY (1u << 2)
+#define PSR_CAT_CDP_CAPABILITY (1u << 2)
/* L3 CDP Enable bit*/
-#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
+#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0
+
+/* Used by psr_get_info() */
+#define CBM_LEN 0
+#define COS_MAX 1
+#define PSR_FLAG 2
struct psr_cmt_l3 {
unsigned int features;
@@ -63,8 +68,8 @@ int psr_alloc_rmid(struct domain *d);
void psr_free_rmid(struct domain *d);
void psr_ctxt_switch_to(struct domain *d);
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags);
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len);
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v6 06/24] x86: refactor psr: implement get hw info flow.
2017-02-08 8:15 ` [PATCH v6 06/24] x86: refactor psr: implement get hw info flow Yi Sun
@ 2017-02-10 21:34 ` Konrad Rzeszutek Wilk
0 siblings, 0 replies; 37+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-10 21:34 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 08, 2017 at 04:15:58PM +0800, Yi Sun wrote:
> This patch implements get HW info flow including L3 CAT callback
> function.
>
> It also changes sysctl interface to make it more general.
>
> With this patch, 'psr-hwinfo' can work for L3 CAT.
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v6 07/24] x86: refactor psr: implement get value flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (5 preceding siblings ...)
2017-02-08 8:15 ` [PATCH v6 06/24] x86: refactor psr: implement get hw info flow Yi Sun
@ 2017-02-08 8:15 ` Yi Sun
2017-02-09 11:05 ` Wei Liu
2017-02-10 21:38 ` Konrad Rzeszutek Wilk
2017-02-08 8:16 ` [PATCH v6 08/24] x86: refactor psr: set value: implement framework Yi Sun
` (16 subsequent siblings)
23 siblings, 2 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:15 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements get value flow including L3 CAT callback
function.
It also changes domctl interface to make it more general.
With this patch, 'psr-cat-show' can work for L3 CAT but not for
L3 code/data which is implemented in patch "x86: refactor psr:
implement get value flow for CDP.".
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v6:
- modify commit message to make it clearer.
- remove one extra space in code.
- write a helper function to move get info and get val functions into
it. Because most codes of 'get_info' and 'get_val' are same.
---
xen/arch/x86/domctl.c | 18 +++++++++---------
xen/arch/x86/psr.c | 43 ++++++++++++++++++++++++++++++++++++++-----
xen/include/asm-x86/psr.h | 4 ++--
3 files changed, 49 insertions(+), 16 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 8e5259f..098e399 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1440,23 +1440,23 @@ long arch_do_domctl(
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3);
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_CODE);
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_DATA);
copyback = 1;
break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index d4a9003..0cb9971 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -116,6 +116,9 @@ struct feat_ops {
/* get_feat_info is used to get feature HW info. */
bool (*get_feat_info)(const struct feat_node *feat,
uint32_t data[], unsigned int array_len);
+ /* get_val is used to get feature COS register value. */
+ bool (*get_val)(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val);
};
/*
@@ -257,9 +260,22 @@ static bool l3_cat_get_feat_info(const struct feat_node *feat,
return true;
}
+static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val)
+{
+ if ( cos > feat->info.l3_cat_info.cos_max )
+ /* Use default value. */
+ cos = 0;
+
+ *val = feat->cos_reg_val[cos];
+
+ return true;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
+ .get_val = l3_cat_get_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -479,12 +495,14 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
return socket_info + socket;
}
-int psr_get_info(unsigned int socket, enum cbm_type type,
- uint32_t data[], unsigned int array_len)
+static int __psr_get(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len,
+ struct domain *d, uint64_t *val)
{
const struct psr_socket_info *info = get_socket_info(socket);
const struct feat_node *feat;
enum psr_feat_type feat_type;
+ unsigned int cos;
if ( IS_ERR(info) )
return PTR_ERR(info);
@@ -495,6 +513,15 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
if ( feat->feature != feat_type )
continue;
+ if ( d )
+ {
+ cos = d->arch.psr_cos_ids[socket];
+ if ( feat->ops.get_val(feat, cos, type, val) )
+ return 0;
+ else
+ break;
+ }
+
if ( feat->ops.get_feat_info(feat, data, array_len) )
return 0;
else
@@ -504,10 +531,16 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
return -ENOENT;
}
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type)
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t data[], unsigned int array_len)
+{
+ return __psr_get(socket, type, data, array_len, NULL, NULL);
+}
+
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint64_t *val, enum cbm_type type)
{
- return 0;
+ return __psr_get(socket, type, NULL, 0, d, val);
}
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 0342a80..569e7df 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -70,8 +70,8 @@ void psr_ctxt_switch_to(struct domain *d);
int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type);
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint64_t *val, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v6 07/24] x86: refactor psr: implement get value flow.
2017-02-08 8:15 ` [PATCH v6 07/24] x86: refactor psr: implement get value flow Yi Sun
@ 2017-02-09 11:05 ` Wei Liu
2017-02-10 21:38 ` Konrad Rzeszutek Wilk
1 sibling, 0 replies; 37+ messages in thread
From: Wei Liu @ 2017-02-09 11:05 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 08, 2017 at 04:15:59PM +0800, Yi Sun wrote:
> static void __init parse_psr_bool(char *s, char *value, char *feature,
> @@ -479,12 +495,14 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
> return socket_info + socket;
> }
>
> -int psr_get_info(unsigned int socket, enum cbm_type type,
> - uint32_t data[], unsigned int array_len)
> +static int __psr_get(unsigned int socket, enum cbm_type type,
> + uint32_t data[], unsigned int array_len,
> + struct domain *d, uint64_t *val)
Double underscore is part of the reserved namespace. Calling this
function psr_get should be fine.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v6 07/24] x86: refactor psr: implement get value flow.
2017-02-08 8:15 ` [PATCH v6 07/24] x86: refactor psr: implement get value flow Yi Sun
2017-02-09 11:05 ` Wei Liu
@ 2017-02-10 21:38 ` Konrad Rzeszutek Wilk
1 sibling, 0 replies; 37+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-10 21:38 UTC (permalink / raw)
To: Yi Sun
Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel
On Wed, Feb 08, 2017 at 04:15:59PM +0800, Yi Sun wrote:
> This patch implements get value flow including L3 CAT callback
> function.
>
> It also changes domctl interface to make it more general.
>
> With this patch, 'psr-cat-show' can work for L3 CAT but not for
> L3 code/data which is implemented in patch "x86: refactor psr:
> implement get value flow for CDP.".
>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Nice thinking with:
> + if ( d )
> + {
> + cos = d->arch.psr_cos_ids[socket];
> + if ( feat->ops.get_val(feat, cos, type, val) )
> + return 0;
> + else
> + break;
> + }
> +
.. snip..
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
And sorry for recommending the __ in the function name - I forgot
the C standard uses for __ as reserved for compiler/etc needs.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v6 08/24] x86: refactor psr: set value: implement framework.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (6 preceding siblings ...)
2017-02-08 8:15 ` [PATCH v6 07/24] x86: refactor psr: implement get value flow Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
` (15 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
As set value flow is the most complicated one in psr, it will be
divided to some patches to make things clearer. This patch
implements the set value framework to show a whole picture firstly.
It also changes domctl interface to make it more general.
To make the set value flow be general and can support multiple features
at same time, it includes below steps:
1. Get COS ID of current domain using.
2. Assemble a value array to store all features current value
in it and replace the current value of the feature which is
being set to the new input value.
3. Find if there is already a COS ID on which all features'
values are same as the array. Then, we can reuse this COS
ID.
4. If fail to find, we need pick an available COS ID. Only COS ID which ref
is 0 or 1 can be picked.
5. Write all features MSRs according to the COS ID.
6. Update ref according to COS ID.
7. Save the COS ID into current domain's psr_cos_ids[socket] so that we
can know which COS the domain is using on the socket.
So, some functions are abstracted and the callback functions will be
implemented in next patches.
Here is an example to understand the process. The CPU supports
two featuers, e.g. L3 CAT and L2 CAT. user wants to set L3 CAT
of Dom1 to 0x1ff.
1. Get the old_cos of Dom1 which is 0. L3 CAT is the first
element of feature list. The COS registers values are below at
this time.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | ... | ... | ... |
-------------------------------
L2 CAT | 0xff | ... | ... | ... |
-------------------------------
2. Assemble The value array to be:
val[0]: 0x1ff
val[1]: 0xff
3. It cannot find a matching COS.
4. Allocate COS 1 to store the value set.
5. Write the COS 1 registers. The COS registers values are
changed to below now.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | 0x1ff | ... | ... |
-------------------------------
L2 CAT | 0xff | 0xff | ... | ... |
-------------------------------
6. The ref[1] is increased to 1 because Dom1 is using it now.
7. Save 1 to Dom1's psr_cos_ids[socket].
Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
of Dom2 is 0 too. Repeat above flow.
The val array assembled is:
val[0]: 0x1ff
val[1]: 0xff
So, it can find a matching COS, COS 1. Then, it can reuse COS 1
for Dom2.
The ref[1] is increased to 2 now because both Dom1 and Dom2 are
using this COS ID. Set 1 to Dom2's psr_cos_ids[socket].
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/domctl.c | 18 ++---
xen/arch/x86/psr.c | 202 +++++++++++++++++++++++++++++++++++++++++++++-
xen/include/asm-x86/psr.h | 4 +-
3 files changed, 210 insertions(+), 14 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 098e399..f8f5539 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1422,21 +1422,21 @@ long arch_do_domctl(
switch ( domctl->u.psr_cat_op.cmd )
{
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_CODE);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_DATA);
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0cb9971..243344b 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -543,18 +543,214 @@ int psr_get_val(struct domain *d, unsigned int socket,
return __psr_get(socket, type, NULL, 0, d, val);
}
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type)
+/* Set value functions */
+static unsigned int get_cos_num(const struct psr_socket_info *info)
{
return 0;
}
+static int assemble_val_array(uint64_t *val,
+ uint32_t array_len,
+ const struct psr_socket_info *info,
+ unsigned int old_cos)
+{
+ return -EINVAL;
+}
+
+static int set_new_val_to_array(uint64_t *val,
+ uint32_t array_len,
+ const struct psr_socket_info *info,
+ enum psr_feat_type feat_type,
+ enum cbm_type type,
+ uint64_t m)
+{
+ return -EINVAL;
+}
+
+static int find_cos(const uint64_t *val, uint32_t array_len,
+ enum psr_feat_type feat_type,
+ const struct psr_socket_info *info)
+{
+ return -ENOENT;
+}
+
+static int pick_avail_cos(const struct psr_socket_info *info,
+ const uint64_t *val, uint32_t array_len,
+ unsigned int old_cos,
+ enum psr_feat_type feat_type)
+{
+ return -ENOENT;
+}
+
+static int write_psr_msr(unsigned int socket, unsigned int cos,
+ const uint64_t *val)
+{
+ return -ENOENT;
+}
+
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t val, enum cbm_type type)
+{
+ unsigned int old_cos;
+ int cos, ret;
+ unsigned int *ref;
+ uint64_t *val_array;
+ struct psr_socket_info *info = get_socket_info(socket);
+ uint32_t array_len;
+ enum psr_feat_type feat_type;
+
+ if ( IS_ERR(info) )
+ return PTR_ERR(info);
+
+ feat_type = psr_cbm_type_to_feat_type(type);
+ if ( !test_bit(feat_type, &info->feat_mask) )
+ return -ENOENT;
+
+ /*
+ * Step 0:
+ * old_cos means the COS ID current domain is using. By default, it is 0.
+ *
+ * For every COS ID, there is a reference count to record how many domains
+ * are using the COS register corresponding to this COS ID.
+ * - If ref[old_cos] is 0, that means this COS is not used by any domain.
+ * - If ref[old_cos] is 1, that means this COS is only used by current
+ * domain.
+ * - If ref[old_cos] is more than 1, that mean multiple domains are using
+ * this COS.
+ */
+ old_cos = d->arch.psr_cos_ids[socket];
+ if ( old_cos > MAX_COS_REG_CNT )
+ return -EOVERFLOW;
+
+ ref = info->cos_ref;
+
+ /*
+ * Step 1:
+ * Assemle a value array to store all featues cos_reg_val[old_cos].
+ * And, set the input val into array according to the feature's
+ * position in array.
+ */
+ array_len = get_cos_num(info);
+ val_array = xzalloc_array(uint64_t, array_len);
+ if ( !val_array )
+ return -ENOMEM;
+
+ if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 )
+ {
+ xfree(val_array);
+ return ret;
+ }
+
+ if ( (ret = set_new_val_to_array(val_array, array_len, info,
+ feat_type, type, val)) != 0 )
+ {
+ xfree(val_array);
+ return ret;
+ }
+
+ /*
+ * Lock here to make sure the ref is not changed during find and
+ * write process.
+ */
+ spin_lock(&info->ref_lock);
+
+ /*
+ * Step 2:
+ * Try to find if there is already a COS ID on which all features' values
+ * are same as the array. Then, we can reuse this COS ID.
+ */
+ cos = find_cos(val_array, array_len, feat_type, info);
+ if ( cos >= 0 )
+ {
+ if ( cos == old_cos )
+ {
+ spin_unlock(&info->ref_lock);
+ xfree(val_array);
+ return 0;
+ }
+ }
+ else
+ {
+ /*
+ * Step 3:
+ * If fail to find, we need allocate a new COS ID.
+ * If multiple domains are using same COS ID, its ref is more
+ * than 1. That means we cannot free this COS to make current domain
+ * use it. Because other domains are using the value saved in the COS.
+ * Unless the ref is changed to 1 (mean only current domain is using
+ * it), we cannot allocate the COS ID to current domain.
+ * So, only the COS ID which ref is 1 or 0 can be allocated.
+ */
+ cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
+ if ( cos < 0 )
+ {
+ spin_unlock(&info->ref_lock);
+ xfree(val_array);
+ return cos;
+ }
+
+ /*
+ * Step 4:
+ * Write all features MSRs according to the COS ID.
+ */
+ ret = write_psr_msr(socket, cos, val_array);
+ if ( ret )
+ {
+ spin_unlock(&info->ref_lock);
+ xfree(val_array);
+ return ret;
+ }
+ }
+
+ /*
+ * Step 5:
+ * Update ref according to COS ID.
+ */
+ ref[cos]++;
+ ASSERT(ref[cos] || cos == 0);
+ ref[old_cos]--;
+ spin_unlock(&info->ref_lock);
+
+ /*
+ * Step 6:
+ * Save the COS ID into current domain's psr_cos_ids[] so that we can know
+ * which COS the domain is using on the socket. One domain can only use
+ * one COS ID at same time on each socket.
+ */
+ d->arch.psr_cos_ids[socket] = cos;
+ xfree(val_array);
+
+ return 0;
+}
+
/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
static void psr_free_cos(struct domain *d)
{
- if( !d->arch.psr_cos_ids )
+ unsigned int socket, cos;
+
+ if ( !d->arch.psr_cos_ids )
return;
+ /* Domain is free so its cos_ref should be decreased. */
+ for ( socket = 0; socket < nr_sockets; socket++ )
+ {
+ struct psr_socket_info *info;
+
+ /* cos 0 is default one which does not need be handled. */
+ if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
+ continue;
+
+ /*
+ * If domain uses other cos ids, all corresponding refs must have been
+ * increased 1 for this domain. So, we need decrease them.
+ */
+ info = socket_info + socket;
+ ASSERT(info->cos_ref[cos] || cos == 0);
+ spin_lock(&info->ref_lock);
+ info->cos_ref[cos]--;
+ spin_unlock(&info->ref_lock);
+ }
+
xfree(d->arch.psr_cos_ids);
d->arch.psr_cos_ids = NULL;
}
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 569e7df..fde7882 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -72,8 +72,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t data[], unsigned int array_len);
int psr_get_val(struct domain *d, unsigned int socket,
uint64_t *val, enum cbm_type type);
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type);
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t val, enum cbm_type type);
int psr_domain_init(struct domain *d);
void psr_domain_free(struct domain *d);
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 09/24] x86: refactor psr: set value: assemble features value array.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (7 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 08/24] x86: refactor psr: set value: implement framework Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
` (14 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
Only can one COS ID be used by one domain at one time. That means all enabled
features' COS registers at this COS ID are valid for this domain at that time.
When user updates a feature's value, we need make sure all other features'
values are not affected. So, we firstly need assemble an array which contains
all features current values and replace the setting feature's value in array
to new value.
Then, we can try to find if there is a COS ID on which all features' COS
registers values are same as the array. If we can find, we just use this COS
ID. If fail to find, we need allocate a new COS ID.
This patch implements value array assembling flow.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v6:
- change 'assemble_val_array' to 'combine_val_array'.
- check return value of 'get_old_val'.
- replace some 'EINVAL' to 'ENOSPC'.
---
xen/arch/x86/psr.c | 151 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 146 insertions(+), 5 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 243344b..c3ba7ee 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -119,6 +119,32 @@ struct feat_ops {
/* get_val is used to get feature COS register value. */
bool (*get_val)(const struct feat_node *feat, unsigned int cos,
enum cbm_type type, uint64_t *val);
+ /*
+ * get_cos_num is used to get the COS registers amount used by the
+ * feature for one setting, e.g. CDP uses 2 COSs but CAT uses 1.
+ */
+ unsigned int (*get_cos_num)(const struct feat_node *feat);
+ /*
+ * get_old_val and set_new_val are a pair of functions called in order.
+ * The caller will traverse all features in the list and call both
+ * functions for every feature to do below two things:
+ * 1. get old_cos register value of all supported features and
+ * 2. set the new value for the feature.
+ *
+ * All the values are set into value array according the traversal order,
+ * meaning the same order of feature list members.
+ *
+ * The return value meaning:
+ * 0 - success.
+ * negative - error.
+ */
+ int (*get_old_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos);
+ int (*set_new_val)(uint64_t val[],
+ const struct feat_node *feat,
+ enum cbm_type type,
+ uint64_t m);
};
/*
@@ -204,6 +230,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
return feat_type;
}
+static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
+{
+ unsigned int first_bit, zero_bit;
+
+ /* Set bits should only in the range of [0, cbm_len). */
+ if ( cbm & (~0ull << cbm_len) )
+ return false;
+
+ /* At least one bit need to be set. */
+ if ( cbm == 0 )
+ return false;
+
+ first_bit = find_first_bit(&cbm, cbm_len);
+ zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
+
+ /* Set bits should be contiguous. */
+ if ( zero_bit < cbm_len &&
+ find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
+ return false;
+
+ return true;
+}
+
/* L3 CAT functions implementation. */
static void l3_cat_init_feature(struct cpuid_leaf regs,
struct feat_node *feat,
@@ -272,10 +321,45 @@ static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
return true;
}
+static unsigned int l3_cat_get_cos_num(const struct feat_node *feat)
+{
+ return 1;
+}
+
+static int l3_cat_get_old_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos)
+{
+ if ( old_cos > feat->info.l3_cat_info.cos_max )
+ /* Use default value. */
+ old_cos = 0;
+
+ /* CAT */
+ val[0] = feat->cos_reg_val[old_cos];
+
+ return 0;
+}
+
+static int l3_cat_set_new_val(uint64_t val[],
+ const struct feat_node *feat,
+ enum cbm_type type,
+ uint64_t m)
+{
+ if ( !psr_check_cbm(feat->info.l3_cat_info.cbm_len, m) )
+ return -EINVAL;
+
+ val[0] = m;
+
+ return 0;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
.get_val = l3_cat_get_val,
+ .get_cos_num = l3_cat_get_cos_num,
+ .get_old_val = l3_cat_get_old_val,
+ .set_new_val = l3_cat_set_new_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -546,15 +630,42 @@ int psr_get_val(struct domain *d, unsigned int socket,
/* Set value functions */
static unsigned int get_cos_num(const struct psr_socket_info *info)
{
- return 0;
+ const struct feat_node *feat_tmp;
+ unsigned int num = 0;
+
+ /* Get all features total amount. */
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ num += feat_tmp->ops.get_cos_num(feat_tmp);
+
+ return num;
}
-static int assemble_val_array(uint64_t *val,
+static int combine_val_array(uint64_t *val,
uint32_t array_len,
const struct psr_socket_info *info,
unsigned int old_cos)
{
- return -EINVAL;
+ const struct feat_node *feat;
+ int ret;
+ uint64_t *val_tmp = val;
+
+ if ( !val )
+ return -EINVAL;
+
+ /* Get all features current values according to old_cos. */
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ /* value getting order is same as feature list */
+ ret = feat->ops.get_old_val(val_tmp, feat, old_cos);
+ if ( ret )
+ return ret;
+
+ val_tmp += feat->ops.get_cos_num(feat);
+ if ( val_tmp - val > array_len)
+ return -ENOSPC;
+ }
+
+ return 0;
}
static int set_new_val_to_array(uint64_t *val,
@@ -564,7 +675,37 @@ static int set_new_val_to_array(uint64_t *val,
enum cbm_type type,
uint64_t m)
{
- return -EINVAL;
+ const struct feat_node *feat;
+ int ret;
+ uint64_t *val_tmp = val;
+
+ /* Set new value into array according to feature's position in array. */
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ if ( feat->feature != feat_type )
+ {
+ val_tmp += feat->ops.get_cos_num(feat);
+ if ( val_tmp - val > array_len)
+ return -ENOSPC;
+
+ continue;
+ }
+
+ /*
+ * Value setting position is same as feature list.
+ * Different features may have different setting behaviors, e.g. CDP
+ * has two values (DATA/CODE) which need us to save input value to
+ * different position in the array according to type, so we have to
+ * maintain a callback function.
+ */
+ ret = feat->ops.set_new_val(val_tmp, feat, type, m);
+ if ( ret )
+ return ret;
+ else
+ break;
+ }
+
+ return 0;
}
static int find_cos(const uint64_t *val, uint32_t array_len,
@@ -635,7 +776,7 @@ int psr_set_val(struct domain *d, unsigned int socket,
if ( !val_array )
return -ENOMEM;
- if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 )
+ if ( (ret = combine_val_array(val_array, array_len, info, old_cos)) != 0 )
{
xfree(val_array);
return ret;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 10/24] x86: refactor psr: set value: implement cos finding flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (8 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
` (13 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
Continue with patch:
'x86: refactor psr: set value: assemble features value array'
We can try to find if there is a COS ID on which all features' COS registers
values are same as the array assembled before.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 93 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index c3ba7ee..3d29d33 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -145,6 +145,19 @@ struct feat_ops {
const struct feat_node *feat,
enum cbm_type type,
uint64_t m);
+ /*
+ * compare_val is used in set value process to compare if the
+ * input value array can match all the features' COS registers values
+ * according to input cos id.
+ *
+ * The return value is the amount of entries to skip in the value array
+ * or error.
+ * 1 - one entry in value array.
+ * 2 - two entries in value array, e.g. CDP uses two entries.
+ * negative - error.
+ */
+ int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
+ unsigned int cos, bool *found);
};
/*
@@ -353,6 +366,34 @@ static int l3_cat_set_new_val(uint64_t val[],
return 0;
}
+static int l3_cat_compare_val(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos, bool *found)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
+
+ /*
+ * Different features' cos_max are different. If cos id of the feature
+ * being set exceeds other feature's cos_max, the val of other feature
+ * must be default value. HW supports such case.
+ */
+ if ( cos > feat->info.l3_cat_info.cos_max )
+ {
+ if ( val[0] != l3_def_cbm )
+ {
+ *found = false;
+ return -ENOENT;
+ }
+ *found = true;
+ }
+ else
+ *found = (val[0] == feat->cos_reg_val[cos]);
+
+ return 0;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
@@ -360,6 +401,7 @@ static const struct feat_ops l3_cat_ops = {
.get_cos_num = l3_cat_get_cos_num,
.get_old_val = l3_cat_get_old_val,
.set_new_val = l3_cat_set_new_val,
+ .compare_val = l3_cat_compare_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -712,6 +754,57 @@ static int find_cos(const uint64_t *val, uint32_t array_len,
enum psr_feat_type feat_type,
const struct psr_socket_info *info)
{
+ unsigned int cos;
+ const unsigned int *ref = info->cos_ref;
+ const struct feat_node *feat;
+ const uint64_t *val_tmp = val;
+ int ret;
+ bool found = false;
+ unsigned int cos_max = 0;
+
+ /* cos_max is the one of the feature which is being set. */
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ if ( feat->feature != feat_type )
+ continue;
+
+ cos_max = feat->ops.get_cos_max(feat);
+ if ( cos_max > 0 )
+ break;
+ }
+
+ for ( cos = 0; cos <= cos_max; cos++ )
+ {
+ if ( cos && !ref[cos] )
+ continue;
+
+ /* Not found, need find again from beginning. */
+ val_tmp = val;
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ /*
+ * Compare value according to feature list order.
+ * We must follow this order because value array is assembled
+ * as this order in get_old_set_new().
+ */
+ ret = feat->ops.compare_val(val_tmp, feat, cos, &found);
+ if ( ret < 0 )
+ return ret;
+
+ /* If fail to match, go to next cos to compare. */
+ if ( !found )
+ break;
+
+ val_tmp += feat->ops.get_cos_num(feat);
+ if ( val_tmp - val > array_len )
+ return -EINVAL;
+ }
+
+ /* For this COS ID all entries in the values array did match. Use it. */
+ if ( found )
+ return cos;
+ }
+
return -ENOENT;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 11/24] x86: refactor psr: set value: implement cos id picking flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (9 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 12/24] x86: refactor psr: set value: implement write msr flow Yi Sun
` (12 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
Continue with previous patch:
'x86: refactor psr: set value: implement cos finding flow.'
If fail to find a COS ID, we need pick a new COS ID for domain. Only COS ID
that ref[COS_ID] is 1 or 0 can be picked to input a new set feature values.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 99 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 3d29d33..8f06d60 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -158,6 +158,17 @@ struct feat_ops {
*/
int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
unsigned int cos, bool *found);
+ /*
+ * fits_cos_max is used to check if the input cos id exceeds the
+ * feature's cos_max and if the input value is not the default one.
+ * Even if the associated cos exceeds the cos_max, HW can work with default
+ * value. That is the reason we need check if input value is default one.
+ * If both criteria are fulfilled, that means the input exceeds the range.
+ * If not, that means the input fits the requirements.
+ */
+ bool (*fits_cos_max)(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos);
};
/*
@@ -394,6 +405,25 @@ static int l3_cat_compare_val(const uint64_t val[],
return 0;
}
+static bool l3_cat_fits_cos_max(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
+
+ if ( cos > feat->info.l3_cat_info.cos_max &&
+ val[0] != l3_def_cbm )
+ /*
+ * Exceed cos_max and value to set is not default,
+ * return error.
+ */
+ return false;
+
+ return true;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
@@ -402,6 +432,7 @@ static const struct feat_ops l3_cat_ops = {
.get_old_val = l3_cat_get_old_val,
.set_new_val = l3_cat_set_new_val,
.compare_val = l3_cat_compare_val,
+ .fits_cos_max = l3_cat_fits_cos_max,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -808,11 +839,79 @@ static int find_cos(const uint64_t *val, uint32_t array_len,
return -ENOENT;
}
+static bool fits_cos_max(const uint64_t *val,
+ uint32_t array_len,
+ const struct psr_socket_info *info,
+ unsigned int cos)
+{
+ unsigned int ret;
+ const uint64_t *val_tmp = val;
+ const struct feat_node *feat;
+
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ ret = feat->ops.fits_cos_max(val_tmp, feat, cos);
+ if ( !ret )
+ return false;
+
+ val_tmp += feat->ops.get_cos_num(feat);
+ if ( val_tmp - val > array_len )
+ return false;
+ }
+
+ return true;
+}
+
static int pick_avail_cos(const struct psr_socket_info *info,
const uint64_t *val, uint32_t array_len,
unsigned int old_cos,
enum psr_feat_type feat_type)
{
+ unsigned int cos;
+ unsigned int cos_max = 0;
+ const struct feat_node *feat;
+ const unsigned int *ref = info->cos_ref;
+
+ /*
+ * cos_max is the one of the feature which is being set.
+ */
+ list_for_each_entry(feat, &info->feat_list, list)
+ {
+ if ( feat->feature != feat_type )
+ continue;
+
+ cos_max = feat->ops.get_cos_max(feat);
+ if ( cos_max > 0 )
+ break;
+ }
+
+ if ( !cos_max )
+ return -ENOENT;
+
+ /*
+ * If old cos is referred only by the domain, then use it. And, we cannot
+ * use id 0 because it stores the default values.
+ */
+ if ( old_cos && ref[old_cos] == 1 &&
+ fits_cos_max(val, array_len, info, old_cos) )
+ return old_cos;
+
+ /* Find an unused one other than cos0. */
+ for ( cos = 1; cos <= cos_max; cos++ )
+ {
+ /*
+ * ref is 0 means this COS is not used by other domain and
+ * can be used for current setting.
+ */
+ if ( !ref[cos] )
+ {
+ if ( !fits_cos_max(val, array_len, info, cos) )
+ return -ENOENT;
+
+ return cos;
+ }
+ }
+
return -ENOENT;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 12/24] x86: refactor psr: set value: implement write msr flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (10 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
` (11 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
Continue with previous patch:
'x86: refactor psr: set value: implement cos id picking flow.'
We have got all features values and COS ID to set. Then, we write MSRs of all
features except the setting value is same as original value.
Till now, set value process is completed.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 77 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 8f06d60..23bf8d8 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -169,6 +169,9 @@ struct feat_ops {
bool (*fits_cos_max)(const uint64_t val[],
const struct feat_node *feat,
unsigned int cos);
+ /* write_msr is used to write out feature MSR register. */
+ int (*write_msr)(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat);
};
/*
@@ -424,6 +427,21 @@ static bool l3_cat_fits_cos_max(const uint64_t val[],
return true;
}
+static int l3_cat_write_msr(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat)
+{
+ if ( cos > feat->info.l3_cat_info.cos_max )
+ return -EINVAL;
+
+ if ( feat->cos_reg_val[cos] != val[0] )
+ {
+ feat->cos_reg_val[cos] = val[0];
+ wrmsrl(MSR_IA32_PSR_L3_MASK(cos), val[0]);
+ }
+
+ return 0;
+}
+
static const struct feat_ops l3_cat_ops = {
.get_cos_max = l3_cat_get_cos_max,
.get_feat_info = l3_cat_get_feat_info,
@@ -433,6 +451,7 @@ static const struct feat_ops l3_cat_ops = {
.set_new_val = l3_cat_set_new_val,
.compare_val = l3_cat_compare_val,
.fits_cos_max = l3_cat_fits_cos_max,
+ .write_msr = l3_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -915,10 +934,67 @@ static int pick_avail_cos(const struct psr_socket_info *info,
return -ENOENT;
}
+static unsigned int get_socket_cpu(unsigned int socket)
+{
+ if ( likely(socket < nr_sockets) )
+ return cpumask_any(socket_cpumask[socket]);
+
+ return nr_cpu_ids;
+}
+
+struct cos_write_info
+{
+ unsigned int cos;
+ struct list_head *feat_list;
+ const uint64_t *val;
+};
+
+static void do_write_psr_msr(void *data)
+{
+ struct cos_write_info *info = (struct cos_write_info *)data;
+ unsigned int cos = info->cos;
+ struct list_head *feat_list= info->feat_list;
+ const uint64_t *val = info->val;
+ struct feat_node *feat;
+ int ret;
+
+ if ( !feat_list )
+ return;
+
+ /* We need set all features values into MSRs. */
+ list_for_each_entry(feat, feat_list, list)
+ {
+ ret = feat->ops.write_msr(cos, val, feat);
+ if ( ret < 0 )
+ return;
+
+ val += feat->ops.get_cos_num(feat);
+ }
+}
+
static int write_psr_msr(unsigned int socket, unsigned int cos,
const uint64_t *val)
{
- return -ENOENT;
+ struct psr_socket_info *info = get_socket_info(socket);
+ struct cos_write_info data =
+ {
+ .cos = cos,
+ .feat_list = &info->feat_list,
+ .val = val,
+ };
+
+ if ( socket == cpu_to_socket(smp_processor_id()) )
+ do_write_psr_msr(&data);
+ else
+ {
+ unsigned int cpu = get_socket_cpu(socket);
+
+ if ( cpu >= nr_cpu_ids )
+ return -ENOTSOCK;
+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msr, &data, 1);
+ }
+
+ return 0;
}
int psr_set_val(struct domain *d, unsigned int socket,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 13/24] x86: refactor psr: implement CPU init and free flow for CDP.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (11 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 12/24] x86: refactor psr: set value: implement write msr flow Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 14/24] x86: refactor psr: implement get hw info " Yi Sun
` (10 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the CPU init and free flow for CDP including L3 CDP
initialization callback function.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v6:
- use 'struct cpuid_leaf' in x86_emulate.h.
- use cpuid_count_leaf() to get cpuid info.
---
xen/arch/x86/psr.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 98 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 23bf8d8..0739c1c 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -97,6 +97,7 @@ struct psr_cat_hw_info {
struct feat_hw_info {
union {
struct psr_cat_hw_info l3_cat_info;
+ struct psr_cat_hw_info l3_cdp_info;
};
};
@@ -195,6 +196,22 @@ struct feat_node {
struct list_head list;
};
+/*
+ * get_data - get DATA COS register value from input COS ID.
+ * @feat: the feature list entry.
+ * @cos: the COS ID.
+ */
+#define get_cdp_data(feat, cos) \
+ ( feat->cos_reg_val[cos * 2] )
+
+/*
+ * get_cdp_code - get CODE COS register value from input COS ID.
+ * @feat: the feature list entry.
+ * @cos: the COS ID.
+ */
+#define get_cdp_code(feat, cos) \
+ ( feat->cos_reg_val[cos * 2 + 1] )
+
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -217,6 +234,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
* cpu_add_remove_lock spinlock.
*/
static struct feat_node *feat_l3_cat;
+static struct feat_node *feat_l3_cdp;
/* Common functions. */
static void free_feature(struct psr_socket_info *info)
@@ -454,6 +472,63 @@ static const struct feat_ops l3_cat_ops = {
.write_msr = l3_cat_write_msr,
};
+/* L3 CDP functions implementation. */
+static void l3_cdp_init_feature(struct cpuid_leaf regs,
+ struct feat_node *feat,
+ struct psr_socket_info *info)
+{
+ struct psr_cat_hw_info l3_cdp;
+ unsigned int socket;
+ uint64_t val;
+
+ /* No valid value so do not enable feature. */
+ if ( !regs.a || !regs.d )
+ return;
+
+ l3_cdp.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
+ /* Cut half of cos_max when CDP is enabled. */
+ l3_cdp.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK) >> 1;
+
+ /* cos=0 is reserved as default cbm(all ones). */
+ get_cdp_code(feat, 0) =
+ (1ull << l3_cdp.cbm_len) - 1;
+ get_cdp_data(feat, 0) =
+ (1ull << l3_cdp.cbm_len) - 1;
+
+ /* We only write mask1 since mask0 is always all ones by default. */
+ wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << l3_cdp.cbm_len) - 1);
+ rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
+ wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
+
+ feat->feature = PSR_SOCKET_L3_CDP;
+ ASSERT(!test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask));
+ __set_bit(PSR_SOCKET_L3_CDP, &info->feat_mask);
+
+ feat->info.l3_cdp_info = l3_cdp;
+
+ info->nr_feat++;
+
+ /* Add this feature into list. */
+ list_add_tail(&feat->list, &info->feat_list);
+
+ socket = cpu_to_socket(smp_processor_id());
+ if ( !opt_cpu_info )
+ return;
+
+ printk(XENLOG_INFO "L3 CDP: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+ socket, feat->info.l3_cdp_info.cos_max,
+ feat->info.l3_cdp_info.cbm_len);
+}
+
+static unsigned int l3_cdp_get_cos_max(const struct feat_node *feat)
+{
+ return feat->info.l3_cdp_info.cos_max;
+}
+
+struct feat_ops l3_cdp_ops = {
+ .get_cos_max = l3_cdp_get_cos_max,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1211,12 +1286,21 @@ static void cpu_init_work(void)
{
cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, ®s);
- feat = feat_l3_cat;
- /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
- feat_l3_cat = NULL;
- feat->ops = l3_cat_ops;
-
- l3_cat_init_feature(regs, feat, info);
+ if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
+ !test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) )
+ {
+ feat = feat_l3_cdp;
+ /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
+ feat_l3_cdp = NULL;
+ feat->ops = l3_cdp_ops;
+ l3_cdp_init_feature(regs, feat, info);
+ } else {
+ feat = feat_l3_cat;
+ /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
+ feat_l3_cat = NULL;
+ feat->ops = l3_cat_ops;
+ l3_cat_init_feature(regs, feat, info);
+ }
}
}
@@ -1268,6 +1352,14 @@ static int psr_cpu_prepare(unsigned int cpu)
(feat_l3_cat = xzalloc(struct feat_node)) == NULL )
return -ENOMEM;
+ if ( feat_l3_cdp == NULL &&
+ (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
+ {
+ xfree(feat_l3_cat);
+ feat_l3_cat = NULL;
+ return -ENOMEM;
+ }
+
return 0;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 14/24] x86: refactor psr: implement get hw info flow for CDP.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (12 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 15/24] x86: refactor psr: implement get value " Yi Sun
` (9 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements get HW info flow for CDP including L3 CDP callback
function.
It also changes sysctl function to make it work for CDP.
With this patch, 'psr-hwinfo' can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 18 ++++++++++++++++++
xen/arch/x86/sysctl.c | 24 +++++++++++++++++++++---
2 files changed, 39 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0739c1c..5c307b5 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -267,6 +267,10 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
case PSR_CBM_TYPE_L3:
feat_type = PSR_SOCKET_L3_CAT;
break;
+ case PSR_CBM_TYPE_L3_DATA:
+ case PSR_CBM_TYPE_L3_CODE:
+ feat_type = PSR_SOCKET_L3_CDP;
+ break;
default:
feat_type = PSR_SOCKET_UNKNOWN;
break;
@@ -525,8 +529,22 @@ static unsigned int l3_cdp_get_cos_max(const struct feat_node *feat)
return feat->info.l3_cdp_info.cos_max;
}
+static bool l3_cdp_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], uint32_t array_len)
+{
+ if ( !data || 3 > array_len )
+ return false;
+
+ data[CBM_LEN] = feat->info.l3_cdp_info.cbm_len;
+ data[COS_MAX] = feat->info.l3_cdp_info.cos_max;
+ data[PSR_FLAG] |= XEN_SYSCTL_PSR_CAT_L3_CDP;
+
+ return true;
+}
+
struct feat_ops l3_cdp_ops = {
.get_cos_max = l3_cdp_get_cos_max,
+ .get_feat_info = l3_cdp_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index e340baa..568bfe9 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -181,9 +181,27 @@ long arch_do_sysctl(
ret = psr_get_info(sysctl->u.psr_cat_op.target,
PSR_CBM_TYPE_L3, data, 3);
- sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
- sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
- sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
+ if ( !ret )
+ {
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
+ } else {
+ /*
+ * Check if CDP is enabled.
+ *
+ * Per spec, L3 CAT and CDP cannot co-exist. So, we need replace
+ * output values to CDP's if it is enabled.
+ */
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L3_CODE, data, 3);
+ if ( !ret )
+ {
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_FLAG];
+ }
+ }
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
ret = -EFAULT;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 15/24] x86: refactor psr: implement get value flow for CDP.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (13 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 14/24] x86: refactor psr: implement get hw info " Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 16/24] x86: refactor psr: implement set value callback functions " Yi Sun
` (8 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements L3 CDP get value callback function.
With this patch, 'psr-cat-show' can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5c307b5..df83aa1 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -542,9 +542,25 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
return true;
}
+static bool l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val)
+{
+ if ( cos > feat->info.l3_cdp_info.cos_max )
+ /* Use default value. */
+ cos = 0;
+
+ if ( type == PSR_CBM_TYPE_L3_DATA )
+ *val = get_cdp_data(feat, cos);
+ else
+ *val = get_cdp_code(feat, cos);
+
+ return true;
+}
+
struct feat_ops l3_cdp_ops = {
.get_cos_max = l3_cdp_get_cos_max,
.get_feat_info = l3_cdp_get_feat_info,
+ .get_val = l3_cdp_get_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 16/24] x86: refactor psr: implement set value callback functions for CDP.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (14 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 15/24] x86: refactor psr: implement get value " Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
` (7 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements L3 CDP set value related callback functions.
With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 118 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 118 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index df83aa1..3cfa77d 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -557,10 +557,128 @@ static bool l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
return true;
}
+static unsigned int l3_cdp_get_cos_num(const struct feat_node *feat)
+{
+ return 2;
+}
+
+static int l3_cdp_get_old_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos)
+{
+ if ( old_cos > feat->info.l3_cdp_info.cos_max )
+ /* Use default value. */
+ old_cos = 0;
+
+ /* Data */
+ val[0] = get_cdp_data(feat, old_cos);
+ /* Code */
+ val[1] = get_cdp_code(feat, old_cos);
+
+ return 0;
+}
+
+static int l3_cdp_set_new_val(uint64_t val[],
+ const struct feat_node *feat,
+ enum cbm_type type,
+ uint64_t m)
+{
+ if ( !psr_check_cbm(feat->info.l3_cdp_info.cbm_len, m) )
+ return -EINVAL;
+
+ if ( type == PSR_CBM_TYPE_L3_DATA )
+ val[0] = m;
+ else
+ val[1] = m;
+
+ return 0;
+}
+
+static int l3_cdp_compare_val(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos, bool *found)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.l3_cdp_info.cbm_len) - 1;
+
+ /*
+ * Different features' cos_max are different. If cos id of the feature
+ * being set exceeds other feature's cos_max, the val of other feature
+ * must be default value. HW supports such case.
+ */
+ if ( cos > feat->info.l3_cdp_info.cos_max )
+ {
+ if ( val[0] != l3_def_cbm ||
+ val[1] != l3_def_cbm )
+ {
+ *found = false;
+ return -ENOENT;
+ }
+ *found = true;
+ }
+ else
+ *found = (val[0] == get_cdp_data(feat, cos) &&
+ val[1] == get_cdp_code(feat, cos));
+
+ return 0;
+}
+
+static bool l3_cdp_fits_cos_max(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.l3_cdp_info.cbm_len) - 1;
+
+ if ( cos > feat->info.l3_cdp_info.cos_max &&
+ (val[0] != l3_def_cbm || val[1] != l3_def_cbm) )
+ /*
+ * Exceed cos_max and value to set is not default,
+ * return error.
+ */
+ return false;
+
+ return true;
+}
+
+static int l3_cdp_write_msr(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat)
+{
+ /*
+ * If input cos is more than the cos_max of the feature, we should
+ * not set the value.
+ */
+ if ( cos > feat->info.l3_cdp_info.cos_max )
+ return -EINVAL;
+
+ /* Data */
+ if ( get_cdp_data(feat, cos) != val[0] )
+ {
+ get_cdp_data(feat, cos) = val[0];
+ wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(cos), val[0]);
+ }
+ /* Code */
+ if ( get_cdp_code(feat, cos) != val[1] )
+ {
+ get_cdp_code(feat, cos) = val[1];
+ wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(cos), val[1]);
+ }
+
+ return 0;
+}
+
struct feat_ops l3_cdp_ops = {
.get_cos_max = l3_cdp_get_cos_max,
.get_feat_info = l3_cdp_get_feat_info,
.get_val = l3_cdp_get_val,
+ .get_cos_num = l3_cdp_get_cos_num,
+ .get_old_val = l3_cdp_get_old_val,
+ .set_new_val = l3_cdp_set_new_val,
+ .compare_val = l3_cdp_compare_val,
+ .fits_cos_max = l3_cdp_fits_cos_max,
+ .write_msr = l3_cdp_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 17/24] x86: L2 CAT: implement CPU init and free flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (15 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 16/24] x86: refactor psr: implement set value callback functions " Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
` (6 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the CPU init and free flow for L2 CAT including
L2 CAT initialization callback function.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v6:
- use 'struct cpuid_leaf' in x86_emulate.h.
- use cpuid_count_leaf() to get cpuid info.
- adjust macro value position.
---
xen/arch/x86/psr.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++
xen/include/asm-x86/psr.h | 1 +
2 files changed, 73 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 3cfa77d..0ef398e 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -98,6 +98,7 @@ struct feat_hw_info {
union {
struct psr_cat_hw_info l3_cat_info;
struct psr_cat_hw_info l3_cdp_info;
+ struct psr_cat_hw_info l2_cat_info;
};
};
@@ -235,6 +236,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
*/
static struct feat_node *feat_l3_cat;
static struct feat_node *feat_l3_cdp;
+static struct feat_node *feat_l2_cat;
/* Common functions. */
static void free_feature(struct psr_socket_info *info)
@@ -681,6 +683,53 @@ struct feat_ops l3_cdp_ops = {
.write_msr = l3_cdp_write_msr,
};
+/* L2 CAT callback functions implementation. */
+static void l2_cat_init_feature(struct cpuid_leaf regs,
+ struct feat_node *feat,
+ struct psr_socket_info *info)
+{
+ struct psr_cat_hw_info l2_cat;
+ unsigned int socket;
+
+ /* No valid values so do not enable the feature. */
+ if ( !regs.a || !regs.d )
+ return;
+
+ l2_cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
+ l2_cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
+
+ /* cos=0 is reserved as default cbm(all ones). */
+ feat->cos_reg_val[0] = (1ull << l2_cat.cbm_len) - 1;
+
+ feat->feature = PSR_SOCKET_L2_CAT;
+ ASSERT(!test_bit(PSR_SOCKET_L2_CAT, &info->feat_mask));
+ __set_bit(PSR_SOCKET_L2_CAT, &info->feat_mask);
+
+ feat->info.l2_cat_info = l2_cat;
+
+ info->nr_feat++;
+
+ /* Add this feature into list. */
+ list_add_tail(&feat->list, &info->feat_list);
+
+ socket = cpu_to_socket(smp_processor_id());
+ if ( !opt_cpu_info )
+ return;
+
+ printk(XENLOG_INFO "L2 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u.\n",
+ socket, feat->info.l2_cat_info.cos_max,
+ feat->info.l2_cat_info.cbm_len);
+}
+
+static unsigned int l2_cat_get_cos_max(const struct feat_node *feat)
+{
+ return feat->info.l2_cat_info.cos_max;
+}
+
+struct feat_ops l2_cat_ops = {
+ .get_cos_max = l2_cat_get_cos_max,
+};
+
static void __init parse_psr_bool(char *s, char *value, char *feature,
unsigned int mask)
{
@@ -1454,6 +1503,18 @@ static void cpu_init_work(void)
l3_cat_init_feature(regs, feat, info);
}
}
+
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, ®s);
+ if ( regs.b & PSR_RESOURCE_TYPE_L2 )
+ {
+ cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, ®s);
+
+ feat = feat_l2_cat;
+ /* psr_cpu_prepare will allocate it on subsequent CPU onlining. */
+ feat_l2_cat = NULL;
+ feat->ops = l2_cat_ops;
+ l2_cat_init_feature(regs, feat, info);
+ }
}
static void cpu_fini_work(unsigned int cpu)
@@ -1512,6 +1573,17 @@ static int psr_cpu_prepare(unsigned int cpu)
return -ENOMEM;
}
+ if ( feat_l2_cat == NULL &&
+ (feat_l2_cat = xzalloc(struct feat_node)) == NULL )
+ {
+ xfree(feat_l3_cat);
+ feat_l3_cat = NULL;
+
+ xfree(feat_l3_cdp);
+ feat_l3_cdp = NULL;
+ return -ENOMEM;
+ }
+
return 0;
}
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index fde7882..d7ed012 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -23,6 +23,7 @@
/* Resource Type Enumeration */
#define PSR_RESOURCE_TYPE_L3 0x2
+#define PSR_RESOURCE_TYPE_L2 0x4
/* L3 Monitoring Features */
#define PSR_CMT_L3_OCCUPANCY 0x1
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 18/24] x86: L2 CAT: implement get hw info flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (16 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 19/24] x86: L2 CAT: implement get value flow Yi Sun
` (5 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements get HW info flow for L2 CAT including L2 CAT callback
function.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 16 ++++++++++++++++
xen/arch/x86/sysctl.c | 15 +++++++++++++++
xen/include/asm-x86/psr.h | 1 +
xen/include/public/sysctl.h | 6 ++++++
4 files changed, 38 insertions(+)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0ef398e..5d1c2d5 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -273,6 +273,9 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
case PSR_CBM_TYPE_L3_CODE:
feat_type = PSR_SOCKET_L3_CDP;
break;
+ case PSR_CBM_TYPE_L2:
+ feat_type = PSR_SOCKET_L2_CAT;
+ break;
default:
feat_type = PSR_SOCKET_UNKNOWN;
break;
@@ -726,8 +729,21 @@ static unsigned int l2_cat_get_cos_max(const struct feat_node *feat)
return feat->info.l2_cat_info.cos_max;
}
+static bool l2_cat_get_feat_info(const struct feat_node *feat,
+ uint32_t data[], uint32_t array_len)
+{
+ if ( !data || 2 > array_len )
+ return false;
+
+ data[CBM_LEN] = feat->info.l2_cat_info.cbm_len;
+ data[COS_MAX] = feat->info.l2_cat_info.cos_max;
+
+ return true;
+}
+
struct feat_ops l2_cat_ops = {
.get_cos_max = l2_cat_get_cos_max,
+ .get_feat_info = l2_cat_get_feat_info,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 568bfe9..01cf3b7 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -207,6 +207,21 @@ long arch_do_sysctl(
ret = -EFAULT;
break;
}
+ case XEN_SYSCTL_PSR_CAT_get_l2_info:
+ {
+ uint32_t dat[2];
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L2, dat, 2);
+ if ( ret )
+ break;
+
+ sysctl->u.psr_cat_op.u.l2_info.cbm_len = dat[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l2_info.cos_max = dat[COS_MAX];
+
+ if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
+ ret = -EFAULT;
+ break;
+ }
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index d7ed012..2e1b3d0 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -56,6 +56,7 @@ enum cbm_type {
PSR_CBM_TYPE_L3,
PSR_CBM_TYPE_L3_CODE,
PSR_CBM_TYPE_L3_DATA,
+ PSR_CBM_TYPE_L2,
};
extern struct psr_cmt *psr_cmt;
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 00f5e77..cbf5372 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
#define XEN_SYSCTL_PSR_CAT_get_l3_info 0
+#define XEN_SYSCTL_PSR_CAT_get_l2_info 1
struct xen_sysctl_psr_cat_op {
uint32_t cmd; /* IN: XEN_SYSCTL_PSR_CAT_* */
uint32_t target; /* IN */
@@ -754,6 +755,11 @@ struct xen_sysctl_psr_cat_op {
#define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
uint32_t flags; /* OUT: CAT flags */
} l3_info;
+
+ struct {
+ uint32_t cbm_len; /* OUT: CBM length */
+ uint32_t cos_max; /* OUT: Maximum COS */
+ } l2_info;
} u;
};
typedef struct xen_sysctl_psr_cat_op xen_sysctl_psr_cat_op_t;
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 19/24] x86: L2 CAT: implement get value flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (17 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 20/24] x86: L2 CAT: implement set " Yi Sun
` (4 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements L2 CAT get value callback function and
interface in domctl.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/domctl.c | 7 +++++++
xen/arch/x86/psr.c | 12 ++++++++++++
xen/include/public/domctl.h | 1 +
3 files changed, 20 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index f8f5539..68c2d60 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1460,6 +1460,13 @@ long arch_do_domctl(
copyback = 1;
break;
+ case XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM:
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L2);
+ copyback = 1;
+ break;
+
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5d1c2d5..1db77b1 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -741,9 +741,21 @@ static bool l2_cat_get_feat_info(const struct feat_node *feat,
return true;
}
+static bool l2_cat_get_val(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val)
+{
+ if ( cos > feat->info.l2_cat_info.cos_max )
+ cos = 0;
+
+ *val = feat->cos_reg_val[cos];
+
+ return true;
+}
+
struct feat_ops l2_cat_ops = {
.get_cos_max = l2_cat_get_cos_max,
.get_feat_info = l2_cat_get_feat_info,
+ .get_val = l2_cat_get_val,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 85cbb7c..8c183ba 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
uint64_t data; /* IN/OUT */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 20/24] x86: L2 CAT: implement set value flow.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (18 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 19/24] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 21/24] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
` (3 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements L2 CAT set value related callback functions
and domctl interface.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/domctl.c | 6 +++
xen/arch/x86/psr.c | 92 +++++++++++++++++++++++++++++++++++++++++
xen/include/asm-x86/msr-index.h | 1 +
xen/include/public/domctl.h | 1 +
4 files changed, 100 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 68c2d60..38dc087 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1439,6 +1439,12 @@ long arch_do_domctl(
PSR_CBM_TYPE_L3_DATA);
break;
+ case XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM:
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L2);
+ break;
+
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
ret = psr_get_val(d, domctl->u.psr_cat_op.target,
&domctl->u.psr_cat_op.data,
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 1db77b1..4adc5c8 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -752,10 +752,102 @@ static bool l2_cat_get_val(const struct feat_node *feat, unsigned int cos,
return true;
}
+static unsigned int l2_cat_get_cos_num(const struct feat_node *feat)
+{
+ /* L2 CAT uses one COS. */
+ return 1;
+}
+
+static int l2_cat_get_old_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos)
+{
+ if ( old_cos > feat->info.l2_cat_info.cos_max )
+ /* Use default value. */
+ old_cos = 0;
+
+ val[0] = feat->cos_reg_val[old_cos];
+
+ return 0;
+}
+
+static int l2_cat_set_new_val(uint64_t val[],
+ const struct feat_node *feat,
+ enum cbm_type type,
+ uint64_t m)
+{
+ if ( !psr_check_cbm(feat->info.l2_cat_info.cbm_len, m) )
+ return -EINVAL;
+
+ val[0] = m;
+
+ return 0;
+}
+
+static int l2_cat_compare_val(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos, bool *found)
+{
+ uint64_t l2_def_cbm;
+
+ l2_def_cbm = (1ull << feat->info.l2_cat_info.cbm_len) - 1;
+
+ if ( cos > feat->info.l2_cat_info.cos_max )
+ {
+ if ( val[0] != l2_def_cbm )
+ {
+ *found = false;
+ return -ENOENT;
+ }
+ *found = true;
+ }
+ else
+ *found = (val[0] == feat->cos_reg_val[cos]);
+
+ return 0;
+}
+
+static bool l2_cat_fits_cos_max(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos)
+{
+ uint64_t l2_def_cbm;
+
+ l2_def_cbm = (1ull << feat->info.l2_cat_info.cbm_len) - 1;
+
+ if ( cos > feat->info.l2_cat_info.cos_max &&
+ val[0] != l2_def_cbm )
+ /*
+ * Exceed cos_max and value to set is not default,
+ * return error.
+ */
+ return false;
+
+ return true;
+}
+
+static int l2_cat_write_msr(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat)
+{
+ if ( cos > feat->info.l2_cat_info.cos_max )
+ return -EINVAL;
+
+ feat->cos_reg_val[cos] = val[0];
+ wrmsrl(MSR_IA32_PSR_L2_MASK(cos), val[0]);
+
+ return 0;
+}
+
struct feat_ops l2_cat_ops = {
.get_cos_max = l2_cat_get_cos_max,
.get_feat_info = l2_cat_get_feat_info,
.get_val = l2_cat_get_val,
+ .get_cos_num = l2_cat_get_cos_num,
+ .get_old_val = l2_cat_get_old_val,
+ .set_new_val = l2_cat_set_new_val,
+ .compare_val = l2_cat_compare_val,
+ .fits_cos_max = l2_cat_fits_cos_max,
+ .write_msr = l2_cat_write_msr,
};
static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h
index 98dbff1..a41e63a 100644
--- a/xen/include/asm-x86/msr-index.h
+++ b/xen/include/asm-x86/msr-index.h
@@ -343,6 +343,7 @@
#define MSR_IA32_PSR_L3_MASK(n) (0x00000c90 + (n))
#define MSR_IA32_PSR_L3_MASK_CODE(n) (0x00000c90 + (n) * 2 + 1)
#define MSR_IA32_PSR_L3_MASK_DATA(n) (0x00000c90 + (n) * 2)
+#define MSR_IA32_PSR_L2_MASK(n) (0x00000d10 + (n))
/* Intel Model 6 */
#define MSR_P6_PERFCTR(n) (0x000000c1 + (n))
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 8c183ba..523a2cd 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM 6
#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 21/24] tools: L2 CAT: support get HW info for L2 CAT.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (19 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 20/24] x86: L2 CAT: implement set " Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 22/24] tools: L2 CAT: support show cbm " Yi Sun
` (2 subsequent siblings)
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements xl/xc changes to support get HW info
for L2 CAT.
'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
info.
Example(on machine which only supports L2 CAT):
Cache Monitoring Technology (CMT):
Enabled : 0
Cache Allocation Technology (CAT): L2
Socket ID : 0
Maximum COS : 3
CBM length : 8
Default CBM : 0xff
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/include/xenctrl.h | 6 ++---
tools/libxc/xc_psr.c | 39 +++++++++++++++++++++++---------
tools/libxl/libxl.h | 9 ++++++++
tools/libxl/libxl_psr.c | 19 +++++++++++-----
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 52 +++++++++++++++++++++++++++++++++----------
6 files changed, 95 insertions(+), 31 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 85d7fe5..7308332 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2637,9 +2637,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
xc_psr_cat_type type, uint32_t target,
uint64_t *data);
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled);
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len,
+ bool *cdp_enabled);
int xc_get_cpu_levelling_caps(xc_interface *xch, uint32_t *caps);
int xc_get_cpu_featureset(xc_interface *xch, uint32_t index,
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 43b3286..de40615 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -317,24 +317,41 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
return rc;
}
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled)
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len, bool *cdp_enabled)
{
- int rc;
+ int rc = -1;
DECLARE_SYSCTL;
sysctl.cmd = XEN_SYSCTL_psr_cat_op;
- sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
sysctl.u.psr_cat_op.target = socket;
- rc = xc_sysctl(xch, &sysctl);
- if ( !rc )
+ switch ( lvl )
{
- *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
- *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
- *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
- XEN_SYSCTL_PSR_CAT_L3_CDP;
+ case 2:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l2_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.l2_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.l2_info.cbm_len;
+ *cdp_enabled = false;
+ }
+ break;
+ case 3:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
+ *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
+ XEN_SYSCTL_PSR_CAT_L3_CDP;
+ }
+ break;
+ default:
+ errno = EOPNOTSUPP;
+ break;
}
return rc;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 3924464..c75a928 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -904,6 +904,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, const libxl_mac *src);
* If this is defined, the Code and Data Prioritization feature is supported.
*/
#define LIBXL_HAVE_PSR_CDP 1
+
+/*
+ * LIBXL_HAVE_PSR_L2_CAT
+ *
+ * If this is defined, the L2 Cache Allocation Technology feature is supported.
+ */
+#define LIBXL_HAVE_PSR_L2_CAT 1
#endif
/*
@@ -2166,6 +2173,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
* On success, the function returns an array of elements in 'info',
* and the length in 'nr'.
*/
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl);
int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
int *nr);
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr);
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index ec5c79d..68c45da 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -352,8 +352,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
return rc;
}
-int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
- int *nr)
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl)
{
GC_INIT(ctx);
int rc;
@@ -380,9 +380,8 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
libxl_for_each_set_bit(socketid, socketmap) {
ptr[i].id = socketid;
- if (xc_psr_cat_get_l3_info(ctx->xch, socketid, &ptr[i].cos_max,
- &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
- libxl__psr_cat_log_err_msg(gc, errno);
+ if (xc_psr_cat_get_info(ctx->xch, socketid, lvl, &ptr[i].cos_max,
+ &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
rc = ERROR_FAIL;
free(ptr);
goto out;
@@ -398,6 +397,16 @@ out:
return rc;
}
+int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr)
+{
+ int rc;
+
+ rc = libxl_psr_cat_get_info(ctx, info, nr, 3);
+
+ return rc;
+}
+
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr)
{
int i;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a612d1f..5a401b8 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -899,6 +899,7 @@ libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
(1, "L3_CBM"),
(2, "L3_CBM_CODE"),
(3, "L3_CBM_DATA"),
+ (4, "L2_CBM"),
])
libxl_psr_cat_info = Struct("psr_cat_info", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 358757f..062709f 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9338,21 +9338,19 @@ int main_psr_cmt_show(int argc, char **argv)
}
#endif
-#ifdef LIBXL_HAVE_PSR_CAT
-static int psr_cat_hwinfo(void)
+#if defined(LIBXL_HAVE_PSR_CAT) || defined(LIBXL_HAVE_PSR_L2_CAT)
+static int psr_l3_cat_hwinfo(void)
{
- int rc;
- int i, nr;
+ int rc, nr;
+ unsigned int i;
uint32_t l3_cache_size;
libxl_psr_cat_info *info;
- printf("Cache Allocation Technology (CAT):\n");
-
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
- if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (rc)
return rc;
- }
+
+ printf("Cache Allocation Technology (CAT): L3\n");
for (i = 0; i < nr; i++) {
rc = libxl_psr_cmt_get_l3_cache_size(ctx, info[i].id, &l3_cache_size);
@@ -9461,7 +9459,7 @@ static int psr_cat_show(uint32_t domid)
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
if (rc) {
fprintf(stderr, "Failed to get cat info\n");
return rc;
@@ -9478,6 +9476,32 @@ out:
return rc;
}
+static int psr_l2_cat_hwinfo(void)
+{
+ int rc;
+ unsigned int i;
+ int nr;
+ libxl_psr_cat_info *info;
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 2);
+ if (rc)
+ return rc;
+
+ printf("Cache Allocation Technology (CAT): L2\n");
+
+ for (i = 0; i < nr; i++) {
+ /* There is no CMT on L2 cache so far. */
+ printf("%-16s: %u\n", "Socket ID", info[i].id);
+ printf("%-16s: %u\n", "Maximum COS", info[i].cos_max);
+ printf("%-16s: %u\n", "CBM length", info[i].cbm_len);
+ printf("%-16s: %#llx\n", "Default CBM",
+ (1ull << info[i].cbm_len) - 1);
+ }
+
+ libxl_psr_cat_info_list_free(info, nr);
+ return rc;
+}
+
int main_psr_cat_cbm_set(int argc, char **argv)
{
uint32_t domid;
@@ -9595,7 +9619,11 @@ int main_psr_hwinfo(int argc, char **argv)
ret = psr_cmt_hwinfo();
if (!ret && (all || cat))
- ret = psr_cat_hwinfo();
+ ret = psr_l3_cat_hwinfo();
+
+ /* L2 CAT is independent of CMT and L3 CAT */
+ if (all || cat)
+ ret = psr_l2_cat_hwinfo();
return ret;
}
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 22/24] tools: L2 CAT: support show cbm for L2 CAT.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (20 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 21/24] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 23/24] tools: L2 CAT: support set " Yi Sun
2017-02-08 8:16 ` [PATCH v6 24/24] docs: add L2 CAT description in docs Yi Sun
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements changes in xl/xc changes to support
showing CBM of L2 CAT.
The new level option is introduced to original CAT showing
command in order to show CBM for specified level CAT.
- 'xl psr-cat-show' is updated to show CBM of a domain
according to input cache level.
Examples:
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v6:
- check if input level is correct.
---
tools/libxc/include/xenctrl.h | 1 +
tools/libxc/xc_psr.c | 3 ++
tools/libxl/xl_cmdimpl.c | 85 +++++++++++++++++++++++++++++--------------
tools/libxl/xl_cmdtable.c | 3 +-
4 files changed, 63 insertions(+), 29 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 7308332..0e2c2d8 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2613,6 +2613,7 @@ enum xc_psr_cat_type {
XC_PSR_CAT_L3_CBM = 1,
XC_PSR_CAT_L3_CBM_CODE = 2,
XC_PSR_CAT_L3_CBM_DATA = 3,
+ XC_PSR_CAT_L2_CBM = 4,
};
typedef enum xc_psr_cat_type xc_psr_cat_type;
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index de40615..6f114fb 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -299,6 +299,9 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 062709f..b515e7d 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9386,7 +9386,7 @@ static void psr_cat_print_one_domain_cbm_type(uint32_t domid, uint32_t socketid,
}
static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
char *domain_name;
@@ -9394,27 +9394,38 @@ static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
printf("%5d%25s", domid, domain_name);
free(domain_name);
- if (!cdp_enabled) {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM);
- } else {
- psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ switch (lvl) {
+ case 3:
+ if (!cdp_enabled) {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM);
+ } else {
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ }
+ break;
+ case 2:
psr_cat_print_one_domain_cbm_type(domid, socketid,
- LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+ LIBXL_PSR_CBM_TYPE_L2_CBM);
+ break;
+ default:
+ printf("Input lvl %d is wrong!", lvl);
+ break;
}
printf("\n");
}
static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+ bool cdp_enabled, unsigned int lvl)
{
int i, nr_domains;
libxl_dominfo *list;
if (domid != INVALID_DOMID) {
- psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled, lvl);
return 0;
}
@@ -9424,49 +9435,59 @@ static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
}
for (i = 0; i < nr_domains; i++)
- psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
+ psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled, lvl);
libxl_dominfo_list_free(list, nr_domains);
return 0;
}
-static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
+static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info,
+ unsigned int lvl)
{
int rc;
uint32_t l3_cache_size;
- rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
- if (rc) {
- fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
- info->id);
- return -1;
+ printf("%-16s: %u\n", "Socket ID", info->id);
+
+ /* So far, CMT only supports L3 cache. */
+ if (lvl == 3) {
+ rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
+ if (rc) {
+ fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
+ info->id);
+ return -1;
+ }
+ printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
}
- printf("%-16s: %u\n", "Socket ID", info->id);
- printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
printf("%-16s: %#llx\n", "Default CBM", (1ull << info->cbm_len) - 1);
if (info->cdp_enabled)
printf("%5s%25s%16s%16s\n", "ID", "NAME", "CBM (code)", "CBM (data)");
else
printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
- return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled);
+ return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled, lvl);
}
-static int psr_cat_show(uint32_t domid)
+static int psr_cat_show(uint32_t domid, unsigned int lvl)
{
int i, nr;
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+ if (lvl != 2 && lvl != 3) {
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
+ }
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, lvl);
if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ fprintf(stderr, "Failed to get %s cat info\n", (lvl == 3)?"L3":"L2");
return rc;
}
for (i = 0; i < nr; i++) {
- rc = psr_cat_print_socket(domid, info + i);
+ rc = psr_cat_print_socket(domid, info + i, lvl);
if (rc)
goto out;
}
@@ -9577,11 +9598,19 @@ int main_psr_cat_cbm_set(int argc, char **argv)
int main_psr_cat_show(int argc, char **argv)
{
- int opt;
+ int opt = 0;
uint32_t domid;
+ unsigned int lvl = 3;
- SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cat-show", 0) {
- /* No options */
+ static struct option opts[] = {
+ {"level", 1, 0, 'l'},
+ COMMON_LONG_OPTS
+ };
+
+ SWITCH_FOREACH_OPT(opt, "l:", opts, "psr-cat-show", 0) {
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
if (optind >= argc)
@@ -9593,7 +9622,7 @@ int main_psr_cat_show(int argc, char **argv)
return 2;
}
- return psr_cat_show(domid);
+ return psr_cat_show(domid, lvl);
}
int main_psr_hwinfo(int argc, char **argv)
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index 588d5d9..c5fbad4 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -556,7 +556,8 @@ struct cmd_spec cmd_table[] = {
{ "psr-cat-show",
&main_psr_cat_show, 0, 1,
"Show Cache Allocation Technology information",
- "<Domain>",
+ "[options] <Domain>",
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
},
#endif
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 23/24] tools: L2 CAT: support set cbm for L2 CAT.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (21 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 22/24] tools: L2 CAT: support show cbm " Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
2017-02-08 8:16 ` [PATCH v6 24/24] docs: add L2 CAT description in docs Yi Sun
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch implements the xl/xc changes to support set CBM
for L2 CAT.
The new level option is introduced to original CAT setting
command in order to set CBM for specified level CAT.
- 'xl psr-cat-cbm-set' is updated to set cache capacity
bitmasks(CBM) for a domain according to input cache level.
root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v6:
- rename 'psr-cat-cbm-set' to 'psr-cat-set' per Kevin's comment.
- return 'EXIT_FAILURE' for error case.
- print error info when input level is wrong.
---
tools/libxc/xc_psr.c | 3 +++
tools/libxl/xl_cmdimpl.c | 34 +++++++++++++++++++++++-----------
tools/libxl/xl_cmdtable.c | 3 ++-
3 files changed, 28 insertions(+), 12 deletions(-)
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 6f114fb..8fd6812 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -266,6 +266,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index b515e7d..9d80a5d 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9534,19 +9534,21 @@ int main_psr_cat_cbm_set(int argc, char **argv)
char *value;
libxl_string_list socket_list;
unsigned long start, end;
- int i, j, len;
+ unsigned int i, j, len;
+ unsigned int lvl = 3;
static struct option opts[] = {
{"socket", 1, 0, 's'},
{"data", 0, 0, 'd'},
{"code", 0, 0, 'c'},
+ {"level", 1, 0, 'l'},
COMMON_LONG_OPTS
};
libxl_socket_bitmap_alloc(ctx, &target_map, 0);
libxl_bitmap_set_none(&target_map);
- SWITCH_FOREACH_OPT(opt, "s:cd", opts, "psr-cat-cbm-set", 2) {
+ SWITCH_FOREACH_OPT(opt, "s:l:cd", opts, "psr-cat-set", 2) {
case 's':
trim(isspace, optarg, &value);
split_string_into_string_list(value, ",", &socket_list);
@@ -9566,24 +9568,34 @@ int main_psr_cat_cbm_set(int argc, char **argv)
case 'c':
opt_code = 1;
break;
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
- if (opt_data && opt_code) {
- fprintf(stderr, "Cannot handle -c and -d at the same time\n");
- return -1;
- } else if (opt_data) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
- } else if (opt_code) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ if (lvl == 2)
+ type = LIBXL_PSR_CBM_TYPE_L2_CBM;
+ else if (lvl == 3) {
+ if (opt_data && opt_code) {
+ fprintf(stderr, "Cannot handle -c and -d at the same time\n");
+ return EXIT_FAILURE;
+ } else if (opt_data) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
+ } else if (opt_code) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ } else {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ }
} else {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+ return EXIT_FAILURE;
}
if (libxl_bitmap_is_empty(&target_map))
libxl_bitmap_set_any(&target_map);
if (argc != optind + 2) {
- help("psr-cat-cbm-set");
+ help("psr-cat-set");
return 2;
}
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index c5fbad4..433ed6a 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -545,11 +545,12 @@ struct cmd_spec cmd_table[] = {
},
#endif
#ifdef LIBXL_HAVE_PSR_CAT
- { "psr-cat-cbm-set",
+ { "psr-cat-set",
&main_psr_cat_cbm_set, 0, 1,
"Set cache capacity bitmasks(CBM) for a domain",
"[options] <Domain> <CBM>",
"-s <socket> Specify the socket to process, otherwise all sockets are processed\n"
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
"-c Set code CBM if CDP is supported\n"
"-d Set data CBM if CDP is supported\n"
},
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v6 24/24] docs: add L2 CAT description in docs.
2017-02-08 8:15 [PATCH v6 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
` (22 preceding siblings ...)
2017-02-08 8:16 ` [PATCH v6 23/24] tools: L2 CAT: support set " Yi Sun
@ 2017-02-08 8:16 ` Yi Sun
23 siblings, 0 replies; 37+ messages in thread
From: Yi Sun @ 2017-02-08 8:16 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng
This patch adds L2 CAT description in related documents.
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
docs/man/xl.pod.1.in | 25 ++++++++++++++++++++++---
docs/misc/xl-psr.markdown | 10 ++++++++--
2 files changed, 30 insertions(+), 5 deletions(-)
diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 7caed08..5e7676e 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -1711,6 +1711,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
a domain is attached to the monitoring service, monitoring data can be shown
for any of these monitoring types.
+There is no cache monitoring and memory bandwidth monitoring on L2 cache so
+far.
+
=over 4
=item B<psr-cmt-attach> [I<domain-id>]
@@ -1735,7 +1738,7 @@ monitor types are:
Intel Broadwell and later server platforms offer capabilities to configure and
make use of the Cache Allocation Technology (CAT) mechanisms, which enable more
-cache resources (i.e. L3 cache) to be made available for high priority
+cache resources (i.e. L3/L2 cache) to be made available for high priority
applications. In the Xen implementation, CAT is used to control cache allocation
on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
(CBM) for the domain.
@@ -1745,7 +1748,7 @@ Intel Broadwell and later server platforms also offer Code/Data Prioritization
applications. CDP is used on a per VM basis in the Xen implementation. To
specify code or data CBM for the domain, CDP feature must be enabled and CBM
type options need to be specified when setting CBM, and the type options (code
-and data) are mutually exclusive.
+and data) are mutually exclusive. There is no CDP support on L2 so far.
=over 4
@@ -1762,6 +1765,11 @@ B<OPTIONS>
Specify the socket to process, otherwise all sockets are processed.
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
=item B<-c>, B<--code>
Set code CBM when CDP is enabled.
@@ -1772,10 +1780,21 @@ Set data CBM when CDP is enabled.
=back
-=item B<psr-cat-show> [I<domain-id>]
+=item B<psr-cat-show> [I<OPTIONS>] [I<domain-id>]
Show CAT settings for a certain domain or all domains.
+B<OPTIONS>
+
+=over 4
+
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
+=back
+
=back
=head1 IGNORED FOR COMPATIBILITY WITH XM
diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
index c3c1e8e..bd2b6bd 100644
--- a/docs/misc/xl-psr.markdown
+++ b/docs/misc/xl-psr.markdown
@@ -70,7 +70,7 @@ total-mem-bandwidth instead of cache-occupancy). E.g. after a `xl psr-cmt-attach
Cache Allocation Technology (CAT) is a new feature available on Intel
Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
-partition cache allocation (i.e. L3 cache) based on application priority or
+partition cache allocation (i.e. L3/L2 cache) based on application priority or
Class of Service (COS). Each COS is configured using capacity bitmasks (CBM)
which represent cache capacity and indicate the degree of overlap and
isolation between classes. System cache resource is divided into numbers of
@@ -119,13 +119,19 @@ A cbm is valid only when:
In a multi-socket system, the same cbm will be set on each socket by default.
Per socket cbm can be specified with the `--socket SOCKET` option.
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
+
Setting the CBM may not be successful if insufficient COS is available. In
such case unused COS(es) may be freed by setting CBM of all related domains to
its default value(all-ones).
Per domain CBM settings can be shown by:
-`xl psr-cat-show`
+`xl psr-cat-show [OPTIONS] <domid>`
+
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
## Code and Data Prioritization (CDP)
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 37+ messages in thread