* [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-10-25 13:37 ` Jan Beulich
` (3 more replies)
2016-10-25 3:40 ` [PATCH v3 02/15] x86: refactor psr: Split 'ref' out Yi Sun
` (14 subsequent siblings)
15 siblings, 4 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 314 insertions(+)
create mode 100644 docs/features/l2_cat.pandoc
diff --git a/docs/features/l2_cat.pandoc b/docs/features/l2_cat.pandoc
new file mode 100644
index 0000000..8544510
--- /dev/null
+++ b/docs/features/l2_cat.pandoc
@@ -0,0 +1,314 @@
+% Intel L2 Cache Allocation Technology (L2 CAT) Feature
+% Revision 2.0
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+ Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+ Component(s): Hypervisor, toolstack
+
+ Hardware: Atom codename Goldmont and beyond
+---------------- ----------------------------------------------------
+
+# Overview
+
+L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
+CPU's shared L2 cache based on application priority or Class of Service
+(COS). Each CLOS is configured using capacity bitmasks (CBM) which
+represent cache capacity and indicate the degree of overlap and
+isolation between classes. Once L2 CAT is configured, the processor
+allows access to portions of L2 cache according to the established
+class of service (COS).
+
+# Technical information
+
+L2 CAT is a member of Intel PSR features and part of CAT, it shares
+some base PSR infrastructure in Xen.
+
+## Hardware perspective
+
+L2 CAT defines a new range MSRs to assign different L2 cache access
+patterns which are known as CBMs (Capacity BitMask), each CBM is
+associated with a COS.
+
+```
+
+ +----------------------------+----------------+
+ IA32_PQR_ASSOC | MSR (per socket) | Address |
+ +----+---+-------+ +----------------------------+----------------+
+ | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
+ +----+---+-------+ +----------------------------+----------------+
+ └-------------> | ... | ... |
+ +----------------------------+----------------+
+ | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
+ +----------------------------+----------------+
+```
+
+When context switch happens, the COS of VCPU is written to per-thread
+MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
+according to the corresponding CBM.
+
+## The relationship between L2 CAT and L3 CAT/CDP
+
+L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
+while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.
+
+L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
+the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
+patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of
+L2 CAT must be continuous too.
+
+N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
+associate register `IA32_PQR_ASSOC`, which means one COS associate to a
+pair of L2 CBM and L3 CBM.
+Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
+other PSR features in future). In some cases, a VM is permitted to have a
+COS that is beyond one (or more) of PSR features but within the others.
+For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
+L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
+COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
+limit) since COS 9 is beyond the max COS (8) of L2 CAT.
+
+## Design Overview
+
+* Core COS/CBM association
+
+ When enforcing L2 CAT, all cores of domains have the same default
+ COS (COS0) which associated to the fully open CBM (all ones bitmask)
+ to access all L2 cache. The default COS is used only in hypervisor
+ and is transparent to tool stack and user.
+
+ System administrator can change PSR allocation policy at runtime by
+ tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds
+ to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP
+ is enabled, one COS corresponds to a 3-tuple, like [L2 CBM,
+ L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled,
+ things would be easier, one COS corresponds to one L2 CBM.
+
+* VCPU schedule
+
+ This part reuses L3 CAT COS infrastructure.
+
+* Multi-sockets
+
+ Different sockets may have different L2 CAT capability (e.g. max COS)
+ although it is consistent on the same socket. So the capability of
+ per-socket L2 CAT is specified.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+ 1. Ext: Boot line parameter "psr=cat" now will enable L2 CAT and L3
+ CAT if hardware supported.
+
+ 2. New: SYSCTL:
+ - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
+
+ 3. New: DOMCTL:
+ - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
+ - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
+
+* xl interfaces:
+
+ 1. Ext: psr-cat-show -l2 domain-id
+ Show L2 cbm for a domain.
+ => XEN_SYSCTL_PSR_CAT_get_l2_info /
+ XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
+
+ 2. Ext: psr-mba-set -l2 domain-id cbm
+ Set L2 cbm for a domain.
+ => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
+
+ 3. Ext: psr-hwinfo
+ Show PSR HW information, including L2 CAT
+ => XEN_SYSCTL_PSR_CAT_get_l2_info
+
+* Key data structure:
+
+ 1. Feature HW info
+
+ ```
+ struct psr_cat_hw_info {
+ unsigned int cbm_len;
+ unsigned int cos_max;
+ };
+ ```
+
+ - Member `cbm_len`
+
+ `cbm_len` is one of the hardware info of CAT.
+
+ - Member `cos_max`
+
+ `cos_max` is one of the hardware info of CAT.
+
+ 2. Feature list node
+
+ ```
+ struct feat_node {
+ enum psr_feat_type feature;
+ struct feat_ops ops;
+ struct psr_cat_hw_info info;
+ uint64_t cos_reg_val[MAX_COS_REG_NUM];
+ struct list_head list;
+ };
+ ```
+
+ When a PSR enforcement feature is enabled, it will be added into a
+ feature list. The head of the list is created in psr initialization.
+
+ - Member `feature`
+
+ `feature` is an integer number, to indicate which feature the list entry
+ corresponds to.
+
+ - Member `ops`
+
+ `ops` maintains a callback function list of the feature. It will be introduced
+ in details later.
+
+ - Member `info`
+
+ `info` maintains the feature HW information which can be got through
+ psr_hwinfo command.
+
+ - Member `cos_reg_val`
+
+ `cos_reg_val` is an array to maintain the value set in all COS registers of
+ the feature.
+
+ 3. Per-socket PSR features information structure
+
+ ```
+ struct psr_cat_socket_info {
+ unsigned int feat_mask;
+ unsigned int nr_feat;
+ struct list_head feat_list;
+ unsigned int cos_ref[MAX_COS_REG_NUM];
+ spinlock_t ref_lock;
+ };
+ ```
+
+ We collect all PSR allocation features information of a socket in
+ this `struct psr_cat_socket_info`.
+
+ - Member `feat_mask`
+
+ `feat_mask` is a bitmap, to indicate which feature is enabled on
+ current socket. We define `feat_mask` bitmap as:
+
+ bit 0~1: L3 CAT status, [01] stands for L3 CAT only and [10]
+ stands for L3 CDP is enalbed.
+
+ bit 2: L2 CAT status.
+
+ - Member `cos_ref`
+
+ `cos_ref` is an array which maintains the reference of one COS.
+ If the COS is used by one domain, the reference will increase one.
+ If a domain releases the COS, the reference will decrease one. The
+ array is indexed by COS.
+
+ 4. Feature operation functions structure
+
+ ```
+ struct feat_ops {
+ void (*init_feature)(unsigned int eax, unsigned int ebx,
+ unsigned int ecx, unsigned int edx,
+ struct feat_node *feat,
+ struct psr_cat_socket_info *info);
+ int (*get_feat_info)(const struct feat_node *feat, enum cbm_type type,
+ uint32_t dat[], uint32_t array_len);
+ int (*get_val)(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val);
+ unsigned int (*get_max_cos_max)(const struct feat_node *feat);
+ unsigned int (*get_cos_num)(const struct feat_node *feat);
+ int (*get_old_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos);
+ int (*set_new_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos,
+ enum cbm_type type,
+ uint64_t m);
+ int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
+ unsigned int cos, bool *found);
+ unsigned int (*get_cos_max_from_type)(const struct feat_node *feat,
+ enum cbm_type type);
+ unsigned int (*exceeds_cos_max)(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos);
+ int (*write_msr)(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat);
+ };
+ ```
+
+ We abstract above callback functions to encapsulate the feature specific
+ behaviors into them. Then, it is easy to add a new feature. We just need:
+ 1) Implement such ops and callback functions for every feature.
+ 2) Register the ops into `struct feat_node`.
+ 3) Add the feature into feature list during CPU initialization.
+
+# User information
+
+* Feature Enabling:
+
+ Add "psr=cat" to boot line parameter to enable all supported level CAT
+ features.
+
+* xl interfaces:
+
+ 1. `psr-cat-show [OPTIONS] domain-id`:
+
+ Show domain L2 or L3 CAT CBM.
+
+ New option `-l` is added.
+ `-l2`: Specify cbm for L2 cache.
+ `-l3`: Specify cbm for L3 cache.
+
+ If neither `-l2` nor `-l3` is given, level 3 is the default option.
+
+ 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
+
+ Set domain L2 or L3 CBM.
+
+ New option `-l` is added.
+ `-l2`: Specify cbm for L2 cache.
+ `-l3`: Specify cbm for L3 cache.
+
+ If neither `-l2` nor `-l3` is given, level 3 is the default option.
+
+ 3. `psr-hwinfo [OPTIONS]`:
+
+ Show L2 & L3 CAT HW informations on every socket.
+
+# References
+
+[Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3, chapter 17.17](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date Revision Version Notes
+---------- -------- -------- -------------------------------------------
+2016-08-12 1.0 Xen 4.7 Initial design
+2016-09-21 2.0 Xen 4.7 Changes according to review comments.
+ 1. L2 CBM bits should be continuous too.
+ 2. Describe 'struct feat_info'
+ 3. Update 'struct feat_list" description.
+ 4. Update 'struct feat_ops" description.
+ 5. Update `psr-cat-show` description.
+ 6. Update `psr-cat-cbm-set` description.
+ 7. Add volume and chapter info in References.
+2016-10-24 3.0 Xen 4.7 Changes according to review comments.
+ 1. Update 'struct feat_list' to 'struct feat_node'.
+ 2. Update description of 'struct psr_cat_socket_info'.
+ 3. Update `psr-cat-show` description.
+ 4. Update `psr-cat-cbm-set` description.
+---------- -------- -------- -------------------------------------------
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-10-25 3:40 ` [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document Yi Sun
@ 2016-10-25 13:37 ` Jan Beulich
2016-10-26 1:01 ` Yi Sun
2016-10-30 15:51 ` Meng Xu
` (2 subsequent siblings)
3 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2016-10-25 13:37 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
>>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote:
> +# References
> +
> +[Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3, chapter 17.17]
> (http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
Please don't refer to SDM chapters by number. The numbering
changes too frequently, while section titles a reasonably stable.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-10-25 13:37 ` Jan Beulich
@ 2016-10-26 1:01 ` Yi Sun
0 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-26 1:01 UTC (permalink / raw)
To: Jan Beulich
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
On 16-10-25 07:37:24, Jan Beulich wrote:
> >>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote:
> > +# References
> > +
> > +[Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3, chapter 17.17]
> > (http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
>
> Please don't refer to SDM chapters by number. The numbering
> changes too frequently, while section titles a reasonably stable.
>
Got it, thanks!
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-10-25 3:40 ` [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document Yi Sun
2016-10-25 13:37 ` Jan Beulich
@ 2016-10-30 15:51 ` Meng Xu
2016-11-01 4:40 ` Yi Sun
2016-11-11 21:33 ` Konrad Rzeszutek Wilk
2016-11-25 17:39 ` Dario Faggioli
3 siblings, 1 reply; 42+ messages in thread
From: Meng Xu @ 2016-10-30 15:51 UTC (permalink / raw)
To: Yi Sun
Cc: Wei Liu, he.chen, Andrew Cooper, Ian Jackson, Jan Beulich,
Chao Peng, xen-devel
Hi Yi,
On Mon, Oct 24, 2016 at 11:40 PM, Yi Sun <yi.y.sun@linux.intel.com> wrote:
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 314 insertions(+)
> create mode 100644 docs/features/l2_cat.pandoc
>
> diff --git a/docs/features/l2_cat.pandoc b/docs/features/l2_cat.pandoc
> new file mode 100644
> index 0000000..8544510
> --- /dev/null
> +++ b/docs/features/l2_cat.pandoc
> @@ -0,0 +1,314 @@
> +% Intel L2 Cache Allocation Technology (L2 CAT) Feature
> +% Revision 2.0
> +
> +\clearpage
> +
> +# Basics
> +
> +---------------- ----------------------------------------------------
> + Status: **Tech Preview**
> +
> +Architecture(s): Intel x86
> +
> + Component(s): Hypervisor, toolstack
> +
> + Hardware: Atom codename Goldmont and beyond
> +---------------- ----------------------------------------------------
I'm interested in trying out your code.
I'm planning to purchase a SoC with the Atom Goldmont processors.
Do you have some suggestions about the SoC I should purchase?
I would prefer to use the same SoC as you have. :-)
Thank you very much!
Meng
-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-10-30 15:51 ` Meng Xu
@ 2016-11-01 4:40 ` Yi Sun
0 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-11-01 4:40 UTC (permalink / raw)
To: Meng Xu
Cc: Wei Liu, he.chen, Andrew Cooper, Ian Jackson, Jan Beulich,
Chao Peng, xen-devel
Hi, Xu,
On 16-10-30 11:51:35, Meng Xu wrote:
> Hi Yi,
>
> On Mon, Oct 24, 2016 at 11:40 PM, Yi Sun <yi.y.sun@linux.intel.com> wrote:
> > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> > ---
> > docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 314 insertions(+)
> > create mode 100644 docs/features/l2_cat.pandoc
> >
> > +---------------- ----------------------------------------------------
> > + Status: **Tech Preview**
> > +
> > +Architecture(s): Intel x86
> > +
> > + Component(s): Hypervisor, toolstack
> > +
> > + Hardware: Atom codename Goldmont and beyond
> > +---------------- ----------------------------------------------------
>
> I'm interested in trying out your code.
> I'm planning to purchase a SoC with the Atom Goldmont processors.
> Do you have some suggestions about the SoC I should purchase?
> I would prefer to use the same SoC as you have. :-)
>
> Thank you very much!
>
> Meng
>
Thank you for being interesting in this feature! The CPU name is Denverton.
Thanks,
Sun Yi
> -----------
> Meng Xu
> PhD Student in Computer and Information Science
> University of Pennsylvania
> http://www.cis.upenn.edu/~mengxu/
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-10-25 3:40 ` [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document Yi Sun
2016-10-25 13:37 ` Jan Beulich
2016-10-30 15:51 ` Meng Xu
@ 2016-11-11 21:33 ` Konrad Rzeszutek Wilk
2016-11-14 2:15 ` Yi Sun
2016-11-25 17:19 ` Dario Faggioli
2016-11-25 17:39 ` Dario Faggioli
3 siblings, 2 replies; 42+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-11-11 21:33 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
chao.p.peng, xen-devel
On Tue, Oct 25, 2016 at 11:40:49AM +0800, Yi Sun wrote:
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 314 insertions(+)
> create mode 100644 docs/features/l2_cat.pandoc
>
> diff --git a/docs/features/l2_cat.pandoc b/docs/features/l2_cat.pandoc
> new file mode 100644
> index 0000000..8544510
> --- /dev/null
> +++ b/docs/features/l2_cat.pandoc
> @@ -0,0 +1,314 @@
> +% Intel L2 Cache Allocation Technology (L2 CAT) Feature
> +% Revision 2.0
> +
> +\clearpage
> +
> +# Basics
> +
> +---------------- ----------------------------------------------------
> + Status: **Tech Preview**
> +
> +Architecture(s): Intel x86
> +
> + Component(s): Hypervisor, toolstack
> +
> + Hardware: Atom codename Goldmont and beyond
> +---------------- ----------------------------------------------------
> +
> +# Overview
> +
> +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
Could you define CAT?
> +CPU's shared L2 cache based on application priority or Class of Service
> +(COS). Each CLOS is configured using capacity bitmasks (CBM) which
> +represent cache capacity and indicate the degree of overlap and
> +isolation between classes. Once L2 CAT is configured, the processor
> +allows access to portions of L2 cache according to the established
> +class of service (COS).
> +
> +# Technical information
> +
> +L2 CAT is a member of Intel PSR features and part of CAT, it shares
Could you define 'PSR' here? Usually when you introduce an acronym
you do something like:
Intel Problem Solver Resolver (PSR)
and that makes it easy for folks to map the acronym to the full feature.
> +some base PSR infrastructure in Xen.
> +
> +## Hardware perspective
> +
> +L2 CAT defines a new range MSRs to assign different L2 cache access
> +patterns which are known as CBMs (Capacity BitMask), each CBM is
> +associated with a COS.
> +
> +```
> +
> + +----------------------------+----------------+
> + IA32_PQR_ASSOC | MSR (per socket) | Address |
> + +----+---+-------+ +----------------------------+----------------+
> + | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
> + +----+---+-------+ +----------------------------+----------------+
> + └-------------> | ... | ... |
> + +----------------------------+----------------+
> + | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
> + +----------------------------+----------------+
> +```
> +
> +When context switch happens, the COS of VCPU is written to per-thread
> +MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
> +according to the corresponding CBM.
> +
> +## The relationship between L2 CAT and L3 CAT/CDP
> +
> +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
Could you define CDP?
> +while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.
> +
> +L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
> +the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
> +patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of
> +L2 CAT must be continuous too.
> +
> +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
> +associate register `IA32_PQR_ASSOC`, which means one COS associate to a
s/COS associate/COS associate's/ ?
> +pair of L2 CBM and L3 CBM.
> +Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
> +other PSR features in future). In some cases, a VM is permitted to have a
> +COS that is beyond one (or more) of PSR features but within the others.
> +For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
> +L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
> +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
> +limit) since COS 9 is beyond the max COS (8) of L2 CAT.
Thanks for the explanation.
..snip..
[didnt' have any questions below]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-11-11 21:33 ` Konrad Rzeszutek Wilk
@ 2016-11-14 2:15 ` Yi Sun
2016-11-25 17:19 ` Dario Faggioli
1 sibling, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-11-14 2:15 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
chao.p.peng, xen-devel
On 16-11-11 16:33:09, Konrad Rzeszutek Wilk wrote:
> On Tue, Oct 25, 2016 at 11:40:49AM +0800, Yi Sun wrote:
> > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> > ---
> > docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 314 insertions(+)
> > create mode 100644 docs/features/l2_cat.pandoc
> >
> > +# Overview
> > +
> > +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
>
> Could you define CAT?
>
Sure, thanks!
> > +# Technical information
> > +
> > +L2 CAT is a member of Intel PSR features and part of CAT, it shares
>
> Could you define 'PSR' here? Usually when you introduce an acronym
> you do something like:
>
> Intel Problem Solver Resolver (PSR)
>
> and that makes it easy for folks to map the acronym to the full feature.
>
Thanks for tips!
> > +## The relationship between L2 CAT and L3 CAT/CDP
> > +
> > +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
>
> Could you define CDP?
>
Sure, thanks!
> > +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
> > +associate register `IA32_PQR_ASSOC`, which means one COS associate to a
>
> s/COS associate/COS associate's/ ?
>
Thanks!
> > +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
> > +limit) since COS 9 is beyond the max COS (8) of L2 CAT.
>
> Thanks for the explanation.
>
>
> ..snip..
> [didnt' have any questions below]
Thanks a lot for review!
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-11-11 21:33 ` Konrad Rzeszutek Wilk
2016-11-14 2:15 ` Yi Sun
@ 2016-11-25 17:19 ` Dario Faggioli
2016-11-29 5:20 ` Yi Sun
1 sibling, 1 reply; 42+ messages in thread
From: Dario Faggioli @ 2016-11-25 17:19 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
xen-devel, chao.p.peng
[-- Attachment #1.1: Type: text/plain, Size: 3522 bytes --]
On Fri, 2016-11-11 at 16:33 -0500, Konrad Rzeszutek Wilk wrote:
> On Tue, Oct 25, 2016 at 11:40:49AM +0800, Yi Sun wrote:
> > --- /dev/null
> > +++ b/docs/features/l2_cat.pandoc
> > @@ -0,0 +1,314 @@
> > +% Intel L2 Cache Allocation Technology (L2 CAT) Feature
> > +% Revision 2.0
> > +
> > +\clearpage
> > +
> > +# Basics
> > +
> > +---------------- -----------------------------------------------
> > -----
> > + Status: **Tech Preview**
> > +
> > +Architecture(s): Intel x86
> > +
> > + Component(s): Hypervisor, toolstack
> > +
> > + Hardware: Atom codename Goldmont and beyond
>
Atom codename Goldmont and beyond CPUs
It may sound obvious, but I'd explicitly add that bit.
> > +---------------- -----------------------------------------------
> > -----
> > +
> > +# Overview
> > +
> > +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
>
> Could you define CAT?
>
> >
> > +CPU's shared L2 cache based on application priority or Class of
> > Service
> > +(COS). Each CLOS is configured using capacity bitmasks (CBM) which
> > +represent cache capacity and indicate the degree of overlap and
> > +isolation between classes. Once L2 CAT is configured, the
> > processor
> > +allows access to portions of L2 cache according to the established
> > +class of service (COS).
> > +
> > +# Technical information
> > +
> > +L2 CAT is a member of Intel PSR features and part of CAT, it
> > shares
>
> Could you define 'PSR' here? Usually when you introduce an acronym
> you do something like:
>
> Intel Problem Solver Resolver (PSR)
>
Wasn't it the 'Intel Probabilistic Silicon Reorganizer' ? :-D :-D
> and that makes it easy for folks to map the acronym to the full
> feature.
>
Actually, given the density of acronyms, I'd say it would be good to
add a "## Terminology" section at the top, and define all of them there
upfront.
Also, I see that you're meaning this to be committed in tree and act as
the L2 CAT feature document. I know that you've been asked to put it in
tree (although, the request was for docs/misc/) and I think it's good
to have a feature document for L2 CAT.
It contains a lot more technical details than the other (few) feature
documents we have in tree right now. Personally, I'm fine with that,
but I'd say that at least try to filling these sections would be
important:
(from docs/features/template.pandoc)
# Limitations
Information concerning incompatibilities with other features or
hardware combinations.
# Testing
Information concerning how to properly test changes affecting this
feature.
# Areas for improvement
List of enhancements which could be undertaken, e.g. to improve the
feature itself, or improve interaction with other features.
# Known issues
List of known issues or bugs. For tech preview or experimental
features, this section must contain the list of items needing fixing
for its status to be upgraded.
Also, it would be really good to have similar documents for the other
PSR features we have upstream already (perhaps finding a way for not
duplicating all the common information).
Thanks and Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-11-25 17:19 ` Dario Faggioli
@ 2016-11-29 5:20 ` Yi Sun
2016-11-29 12:25 ` Dario Faggioli
0 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2016-11-29 5:20 UTC (permalink / raw)
To: Dario Faggioli
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
xen-devel, chao.p.peng
On 16-11-25 18:19:17, Dario Faggioli wrote:
> On Fri, 2016-11-11 at 16:33 -0500, Konrad Rzeszutek Wilk wrote:
> > On Tue, Oct 25, 2016 at 11:40:49AM +0800, Yi Sun wrote:
> > > --- /dev/null
> > > +++ b/docs/features/l2_cat.pandoc
> > > @@ -0,0 +1,314 @@
> > > +% Intel L2 Cache Allocation Technology (L2 CAT) Feature
> > > +% Revision 2.0
> > > +
> > > +\clearpage
> > > +
> > > +# Basics
> > > +
> > > +---------------- -----------------------------------------------
> > > -----
> > > + Status: **Tech Preview**
> > > +
> > > +Architecture(s): Intel x86
> > > +
> > > + Component(s): Hypervisor, toolstack
> > > +
> > > + Hardware: Atom codename Goldmont and beyond
> >
> Atom codename Goldmont and beyond CPUs
>
> It may sound obvious, but I'd explicitly add that bit.
>
Ok, thanks!
> > > +---------------- -----------------------------------------------
> > > -----
> > > +
> > > +# Overview
> > > +
> > > +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
> >
> > Could you define CAT?
> >
> > >
> > > +CPU's shared L2 cache based on application priority or Class of
> > > Service
> > > +(COS). Each CLOS is configured using capacity bitmasks (CBM) which
> > > +represent cache capacity and indicate the degree of overlap and
> > > +isolation between classes. Once L2 CAT is configured, the
> > > processor
> > > +allows access to portions of L2 cache according to the established
> > > +class of service (COS).
> > > +
> > > +# Technical information
> > > +
> > > +L2 CAT is a member of Intel PSR features and part of CAT, it
> > > shares
> >
> > Could you define 'PSR' here? Usually when you introduce an acronym
> > you do something like:
> >
> > Intel Problem Solver Resolver (PSR)
> >
> Wasn't it the 'Intel Probabilistic Silicon Reorganizer' ? :-D :-D
>
It is 'Intel Platform Shared Resource' indeed. :-)
> > and that makes it easy for folks to map the acronym to the full
> > feature.
> >
> Actually, given the density of acronyms, I'd say it would be good to
> add a "## Terminology" section at the top, and define all of them there
> upfront.
>
Thanks! I will add this section.
> Also, I see that you're meaning this to be committed in tree and act as
> the L2 CAT feature document. I know that you've been asked to put it in
> tree (although, the request was for docs/misc/) and I think it's good
> to have a feature document for L2 CAT.
>
> It contains a lot more technical details than the other (few) feature
> documents we have in tree right now. Personally, I'm fine with that,
> but I'd say that at least try to filling these sections would be
> important:
>
> (from docs/features/template.pandoc)
>
> # Limitations
> Information concerning incompatibilities with other features or
> hardware combinations.
>
> # Testing
> Information concerning how to properly test changes affecting this
> feature.
>
> # Areas for improvement
> List of enhancements which could be undertaken, e.g. to improve the
> feature itself, or improve interaction with other features.
>
> # Known issues
> List of known issues or bugs. For tech preview or experimental
> features, this section must contain the list of items needing fixing
> for its status to be upgraded.
>
Thanks! Will try to add these sections.
> Also, it would be really good to have similar documents for the other
> PSR features we have upstream already (perhaps finding a way for not
> duplicating all the common information).
>
Besides L2 CAT, there are L3 CAT and CDP features for allocation tech.
Also, there are CMT/MBM features for monitoring tech. We will discuss
these to check if we can add these feature documents step by step.
> Thanks and Regards,
> Dario
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-11-29 5:20 ` Yi Sun
@ 2016-11-29 12:25 ` Dario Faggioli
0 siblings, 0 replies; 42+ messages in thread
From: Dario Faggioli @ 2016-11-29 12:25 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
chao.p.peng, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 1260 bytes --]
On Tue, 2016-11-29 at 13:20 +0800, Yi Sun wrote:
> On 16-11-25 18:19:17, Dario Faggioli wrote:
> > On Fri, 2016-11-11 at 16:33 -0500, Konrad Rzeszutek Wilk wrote:
> > > Intel Problem Solver Resolver (PSR)
> > >
> > Wasn't it the 'Intel Probabilistic Silicon Reorganizer' ? :-D :-D
> >
> It is 'Intel Platform Shared Resource' indeed. :-)
>
Well, that's sad to hear. I'd love Inter to introduce something like a
Probabilistic Silicon Reorganizer... That would be a lot of fun! :-)
> > Also, it would be really good to have similar documents for the
> > other
> > PSR features we have upstream already (perhaps finding a way for
> > not
> > duplicating all the common information).
> >
> Besides L2 CAT, there are L3 CAT and CDP features for allocation
> tech.
> Also, there are CMT/MBM features for monitoring tech.
>
Yep, I know them.
> We will discuss
> these to check if we can add these feature documents step by step.
>
That would be awesome!
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-10-25 3:40 ` [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document Yi Sun
` (2 preceding siblings ...)
2016-11-11 21:33 ` Konrad Rzeszutek Wilk
@ 2016-11-25 17:39 ` Dario Faggioli
2016-11-29 4:52 ` Yi Sun
3 siblings, 1 reply; 42+ messages in thread
From: Dario Faggioli @ 2016-11-25 17:39 UTC (permalink / raw)
To: Yi Sun, xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich, chao.p.peng
[-- Attachment #1.1: Type: text/plain, Size: 4760 bytes --]
A couple of more things about this document...
On Tue, 2016-10-25 at 11:40 +0800, Yi Sun wrote:
> diff --git a/docs/features/l2_cat.pandoc
>
The name of the file could be a bit more descriptive. What about
something like:
intel_psr_l2_cat.pandoc
> +## Implementation Description
> +
> +* Hypervisor interfaces:
> +
> + 1. Ext: Boot line parameter "psr=cat" now will enable L2 CAT and
> L3
> + CAT if hardware supported.
> +
> + 2. New: SYSCTL:
> + - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
> +
> + 3. New: DOMCTL:
> + - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a
> domain.
> + - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a
> domain.
> +
What do these 'Ext' and 'New' prefixes mean? You perhaps what to
communicate whether you are extending something that was existing
already, or introducing something new... Well, I don't think that is a
very valuable piece of information.
After all, we want to know that there is a relevant boot parameter, a
sysctl and a domctl interface, no matter whether they're new or not
(especially considering that this is the first and for now only feature
document for PSR).
I'd just kill them.
> +* xl interfaces:
> +
> + 1. Ext: psr-cat-show -l2 domain-id
> + Show L2 cbm for a domain.
> + => XEN_SYSCTL_PSR_CAT_get_l2_info /
> + XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
> +
> + 2. Ext: psr-mba-set -l2 domain-id cbm
> + Set L2 cbm for a domain.
> + => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
> +
> + 3. Ext: psr-hwinfo
> + Show PSR HW information, including L2 CAT
> + => XEN_SYSCTL_PSR_CAT_get_l2_info
> +
Same here.
> +# User information
> +
> +* Feature Enabling:
> +
> + Add "psr=cat" to boot line parameter to enable all supported level
> CAT
> + features.
> +
> +* xl interfaces:
> +
> + 1. `psr-cat-show [OPTIONS] domain-id`:
> +
> + Show domain L2 or L3 CAT CBM.
> +
> + New option `-l` is added.
> + `-l2`: Specify cbm for L2 cache.
> + `-l3`: Specify cbm for L3 cache.
> +
> + If neither `-l2` nor `-l3` is given, level 3 is the default
> option.
> +
Sorry for saying this only now, but wouldn't it be more natural, if
neither -l2 not -l3 is specified, to show both (or, in general, all
that is supported)?
> + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
> +
> + Set domain L2 or L3 CBM.
> +
> + New option `-l` is added.
> + `-l2`: Specify cbm for L2 cache.
> + `-l3`: Specify cbm for L3 cache.
> +
> + If neither `-l2` nor `-l3` is given, level 3 is the default
> option.
> +
> + 3. `psr-hwinfo [OPTIONS]`:
> +
> + Show L2 & L3 CAT HW informations on every socket.
> +------------------------------------------------------------------
> ------
> +Date Revision Version Notes
> +---------- -------- -------- -------------------------------------
> ------
> +2016-08-12 1.0 Xen 4.7 Initial design
> +2016-09-21 2.0 Xen 4.7 Changes according to review comments.
> + 1. L2 CBM bits should be continuous
> too.
> + 2. Describe 'struct feat_info'
> + 3. Update 'struct feat_list"
> description.
> + 4. Update 'struct feat_ops"
> description.
> + 5. Update `psr-cat-show` description.
> + 6. Update `psr-cat-cbm-set`
> description.
> + 7. Add volume and chapter info in
> References.
>
If you want to keep track of this uncommited version here, I think it's
ok. But I'd just cur it short to a quick 1-liner summary and kill the
detailed list of changes (which could then be moved in the patch
changelog, but below the usual '---' so it won't get commited).
Personally, I'd also, avoid specifying any Xen version in for these,
and do that starting from the line corresponding to the first version
of the document that hits the tree. Like this, it somehow gives the
impression that the feature is present in Xen 4.7, which is not true.
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-11-25 17:39 ` Dario Faggioli
@ 2016-11-29 4:52 ` Yi Sun
2016-11-29 12:22 ` Dario Faggioli
0 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2016-11-29 4:52 UTC (permalink / raw)
To: Dario Faggioli
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
chao.p.peng, xen-devel
On 16-11-25 18:39:41, Dario Faggioli wrote:
> A couple of more things about this document...
>
> On Tue, 2016-10-25 at 11:40 +0800, Yi Sun wrote:
> > diff --git a/docs/features/l2_cat.pandoc
> >
> The name of the file could be a bit more descriptive. What about
> something like:
>
> intel_psr_l2_cat.pandoc
>
Thanks for your suggestion!
> > +## Implementation Description
> > +
> > +* Hypervisor interfaces:
> > +
> > + 1. Ext: Boot line parameter "psr=cat" now will enable L2 CAT and
> > L3
> > + CAT if hardware supported.
> > +
> > + 2. New: SYSCTL:
> > + - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
> > +
> > + 3. New: DOMCTL:
> > + - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a
> > domain.
> > + - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a
> > domain.
> > +
> What do these 'Ext' and 'New' prefixes mean? You perhaps what to
> communicate whether you are extending something that was existing
> already, or introducing something new... Well, I don't think that is a
> very valuable piece of information.
>
> After all, we want to know that there is a relevant boot parameter, a
> sysctl and a domctl interface, no matter whether they're new or not
> (especially considering that this is the first and for now only feature
> document for PSR).
>
> I'd just kill them.
>
Yes, your understand is correct. Thanks for your comments! I will kill them.
> > +* xl interfaces:
> > +
> > + 1. Ext: psr-cat-show -l2 domain-id
> > + Show L2 cbm for a domain.
> > + => XEN_SYSCTL_PSR_CAT_get_l2_info /
> > + XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
> > +
> > + 2. Ext: psr-mba-set -l2 domain-id cbm
> > + Set L2 cbm for a domain.
> > + => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
> > +
> > + 3. Ext: psr-hwinfo
> > + Show PSR HW information, including L2 CAT
> > + => XEN_SYSCTL_PSR_CAT_get_l2_info
> > +
> Same here.
>
Thank you!
> > +# User information
> > +
> > +* Feature Enabling:
> > +
> > + Add "psr=cat" to boot line parameter to enable all supported level
> > CAT
> > + features.
> > +
> > +* xl interfaces:
> > +
> > + 1. `psr-cat-show [OPTIONS] domain-id`:
> > +
> > + Show domain L2 or L3 CAT CBM.
> > +
> > + New option `-l` is added.
> > + `-l2`: Specify cbm for L2 cache.
> > + `-l3`: Specify cbm for L3 cache.
> > +
> > + If neither `-l2` nor `-l3` is given, level 3 is the default
> > option.
> > +
> Sorry for saying this only now, but wouldn't it be more natural, if
> neither -l2 not -l3 is specified, to show both (or, in general, all
> that is supported)?
>
This is for backward compatibility. The original command only supports
L3 CAT and it does not have '-l' option.
But for show command, your suggestion is good. We can show both or
prompt user if any one is not supported.
> > + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
> > +
> > + Set domain L2 or L3 CBM.
> > +
> > + New option `-l` is added.
> > + `-l2`: Specify cbm for L2 cache.
> > + `-l3`: Specify cbm for L3 cache.
> > +
> > + If neither `-l2` nor `-l3` is given, level 3 is the default
> > option.
> > +
> > + 3. `psr-hwinfo [OPTIONS]`:
> > +
> > + Show L2 & L3 CAT HW informations on every socket.
>
> > +------------------------------------------------------------------
> > ------
> > +Date Revision Version Notes
> > +---------- -------- -------- -------------------------------------
> > ------
> > +2016-08-12 1.0 Xen 4.7 Initial design
> > +2016-09-21 2.0 Xen 4.7 Changes according to review comments.
> > + 1. L2 CBM bits should be continuous
> > too.
> > + 2. Describe 'struct feat_info'
> > + 3. Update 'struct feat_list"
> > description.
> > + 4. Update 'struct feat_ops"
> > description.
> > + 5. Update `psr-cat-show` description.
> > + 6. Update `psr-cat-cbm-set`
> > description.
> > + 7. Add volume and chapter info in
> > References.
> >
> If you want to keep track of this uncommited version here, I think it's
> ok. But I'd just cur it short to a quick 1-liner summary and kill the
> detailed list of changes (which could then be moved in the patch
> changelog, but below the usual '---' so it won't get commited).
>
> Personally, I'd also, avoid specifying any Xen version in for these,
> and do that starting from the line corresponding to the first version
> of the document that hits the tree. Like this, it somehow gives the
> impression that the feature is present in Xen 4.7, which is not true.
>
> Regards,
> Dario
Thanks a lot for the suggestion! I will remove Xen version and change
log here.
BRs,
Sun Yi
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-11-29 4:52 ` Yi Sun
@ 2016-11-29 12:22 ` Dario Faggioli
2016-11-30 1:42 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Dario Faggioli @ 2016-11-29 12:22 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
xen-devel, chao.p.peng
[-- Attachment #1.1: Type: text/plain, Size: 2120 bytes --]
On Tue, 2016-11-29 at 12:52 +0800, Yi Sun wrote:
> On 16-11-25 18:39:41, Dario Faggioli wrote:
> > On Tue, 2016-10-25 at 11:40 +0800, Yi Sun wrote:
> > > +* xl interfaces:
> > > +
> > > + 1. `psr-cat-show [OPTIONS] domain-id`:
> > > +
> > > + Show domain L2 or L3 CAT CBM.
> > > +
> > > + New option `-l` is added.
> > > + `-l2`: Specify cbm for L2 cache.
> > > + `-l3`: Specify cbm for L3 cache.
> > > +
> > > + If neither `-l2` nor `-l3` is given, level 3 is the default
> > > option.
> > > +
> > Sorry for saying this only now, but wouldn't it be more natural, if
> > neither -l2 not -l3 is specified, to show both (or, in general, all
> > that is supported)?
> >
> This is for backward compatibility. The original command only
> supports
> L3 CAT and it does not have '-l' option.
>
> But for show command, your suggestion is good. We can show both or
> prompt user if any one is not supported.
>
I don't understand. I was actually talking about 'show' already, basing
on this that I see in the patch:
+ 1. `psr-cat-show [OPTIONS] domain-id`:> > >
+
+ Show domain L2 or L3 CAT CBM.
However, now that I look better, I notice that, when explaining the
options, you write "Specify cbm for L2 cache.", which is not what I
expect show to do. Shouldn't that say "Show" or "Display"?
Cut-&-paste error (from below), or there is really something I'm
missing?
> > > + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
> > > +
> > > + Set domain L2 or L3 CBM.
> > > +
> > > + New option `-l` is added.
> > > + `-l2`: Specify cbm for L2 cache.
> > > + `-l3`: Specify cbm for L3 cache.
> > > +
> > > + If neither `-l2` nor `-l3` is given, level 3 is the default
> > > option.
> > > +
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document.
2016-11-29 12:22 ` Dario Faggioli
@ 2016-11-30 1:42 ` Yi Sun
0 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-11-30 1:42 UTC (permalink / raw)
To: Dario Faggioli
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
xen-devel, chao.p.peng
On 16-11-29 13:22:41, Dario Faggioli wrote:
> On Tue, 2016-11-29 at 12:52 +0800, Yi Sun wrote:
> > On 16-11-25 18:39:41, Dario Faggioli wrote:
> > > On Tue, 2016-10-25 at 11:40 +0800, Yi Sun wrote:
> > > > +* xl interfaces:
> > > > +
> > > > + 1. `psr-cat-show [OPTIONS] domain-id`:
> > > > +
> > > > + Show domain L2 or L3 CAT CBM.
> > > > +
> > > > + New option `-l` is added.
> > > > + `-l2`: Specify cbm for L2 cache.
> > > > + `-l3`: Specify cbm for L3 cache.
> > > > +
> > > > + If neither `-l2` nor `-l3` is given, level 3 is the default
> > > > option.
> > > > +
> > > Sorry for saying this only now, but wouldn't it be more natural, if
> > > neither -l2 not -l3 is specified, to show both (or, in general, all
> > > that is supported)?
> > >
> > This is for backward compatibility. The original command only
> > supports
> > L3 CAT and it does not have '-l' option.
> >
> > But for show command, your suggestion is good. We can show both or
> > prompt user if any one is not supported.
> >
> I don't understand. I was actually talking about 'show' already, basing
> on this that I see in the patch:
>
> + 1. `psr-cat-show [OPTIONS] domain-id`:> > >
> +
> + Show domain L2 or L3 CAT CBM.
>
Sorry for confusion. I agree with your suggestion and will try to change
codes to show both L2 and L3.
> However, now that I look better, I notice that, when explaining the
> options, you write "Specify cbm for L2 cache.", which is not what I
> expect show to do. Shouldn't that say "Show" or "Display"?
>
> Cut-&-paste error (from below), or there is really something I'm
> missing?
>
Oh, sorry. It is a Cut-Paste error. Should be 'Show cbm of L2 cache'.
Thanks for finding out this error!
> > > > + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
> > > > +
> > > > + Set domain L2 or L3 CBM.
> > > > +
> > > > + New option `-l` is added.
> > > > + `-l2`: Specify cbm for L2 cache.
> > > > + `-l3`: Specify cbm for L3 cache.
> > > > +
> > > > + If neither `-l2` nor `-l3` is given, level 3 is the default
> > > > option.
> > > > +
>
> Regards,
> Dario
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v3 02/15] x86: refactor psr: Split 'ref' out.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
2016-10-25 3:40 ` [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-11-25 15:19 ` Jan Beulich
2016-10-25 3:40 ` [PATCH v3 03/15] x86: refactor psr: Remove 'struct psr_cat_cbm' Yi Sun
` (13 subsequent siblings)
15 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
'ref' in 'struct psr_cat_cbm' is used as a reference count for
one COS register. It doesn't relate to cbm. So, it can be splitted
out.
This patch removes 'ref' from 'struct psr_cat_cbm' and creates a
'cos_ref' array to manage reference counts of all COS registers.
It also renames the 'cbm_lock' to 'ref_lock' because this lock is
used to protect the reference count.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 55 ++++++++++++++++++++++++++++++++----------------------
1 file changed, 33 insertions(+), 22 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0b5073c..ee20389 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -23,6 +23,9 @@
#define PSR_CAT (1<<1)
#define PSR_CDP (1<<2)
+/* Per spec, the maximum COS register number is 128. */
+#define MAX_COS_REG_NUM 128
+
struct psr_cat_cbm {
union {
uint64_t cbm;
@@ -31,14 +34,19 @@ struct psr_cat_cbm {
uint64_t data;
};
};
- unsigned int ref;
};
struct psr_cat_socket_info {
unsigned int cbm_len;
unsigned int cos_max;
struct psr_cat_cbm *cos_to_cbm;
- spinlock_t cbm_lock;
+ /*
+ * Every entry of cos_ref is the reference count of a COS register.
+ * One entry of cos_ref corresponds to one COS ID.
+ */
+ unsigned int cos_ref[MAX_COS_REG_NUM];
+ /* Protect cos_ref */
+ spinlock_t ref_lock;
};
struct psr_assoc {
@@ -54,7 +62,7 @@ static unsigned long *__read_mostly cdp_socket_enable;
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
-static unsigned int __read_mostly opt_cos_max = 255;
+static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_NUM - 1;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
@@ -400,14 +408,15 @@ static int write_l3_cbm(unsigned int socket, unsigned int cos,
return 0;
}
-static int find_cos(struct psr_cat_cbm *map, unsigned int cos_max,
+static int find_cos(struct psr_cat_cbm *map, unsigned int *ref,
+ unsigned int cos_max,
uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
{
unsigned int cos;
for ( cos = 0; cos <= cos_max; cos++ )
{
- if ( (map[cos].ref || cos == 0) &&
+ if ( (ref[cos] || cos == 0) &&
((!cdp_enabled && map[cos].cbm == cbm_code) ||
(cdp_enabled && map[cos].code == cbm_code &&
map[cos].data == cbm_data)) )
@@ -417,18 +426,18 @@ static int find_cos(struct psr_cat_cbm *map, unsigned int cos_max,
return -ENOENT;
}
-static int pick_avail_cos(struct psr_cat_cbm *map, unsigned int cos_max,
+static int pick_avail_cos(unsigned int *ref, unsigned int cos_max,
unsigned int old_cos)
{
unsigned int cos;
/* If old cos is referred only by the domain, then use it. */
- if ( map[old_cos].ref == 1 && old_cos != 0 )
+ if ( ref[old_cos] == 1 && old_cos != 0 )
return old_cos;
/* Find an unused one other than cos0. */
for ( cos = 1; cos <= cos_max; cos++ )
- if ( map[cos].ref == 0 )
+ if ( ref[cos] == 0 )
return cos;
return -ENOENT;
@@ -443,6 +452,7 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
bool_t cdp_enabled = cdp_is_enabled(socket);
struct psr_cat_cbm *map;
struct psr_cat_socket_info *info = get_cat_socket_info(socket);
+ unsigned int *ref;
if ( IS_ERR(info) )
return PTR_ERR(info);
@@ -457,6 +467,7 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
cos_max = info->cos_max;
old_cos = d->arch.psr_cos_ids[socket];
map = info->cos_to_cbm;
+ ref = info->cos_ref;
switch ( type )
{
@@ -480,22 +491,22 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
return -EINVAL;
}
- spin_lock(&info->cbm_lock);
- cos = find_cos(map, cos_max, cbm_code, cbm_data, cdp_enabled);
+ spin_lock(&info->ref_lock);
+ cos = find_cos(map, ref, cos_max, cbm_code, cbm_data, cdp_enabled);
if ( cos >= 0 )
{
if ( cos == old_cos )
{
- spin_unlock(&info->cbm_lock);
+ spin_unlock(&info->ref_lock);
return 0;
}
}
else
{
- cos = pick_avail_cos(map, cos_max, old_cos);
+ cos = pick_avail_cos(ref, cos_max, old_cos);
if ( cos < 0 )
{
- spin_unlock(&info->cbm_lock);
+ spin_unlock(&info->ref_lock);
return cos;
}
@@ -507,7 +518,7 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
if ( ret )
{
- spin_unlock(&info->cbm_lock);
+ spin_unlock(&info->ref_lock);
return ret;
}
map[cos].code = cbm_code;
@@ -515,9 +526,9 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
}
}
- map[cos].ref++;
- map[old_cos].ref--;
- spin_unlock(&info->cbm_lock);
+ ref[cos]++;
+ ref[old_cos]--;
+ spin_unlock(&info->ref_lock);
d->arch.psr_cos_ids[socket] = cos;
@@ -540,9 +551,9 @@ static void psr_free_cos(struct domain *d)
continue;
info = cat_socket_info + socket;
- spin_lock(&info->cbm_lock);
- info->cos_to_cbm[cos].ref--;
- spin_unlock(&info->cbm_lock);
+ spin_lock(&info->ref_lock);
+ info->cos_ref[cos]--;
+ spin_unlock(&info->ref_lock);
}
xfree(d->arch.psr_cos_ids);
@@ -574,7 +585,7 @@ static int cat_cpu_prepare(unsigned int cpu)
if ( temp_cos_to_cbm == NULL &&
(temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
- opt_cos_max + 1UL)) == NULL )
+ MAX_COS_REG_NUM)) == NULL )
return -ENOMEM;
return 0;
@@ -609,7 +620,7 @@ static void cat_cpu_init(void)
/* cos=0 is reserved as default cbm(all ones). */
info->cos_to_cbm[0].cbm = (1ull << info->cbm_len) - 1;
- spin_lock_init(&info->cbm_lock);
+ spin_lock_init(&info->ref_lock);
set_bit(socket, cat_socket_enable);
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v3 02/15] x86: refactor psr: Split 'ref' out.
2016-10-25 3:40 ` [PATCH v3 02/15] x86: refactor psr: Split 'ref' out Yi Sun
@ 2016-11-25 15:19 ` Jan Beulich
0 siblings, 0 replies; 42+ messages in thread
From: Jan Beulich @ 2016-11-25 15:19 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
>>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote:
> 'ref' in 'struct psr_cat_cbm' is used as a reference count for
> one COS register. It doesn't relate to cbm. So, it can be splitted
> out.
split
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -23,6 +23,9 @@
> #define PSR_CAT (1<<1)
> #define PSR_CDP (1<<2)
>
> +/* Per spec, the maximum COS register number is 128. */
> +#define MAX_COS_REG_NUM 128
Is this the maximum register number or the maximum possible
count of registers? If the latter (which I assume base on how it
gets used below), please prefer COUNT or CNT over NUM.
> @@ -31,14 +34,19 @@ struct psr_cat_cbm {
> uint64_t data;
> };
> };
> - unsigned int ref;
> };
>
> struct psr_cat_socket_info {
> unsigned int cbm_len;
> unsigned int cos_max;
> struct psr_cat_cbm *cos_to_cbm;
> - spinlock_t cbm_lock;
> + /*
> + * Every entry of cos_ref is the reference count of a COS register.
> + * One entry of cos_ref corresponds to one COS ID.
> + */
> + unsigned int cos_ref[MAX_COS_REG_NUM];
> + /* Protect cos_ref */
> + spinlock_t ref_lock;
So till now we had opt_cos_max + 1 refs per socket. Now all of the
sudden we get 128 of them. Is that really needed? Shouldn't this be
a pointer, and space allocated just like for info->cos_to_cbm in
cat_cpu_prepare()?
> @@ -54,7 +62,7 @@ static unsigned long *__read_mostly cdp_socket_enable;
>
> static unsigned int opt_psr;
> static unsigned int __initdata opt_rmid_max = 255;
> -static unsigned int __read_mostly opt_cos_max = 255;
> +static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_NUM - 1;
This change tells me that th previous author thought the specified
maximum is higher. Would you please point me to where this new
lower limit is written down?
> @@ -574,7 +585,7 @@ static int cat_cpu_prepare(unsigned int cpu)
>
> if ( temp_cos_to_cbm == NULL &&
> (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
> - opt_cos_max + 1UL)) == NULL )
> + MAX_COS_REG_NUM)) == NULL )
Why?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v3 03/15] x86: refactor psr: Remove 'struct psr_cat_cbm'.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
2016-10-25 3:40 ` [PATCH v3 01/15] docs: L2 Cache Allocation Technology (CAT) feature document Yi Sun
2016-10-25 3:40 ` [PATCH v3 02/15] x86: refactor psr: Split 'ref' out Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-11-25 15:45 ` Jan Beulich
2016-10-25 3:40 ` [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max' Yi Sun
` (12 subsequent siblings)
15 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
'struct psr_cat_cbm' is defined for L3 CAT only. It is not
appropriate for other features. This patch replaces it with
a generic array, 'uint64_t cos_reg_val[MAX_COS_REG_NUM]',
to save values of COS registers. So 'temp_cos_to_cbm' is not
useful anymore, remove it.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 93 ++++++++++++++++++++++++++----------------------------
1 file changed, 45 insertions(+), 48 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index ee20389..a0342ce 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -26,20 +26,15 @@
/* Per spec, the maximum COS register number is 128. */
#define MAX_COS_REG_NUM 128
-struct psr_cat_cbm {
- union {
- uint64_t cbm;
- struct {
- uint64_t code;
- uint64_t data;
- };
- };
-};
-
struct psr_cat_socket_info {
unsigned int cbm_len;
unsigned int cos_max;
- struct psr_cat_cbm *cos_to_cbm;
+ /*
+ * Store the values of COS registers:
+ * CAT uses 1 entry for one COS ID;
+ * CDP uses 2 entries for one COS ID and DATA is the first one.
+ */
+ uint64_t cos_reg_val[MAX_COS_REG_NUM];
/*
* Every entry of cos_ref is the reference count of a COS register.
* One entry of cos_ref corresponds to one COS ID.
@@ -49,6 +44,21 @@ struct psr_cat_socket_info {
spinlock_t ref_lock;
};
+/*
+ * get_data - get DATA COS register value from input COS ID.
+ * @info: the struct psr_cat_socket_info pointer.
+ * @cos: the COS ID.
+ */
+#define get_cdp_data(info, cos) \
+ info->cos_reg_val[cos * 2]
+/*
+ * get_cdp_code - get CODE COS register value from input COS ID.
+ * @info: the struct psr_cat_socket_info pointer.
+ * @cos: the COS ID.
+ */
+#define get_cdp_code(info, cos) \
+ info->cos_reg_val[cos * 2 + 1]
+
struct psr_assoc {
uint64_t val;
uint64_t cos_mask;
@@ -66,8 +76,6 @@ static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_NUM - 1;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
-static struct psr_cat_cbm *temp_cos_to_cbm;
-
static unsigned int get_socket_cpu(unsigned int socket)
{
if ( likely(socket < nr_sockets) )
@@ -306,6 +314,7 @@ int psr_get_l3_cbm(struct domain *d, unsigned int socket,
{
struct psr_cat_socket_info *info = get_cat_socket_info(socket);
bool_t cdp_enabled = cdp_is_enabled(socket);
+ unsigned int cos = d->arch.psr_cos_ids[socket];
if ( IS_ERR(info) )
return PTR_ERR(info);
@@ -315,21 +324,21 @@ int psr_get_l3_cbm(struct domain *d, unsigned int socket,
case PSR_CBM_TYPE_L3:
if ( cdp_enabled )
return -EXDEV;
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
+ *cbm = info->cos_reg_val[cos];
break;
case PSR_CBM_TYPE_L3_CODE:
if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
+ *cbm = info->cos_reg_val[cos];
else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].code;
+ *cbm = get_cdp_code(info, cos);
break;
case PSR_CBM_TYPE_L3_DATA:
if ( !cdp_enabled )
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
+ *cbm = info->cos_reg_val[cos];
else
- *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].data;
+ *cbm = get_cdp_data(info, cos);
break;
default:
@@ -408,7 +417,7 @@ static int write_l3_cbm(unsigned int socket, unsigned int cos,
return 0;
}
-static int find_cos(struct psr_cat_cbm *map, unsigned int *ref,
+static int find_cos(struct psr_cat_socket_info *info, unsigned int *ref,
unsigned int cos_max,
uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
{
@@ -417,9 +426,9 @@ static int find_cos(struct psr_cat_cbm *map, unsigned int *ref,
for ( cos = 0; cos <= cos_max; cos++ )
{
if ( (ref[cos] || cos == 0) &&
- ((!cdp_enabled && map[cos].cbm == cbm_code) ||
- (cdp_enabled && map[cos].code == cbm_code &&
- map[cos].data == cbm_data)) )
+ ((!cdp_enabled && info->cos_reg_val[cos] == cbm_code) ||
+ (cdp_enabled && get_cdp_code(info, cos) == cbm_code &&
+ get_cdp_data(info, cos) == cbm_data)) )
return cos;
}
@@ -450,7 +459,6 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
int cos, ret;
uint64_t cbm_data, cbm_code;
bool_t cdp_enabled = cdp_is_enabled(socket);
- struct psr_cat_cbm *map;
struct psr_cat_socket_info *info = get_cat_socket_info(socket);
unsigned int *ref;
@@ -466,7 +474,6 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
cos_max = info->cos_max;
old_cos = d->arch.psr_cos_ids[socket];
- map = info->cos_to_cbm;
ref = info->cos_ref;
switch ( type )
@@ -478,11 +485,11 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
case PSR_CBM_TYPE_L3_CODE:
cbm_code = cbm;
- cbm_data = map[old_cos].data;
+ cbm_data = get_cdp_data(info, old_cos);
break;
case PSR_CBM_TYPE_L3_DATA:
- cbm_code = map[old_cos].code;
+ cbm_code = get_cdp_code(info, old_cos);
cbm_data = cbm;
break;
@@ -492,7 +499,7 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
}
spin_lock(&info->ref_lock);
- cos = find_cos(map, ref, cos_max, cbm_code, cbm_data, cdp_enabled);
+ cos = find_cos(info, ref, cos_max, cbm_code, cbm_data, cdp_enabled);
if ( cos >= 0 )
{
if ( cos == old_cos )
@@ -512,8 +519,9 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
/* We try to avoid writing MSR. */
if ( (cdp_enabled &&
- (map[cos].code != cbm_code || map[cos].data != cbm_data)) ||
- (!cdp_enabled && map[cos].cbm != cbm_code) )
+ (get_cdp_code(info, cos) != cbm_code ||
+ get_cdp_data(info, cos) != cbm_data)) ||
+ (!cdp_enabled && info->cos_reg_val[cos] != cbm_code) )
{
ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
if ( ret )
@@ -521,8 +529,8 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
spin_unlock(&info->ref_lock);
return ret;
}
- map[cos].code = cbm_code;
- map[cos].data = cbm_data;
+ get_cdp_code(info, cos) = cbm_code;
+ get_cdp_data(info, cos) = cbm_data;
}
}
@@ -583,10 +591,7 @@ static int cat_cpu_prepare(unsigned int cpu)
if ( !cat_socket_info )
return 0;
- if ( temp_cos_to_cbm == NULL &&
- (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
- MAX_COS_REG_NUM)) == NULL )
- return -ENOMEM;
+ /* Keep this function for future usage. */
return 0;
}
@@ -615,10 +620,8 @@ static void cat_cpu_init(void)
info->cbm_len = (eax & 0x1f) + 1;
info->cos_max = min(opt_cos_max, edx & 0xffff);
- info->cos_to_cbm = temp_cos_to_cbm;
- temp_cos_to_cbm = NULL;
/* cos=0 is reserved as default cbm(all ones). */
- info->cos_to_cbm[0].cbm = (1ull << info->cbm_len) - 1;
+ info->cos_reg_val[0] = (1ull << info->cbm_len) - 1;
spin_lock_init(&info->ref_lock);
@@ -627,8 +630,10 @@ static void cat_cpu_init(void)
if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
{
- info->cos_to_cbm[0].code = (1ull << info->cbm_len) - 1;
- info->cos_to_cbm[0].data = (1ull << info->cbm_len) - 1;
+ /* CODE */
+ get_cdp_code(info, 0) = (1ull << info->cbm_len) - 1;
+ /* DATA */
+ get_cdp_data(info, 0) = (1ull << info->cbm_len) - 1;
/* We only write mask1 since mask0 is always all ones by default. */
wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->cbm_len) - 1);
@@ -653,14 +658,6 @@ static void cat_cpu_fini(unsigned int cpu)
if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
{
- struct psr_cat_socket_info *info = cat_socket_info + socket;
-
- if ( info->cos_to_cbm )
- {
- xfree(info->cos_to_cbm);
- info->cos_to_cbm = NULL;
- }
-
if ( cdp_is_enabled(socket) )
clear_bit(socket, cdp_socket_enable);
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v3 03/15] x86: refactor psr: Remove 'struct psr_cat_cbm'.
2016-10-25 3:40 ` [PATCH v3 03/15] x86: refactor psr: Remove 'struct psr_cat_cbm' Yi Sun
@ 2016-11-25 15:45 ` Jan Beulich
0 siblings, 0 replies; 42+ messages in thread
From: Jan Beulich @ 2016-11-25 15:45 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
>>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote:
> struct psr_cat_socket_info {
> unsigned int cbm_len;
> unsigned int cos_max;
> - struct psr_cat_cbm *cos_to_cbm;
> + /*
> + * Store the values of COS registers:
> + * CAT uses 1 entry for one COS ID;
> + * CDP uses 2 entries for one COS ID and DATA is the first one.
> + */
> + uint64_t cos_reg_val[MAX_COS_REG_NUM];
Same comment as on the previous patch regarding this fixed size.
> @@ -49,6 +44,21 @@ struct psr_cat_socket_info {
> spinlock_t ref_lock;
> };
>
> +/*
> + * get_data - get DATA COS register value from input COS ID.
> + * @info: the struct psr_cat_socket_info pointer.
> + * @cos: the COS ID.
> + */
> +#define get_cdp_data(info, cos) \
> + info->cos_reg_val[cos * 2]
> +/*
> + * get_cdp_code - get CODE COS register value from input COS ID.
> + * @info: the struct psr_cat_socket_info pointer.
> + * @cos: the COS ID.
> + */
> +#define get_cdp_code(info, cos) \
> + info->cos_reg_val[cos * 2 + 1]
Both macros need to be properly parenthesized.
> @@ -306,6 +314,7 @@ int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> {
> struct psr_cat_socket_info *info = get_cat_socket_info(socket);
> bool_t cdp_enabled = cdp_is_enabled(socket);
> + unsigned int cos = d->arch.psr_cos_ids[socket];
>
> if ( IS_ERR(info) )
> return PTR_ERR(info);
You must not use socket as an array index before having passed
this check (or else you render the check pointless).
> @@ -417,9 +426,9 @@ static int find_cos(struct psr_cat_cbm *map, unsigned int *ref,
> for ( cos = 0; cos <= cos_max; cos++ )
> {
> if ( (ref[cos] || cos == 0) &&
> - ((!cdp_enabled && map[cos].cbm == cbm_code) ||
> - (cdp_enabled && map[cos].code == cbm_code &&
> - map[cos].data == cbm_data)) )
> + ((!cdp_enabled && info->cos_reg_val[cos] == cbm_code) ||
> + (cdp_enabled && get_cdp_code(info, cos) == cbm_code &&
> + get_cdp_data(info, cos) == cbm_data)) )
ref[cos] and get_cdp_code(info, cos) reference different array
indexes, so one could at least suspect ref counting and values
have gone out of sync here (the halving of ->cos_max during
initialization means it is still correct, but that's non-obvious I would
say). If you want per-register ref counting, and you need ref
counts for pairs (or more generally groups) of registers, then I
think you need a better abstraction. For example, what about
adding a referral array, which - when not matching a certain
"unused" indicator - tells which ref[] index holds the ref count.
E.g. the code register would be the one with the ref count, and
the data register would refer to the code ones' array index.
Depending on how much of the above construct is really a
hardware induced requirement (instead of just how software
chooses to handle things), there may of course also a backref
array be necessary then. In which case there may be an
easier to handle alternative model ...
> @@ -583,10 +591,7 @@ static int cat_cpu_prepare(unsigned int cpu)
> if ( !cat_socket_info )
> return 0;
>
> - if ( temp_cos_to_cbm == NULL &&
> - (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
> - MAX_COS_REG_NUM)) == NULL )
> - return -ENOMEM;
> + /* Keep this function for future usage. */
Unless you need it later in this series, please don't.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max'
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (2 preceding siblings ...)
2016-10-25 3:40 ` [PATCH v3 03/15] x86: refactor psr: Remove 'struct psr_cat_cbm' Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-11-25 16:27 ` Jan Beulich
2016-10-25 3:40 ` [PATCH v3 05/15] x86: refactor psr: Use 'feat_mask' to record featues enabled Yi Sun
` (11 subsequent siblings)
15 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
'cbm_len' and 'cbm_max' are CAT/CDP specific feature HW info.
So encapsulate them into 'struct psr_cat_hw_info'. If new
feature is supported, we can define other structure to save
its HW info.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 33 +++++++++++++++++++--------------
1 file changed, 19 insertions(+), 14 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index a0342ce..97f1c33 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -26,9 +26,14 @@
/* Per spec, the maximum COS register number is 128. */
#define MAX_COS_REG_NUM 128
-struct psr_cat_socket_info {
+/* CAT/CDP HW info data structure. */
+struct psr_cat_hw_info {
unsigned int cbm_len;
unsigned int cos_max;
+};
+
+struct psr_cat_socket_info {
+ struct psr_cat_hw_info l3_info;
/*
* Store the values of COS registers:
* CAT uses 1 entry for one COS ID;
@@ -235,7 +240,7 @@ static inline void psr_assoc_init(void)
if ( test_bit(socket, cat_socket_enable) )
psra->cos_mask = ((1ull << get_count_order(
- cat_socket_info[socket].cos_max)) - 1) << 32;
+ cat_socket_info[socket].l3_info.cos_max)) - 1) << 32;
}
if ( psr_cmt_enabled() || psra->cos_mask )
@@ -299,8 +304,8 @@ int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
if ( IS_ERR(info) )
return PTR_ERR(info);
- *cbm_len = info->cbm_len;
- *cos_max = info->cos_max;
+ *cbm_len = info->l3_info.cbm_len;
+ *cos_max = info->l3_info.cos_max;
*flags = 0;
if ( cdp_is_enabled(socket) )
@@ -465,14 +470,14 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
if ( IS_ERR(info) )
return PTR_ERR(info);
- if ( !psr_check_cbm(info->cbm_len, cbm) )
+ if ( !psr_check_cbm(info->l3_info.cbm_len, cbm) )
return -EINVAL;
if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
type == PSR_CBM_TYPE_L3_DATA) )
return -ENXIO;
- cos_max = info->cos_max;
+ cos_max = info->l3_info.cos_max;
old_cos = d->arch.psr_cos_ids[socket];
ref = info->cos_ref;
@@ -617,11 +622,11 @@ static void cat_cpu_init(void)
{
cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
info = cat_socket_info + socket;
- info->cbm_len = (eax & 0x1f) + 1;
- info->cos_max = min(opt_cos_max, edx & 0xffff);
+ info->l3_info.cbm_len = (eax & 0x1f) + 1;
+ info->l3_info.cos_max = min(opt_cos_max, edx & 0xffff);
/* cos=0 is reserved as default cbm(all ones). */
- info->cos_reg_val[0] = (1ull << info->cbm_len) - 1;
+ info->cos_reg_val[0] = (1ull << info->l3_info.cbm_len) - 1;
spin_lock_init(&info->ref_lock);
@@ -631,23 +636,23 @@ static void cat_cpu_init(void)
cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
{
/* CODE */
- get_cdp_code(info, 0) = (1ull << info->cbm_len) - 1;
+ get_cdp_code(info, 0) = (1ull << info->l3_info.cbm_len) - 1;
/* DATA */
- get_cdp_data(info, 0) = (1ull << info->cbm_len) - 1;
+ get_cdp_data(info, 0) = (1ull << info->l3_info.cbm_len) - 1;
/* We only write mask1 since mask0 is always all ones by default. */
- wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->cbm_len) - 1);
+ wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->l3_info.cbm_len) - 1);
rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
/* Cut half of cos_max when CDP is enabled. */
- info->cos_max >>= 1;
+ info->l3_info.cos_max >>= 1;
set_bit(socket, cdp_socket_enable);
}
printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
- socket, info->cos_max, info->cbm_len,
+ socket, info->l3_info.cos_max, info->l3_info.cbm_len,
cdp_is_enabled(socket) ? "on" : "off");
}
}
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max'
2016-10-25 3:40 ` [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max' Yi Sun
@ 2016-11-25 16:27 ` Jan Beulich
2016-11-25 16:57 ` Jan Beulich
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2016-11-25 16:27 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
>>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote:
> 'cbm_len' and 'cbm_max' are CAT/CDP specific feature HW info.
> So encapsulate them into 'struct psr_cat_hw_info'. If new
> feature is supported, we can define other structure to save
> its HW info.
Part of my problem following you here is that you talk about
cbm_max, but the code changes cos_max, which so far I had
understood would be a generic limit, which appears to be
supported by ...
> @@ -26,9 +26,14 @@
> /* Per spec, the maximum COS register number is 128. */
> #define MAX_COS_REG_NUM 128
>
> -struct psr_cat_socket_info {
> +/* CAT/CDP HW info data structure. */
> +struct psr_cat_hw_info {
> unsigned int cbm_len;
> unsigned int cos_max;
> +};
> +
> +struct psr_cat_socket_info {
> + struct psr_cat_hw_info l3_info;
> /*
> * Store the values of COS registers:
> * CAT uses 1 entry for one COS ID;
... you leaving this comment in place.
Btw., perhaps it would also help if psr.c had (near its beginning) a
glossary of the various acronyms. By having a way to quickly refresh
one's memory of what COS, CAT, CBM, and who know what else
stand for, it might be a little easier to reason about changes like this
one.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max'
2016-11-25 16:27 ` Jan Beulich
@ 2016-11-25 16:57 ` Jan Beulich
2016-11-29 4:38 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2016-11-25 16:57 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
>>> On 25.11.16 at 17:27, <JBeulich@suse.com> wrote:
>>>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote:
>> 'cbm_len' and 'cbm_max' are CAT/CDP specific feature HW info.
>> So encapsulate them into 'struct psr_cat_hw_info'. If new
>> feature is supported, we can define other structure to save
>> its HW info.
>
> Part of my problem following you here is that you talk about
> cbm_max, but the code changes cos_max, which so far I had
> understood would be a generic limit,
So I've gone and looked back at patch 1, where indeed you say
the limits might differ. Which raises the question then what
opt_cos_max is representing.
Having seen v1, v2, and v3 up to patch 5 I start wondering
whether the whole current code wouldn't better be ripped out
and then be replaced by something written from scratch. That's
because the split, while having reduced individual patch size,
doesn't really make the whole thing much better reviewable.
And the original implementation apparently simply didn't have
in mind how future additions on the hardware side could look
like.
Thoughts?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max'
2016-11-25 16:57 ` Jan Beulich
@ 2016-11-29 4:38 ` Yi Sun
2016-11-29 9:43 ` Jan Beulich
0 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2016-11-29 4:38 UTC (permalink / raw)
To: Jan Beulich
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
On 16-11-25 09:57:31, Jan Beulich wrote:
> >>> On 25.11.16 at 17:27, <JBeulich@suse.com> wrote:
> >>>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote:
> >> 'cbm_len' and 'cbm_max' are CAT/CDP specific feature HW info.
> >> So encapsulate them into 'struct psr_cat_hw_info'. If new
> >> feature is supported, we can define other structure to save
> >> its HW info.
> >
> > Part of my problem following you here is that you talk about
> > cbm_max, but the code changes cos_max, which so far I had
> > understood would be a generic limit,
>
> So I've gone and looked back at patch 1, where indeed you say
> the limits might differ. Which raises the question then what
> opt_cos_max is representing.
>
> Having seen v1, v2, and v3 up to patch 5 I start wondering
> whether the whole current code wouldn't better be ripped out
> and then be replaced by something written from scratch. That's
> because the split, while having reduced individual patch size,
> doesn't really make the whole thing much better reviewable.
> And the original implementation apparently simply didn't have
> in mind how future additions on the hardware side could look
> like.
>
> Thoughts?
>
> Jan
Firstly, I want to say sorry that the V3 still does not make you
feel good. I planned to split the big patch based on data structure
changes to make you understand more easily.
The original implementation of psr.c does not consider much about
future features because there was only L3 CAT introduced and we do
not know there will be new features and how they work. That is the
reason I refactor the psr.c to make it be easy to add new features
by abstracting the common things.
To make codes be better reviewable, I propose below three suggestions:
1) I write a design document about refactor to make you more easily
understand the ideas.
2) Replace the psr.c with a new file which discards all old codes so
that you do not need to consider old implementations which may cause
confusion.
3) Discard the refactor codes. Just implement L2 CAT based on current
codes. Because L2 CAT does not have much difference with L3.
How do you think? If you can offer your ideas to design and implement
the codes, that would be a great opportunity for me to learn! Thank you!
BRs,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max'
2016-11-29 4:38 ` Yi Sun
@ 2016-11-29 9:43 ` Jan Beulich
2016-11-30 9:08 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2016-11-29 9:43 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
>>> On 29.11.16 at 05:38, <yi.y.sun@linux.intel.com> wrote:
> To make codes be better reviewable, I propose below three suggestions:
> 1) I write a design document about refactor to make you more easily
> understand the ideas.
This is appreciated, but I don't think it's strictly necessary. Describing
the new design (rather than the changes to the existing one) is what
likely would be more useful (I'm sorry if I've misunderstood what you
said, and you in fact had meant just this), which iirc you already have
in patch 1.
> 2) Replace the psr.c with a new file which discards all old codes so
> that you do not need to consider old implementations which may cause
> confusion.
> 3) Discard the refactor codes. Just implement L2 CAT based on current
> codes. Because L2 CAT does not have much difference with L3.
I don't think introducing a new file is the ideal approach. I'd suggest
to rip out the entire implementation in a first patch, leaving just
empty functions to avoid breaking the build (i.e. perhaps mostly the
ones used by domctl/sysctl, and maybe some init one). Then
introduce new code, ideally of course not in one big patch, but
broken up into logical pieces where possible (one such split would be
that of course you don't need to re-implement domctl/sysctl handling
in the same patch as everything else).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max'
2016-11-29 9:43 ` Jan Beulich
@ 2016-11-30 9:08 ` Yi Sun
2016-11-30 9:42 ` Jan Beulich
0 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2016-11-30 9:08 UTC (permalink / raw)
To: Jan Beulich
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, xen-devel, chao.p.peng
On 16-11-29 02:43:24, Jan Beulich wrote:
> >>> On 29.11.16 at 05:38, <yi.y.sun@linux.intel.com> wrote:
> > To make codes be better reviewable, I propose below three suggestions:
> > 1) I write a design document about refactor to make you more easily
> > understand the ideas.
>
> This is appreciated, but I don't think it's strictly necessary. Describing
> the new design (rather than the changes to the existing one) is what
> likely would be more useful (I'm sorry if I've misunderstood what you
> said, and you in fact had meant just this), which iirc you already have
> in patch 1.
>
> > 2) Replace the psr.c with a new file which discards all old codes so
> > that you do not need to consider old implementations which may cause
> > confusion.
> > 3) Discard the refactor codes. Just implement L2 CAT based on current
> > codes. Because L2 CAT does not have much difference with L3.
>
> I don't think introducing a new file is the ideal approach. I'd suggest
> to rip out the entire implementation in a first patch, leaving just
> empty functions to avoid breaking the build (i.e. perhaps mostly the
> ones used by domctl/sysctl, and maybe some init one). Then
> introduce new code, ideally of course not in one big patch, but
> broken up into logical pieces where possible (one such split would be
> that of course you don't need to re-implement domctl/sysctl handling
> in the same patch as everything else).
>
> Jan
>
Thanks for your suggestion! Just want to confirm if my understanding is
right. So, you mean I can remove all old codes but keep the interfaces as
empty functions to make sure the build can pass. Then, implement whole
functionality step by step. Right?
Thanks,
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max'
2016-11-30 9:08 ` Yi Sun
@ 2016-11-30 9:42 ` Jan Beulich
2016-11-30 10:22 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2016-11-30 9:42 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
>>> On 30.11.16 at 10:08, <yi.y.sun@linux.intel.com> wrote:
> On 16-11-29 02:43:24, Jan Beulich wrote:
>> >>> On 29.11.16 at 05:38, <yi.y.sun@linux.intel.com> wrote:
>> > To make codes be better reviewable, I propose below three suggestions:
>> > 1) I write a design document about refactor to make you more easily
>> > understand the ideas.
>>
>> This is appreciated, but I don't think it's strictly necessary. Describing
>> the new design (rather than the changes to the existing one) is what
>> likely would be more useful (I'm sorry if I've misunderstood what you
>> said, and you in fact had meant just this), which iirc you already have
>> in patch 1.
>>
>> > 2) Replace the psr.c with a new file which discards all old codes so
>> > that you do not need to consider old implementations which may cause
>> > confusion.
>> > 3) Discard the refactor codes. Just implement L2 CAT based on current
>> > codes. Because L2 CAT does not have much difference with L3.
>>
>> I don't think introducing a new file is the ideal approach. I'd suggest
>> to rip out the entire implementation in a first patch, leaving just
>> empty functions to avoid breaking the build (i.e. perhaps mostly the
>> ones used by domctl/sysctl, and maybe some init one). Then
>> introduce new code, ideally of course not in one big patch, but
>> broken up into logical pieces where possible (one such split would be
>> that of course you don't need to re-implement domctl/sysctl handling
>> in the same patch as everything else).
>>
> Thanks for your suggestion! Just want to confirm if my understanding is
> right. So, you mean I can remove all old codes but keep the interfaces as
> empty functions to make sure the build can pass. Then, implement whole
> functionality step by step. Right?
Yes.
Thanks, Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max'
2016-11-30 9:42 ` Jan Beulich
@ 2016-11-30 10:22 ` Yi Sun
0 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-11-30 10:22 UTC (permalink / raw)
To: Jan Beulich
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
On 16-11-30 02:42:20, Jan Beulich wrote:
> >>> On 30.11.16 at 10:08, <yi.y.sun@linux.intel.com> wrote:
> > On 16-11-29 02:43:24, Jan Beulich wrote:
> >> >>> On 29.11.16 at 05:38, <yi.y.sun@linux.intel.com> wrote:
> >> > To make codes be better reviewable, I propose below three suggestions:
> >> > 1) I write a design document about refactor to make you more easily
> >> > understand the ideas.
> >>
> >> This is appreciated, but I don't think it's strictly necessary. Describing
> >> the new design (rather than the changes to the existing one) is what
> >> likely would be more useful (I'm sorry if I've misunderstood what you
> >> said, and you in fact had meant just this), which iirc you already have
> >> in patch 1.
> >>
> >> > 2) Replace the psr.c with a new file which discards all old codes so
> >> > that you do not need to consider old implementations which may cause
> >> > confusion.
> >> > 3) Discard the refactor codes. Just implement L2 CAT based on current
> >> > codes. Because L2 CAT does not have much difference with L3.
> >>
> >> I don't think introducing a new file is the ideal approach. I'd suggest
> >> to rip out the entire implementation in a first patch, leaving just
> >> empty functions to avoid breaking the build (i.e. perhaps mostly the
> >> ones used by domctl/sysctl, and maybe some init one). Then
> >> introduce new code, ideally of course not in one big patch, but
> >> broken up into logical pieces where possible (one such split would be
> >> that of course you don't need to re-implement domctl/sysctl handling
> >> in the same patch as everything else).
> >>
> > Thanks for your suggestion! Just want to confirm if my understanding is
> > right. So, you mean I can remove all old codes but keep the interfaces as
> > empty functions to make sure the build can pass. Then, implement whole
> > functionality step by step. Right?
>
> Yes.
>
> Thanks, Jan
Got it. Thanks!
Sun Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v3 05/15] x86: refactor psr: Use 'feat_mask' to record featues enabled.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (3 preceding siblings ...)
2016-10-25 3:40 ` [PATCH v3 04/15] x86: refactor psr: Encapsulate 'cbm_len' and 'cbm_max' Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-11-25 16:36 ` Jan Beulich
2016-10-25 3:40 ` [PATCH v3 06/15] x86: refactor psr: Create feature list Yi Sun
` (10 subsequent siblings)
15 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
'cdp_socket_enable' and 'cat_socket_enable' are used to mask if
CDP/CAT are enabled on socket. But they are specific for CAT and
CDP. So, replace them with 'feat_mask' which is a general mask.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 44 +++++++++++++++++++++++---------------------
1 file changed, 23 insertions(+), 21 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 97f1c33..aa902d2 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -26,6 +26,9 @@
/* Per spec, the maximum COS register number is 128. */
#define MAX_COS_REG_NUM 128
+#define PSR_SOCKET_L3_CAT 0
+#define PSR_SOCKET_L3_CDP 1
+
/* CAT/CDP HW info data structure. */
struct psr_cat_hw_info {
unsigned int cbm_len;
@@ -33,6 +36,8 @@ struct psr_cat_hw_info {
};
struct psr_cat_socket_info {
+ /* bit 1~0: [01]->L3 CAT-only, [10]->L3 CDP */
+ unsigned int feat_mask;
struct psr_cat_hw_info l3_info;
/*
* Store the values of COS registers:
@@ -71,9 +76,7 @@ struct psr_assoc {
struct psr_cmt *__read_mostly psr_cmt;
-static unsigned long *__read_mostly cat_socket_enable;
static struct psr_cat_socket_info *__read_mostly cat_socket_info;
-static unsigned long *__read_mostly cdp_socket_enable;
static unsigned int opt_psr;
static unsigned int __initdata opt_rmid_max = 255;
@@ -238,7 +241,7 @@ static inline void psr_assoc_init(void)
{
unsigned int socket = cpu_to_socket(smp_processor_id());
- if ( test_bit(socket, cat_socket_enable) )
+ if ( cat_socket_info[socket].feat_mask )
psra->cos_mask = ((1ull << get_count_order(
cat_socket_info[socket].l3_info.cos_max)) - 1) << 32;
}
@@ -285,7 +288,7 @@ static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
if ( socket >= nr_sockets )
return ERR_PTR(-ENOTSOCK);
- if ( !test_bit(socket, cat_socket_enable) )
+ if ( !cat_socket_info[socket].feat_mask )
return ERR_PTR(-ENOENT);
return cat_socket_info + socket;
@@ -293,7 +296,7 @@ static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
static inline bool_t cdp_is_enabled(unsigned int socket)
{
- return cdp_socket_enable && test_bit(socket, cdp_socket_enable);
+ return test_bit(PSR_SOCKET_L3_CDP, &cat_socket_info[socket].feat_mask);
}
int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
@@ -558,7 +561,7 @@ static void psr_free_cos(struct domain *d)
if( !d->arch.psr_cos_ids )
return;
- for_each_set_bit(socket, cat_socket_enable, nr_sockets)
+ for( socket = 0; socket < nr_sockets; socket++ )
{
if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
continue;
@@ -614,14 +617,14 @@ static void cat_cpu_init(void)
return;
socket = cpu_to_socket(cpu);
- if ( test_bit(socket, cat_socket_enable) )
+ info = cat_socket_info + socket;
+ if ( info->feat_mask )
return;
cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
if ( ebx & PSR_RESOURCE_TYPE_L3 )
{
cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
- info = cat_socket_info + socket;
info->l3_info.cbm_len = (eax & 0x1f) + 1;
info->l3_info.cos_max = min(opt_cos_max, edx & 0xffff);
@@ -630,10 +633,8 @@ static void cat_cpu_init(void)
spin_lock_init(&info->ref_lock);
- set_bit(socket, cat_socket_enable);
-
if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
- cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
+ !test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) )
{
/* CODE */
get_cdp_code(info, 0) = (1ull << info->l3_info.cbm_len) - 1;
@@ -649,8 +650,11 @@ static void cat_cpu_init(void)
/* Cut half of cos_max when CDP is enabled. */
info->l3_info.cos_max >>= 1;
- set_bit(socket, cdp_socket_enable);
+ __set_bit(PSR_SOCKET_L3_CDP, &info->feat_mask);
}
+ else
+ __set_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
+
printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
socket, info->l3_info.cos_max, info->l3_info.cbm_len,
cdp_is_enabled(socket) ? "on" : "off");
@@ -663,17 +667,17 @@ static void cat_cpu_fini(unsigned int cpu)
if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
{
- if ( cdp_is_enabled(socket) )
- clear_bit(socket, cdp_socket_enable);
+ struct psr_cat_socket_info *info = cat_socket_info + socket;
- clear_bit(socket, cat_socket_enable);
+ if ( cdp_is_enabled(socket) )
+ clear_bit(PSR_SOCKET_L3_CDP, &info->feat_mask);
+ else
+ clear_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
}
}
static void __init psr_cat_free(void)
{
- xfree(cat_socket_enable);
- cat_socket_enable = NULL;
xfree(cat_socket_info);
cat_socket_info = NULL;
}
@@ -686,12 +690,10 @@ static void __init init_psr_cat(void)
return;
}
- cat_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
cat_socket_info = xzalloc_array(struct psr_cat_socket_info, nr_sockets);
- cdp_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
- if ( !cat_socket_enable || !cat_socket_info )
- psr_cat_free();
+ if ( !cat_socket_info )
+ printk(XENLOG_INFO "Fail to alloc socket_info!\n");
}
static int psr_cpu_prepare(unsigned int cpu)
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v3 05/15] x86: refactor psr: Use 'feat_mask' to record featues enabled.
2016-10-25 3:40 ` [PATCH v3 05/15] x86: refactor psr: Use 'feat_mask' to record featues enabled Yi Sun
@ 2016-11-25 16:36 ` Jan Beulich
0 siblings, 0 replies; 42+ messages in thread
From: Jan Beulich @ 2016-11-25 16:36 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
>>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote:
> @@ -33,6 +36,8 @@ struct psr_cat_hw_info {
> };
>
> struct psr_cat_socket_info {
> + /* bit 1~0: [01]->L3 CAT-only, [10]->L3 CDP */
> + unsigned int feat_mask;
For the moment this looks like it wants to be an enumeration. Are
you going to later add bits such that more than one can be set at
a time?
> @@ -238,7 +241,7 @@ static inline void psr_assoc_init(void)
> {
> unsigned int socket = cpu_to_socket(smp_processor_id());
>
> - if ( test_bit(socket, cat_socket_enable) )
> + if ( cat_socket_info[socket].feat_mask )
> psra->cos_mask = ((1ull << get_count_order(
> cat_socket_info[socket].l3_info.cos_max)) - 1) << 32;
With it being unclear what this is going to be used for later, it
seems questionable that you test for any flag to be set, rather
than just the low two ones. The more that this guard a
construct which has been made (more obvious that it is?) L3
CAT specific by a prior patch.
This would then of course also affect other parts of the patch.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH v3 06/15] x86: refactor psr: Create feature list.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (4 preceding siblings ...)
2016-10-25 3:40 ` [PATCH v3 05/15] x86: refactor psr: Use 'feat_mask' to record featues enabled Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-10-25 3:40 ` [PATCH v3 07/15] x86: refactor psr: Implement feature operations structure Yi Sun
` (9 subsequent siblings)
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
This patch creates a feature list to manage features enabled on
socket. A feature list entry contains the feature's info, including
which feature it is, the feature HW info and the COS registers
values.
So far, only is L3 CAT/CDP added into list. So, the operations are
L3 CAT/CDP specific. The universal operations will be abstracted
and implemented in following patches.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 192 +++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 142 insertions(+), 50 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index aa902d2..38a64f0 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -17,6 +17,7 @@
#include <xen/cpu.h>
#include <xen/err.h>
#include <xen/sched.h>
+#include <xen/list.h>
#include <asm/psr.h>
#define PSR_CMT (1<<0)
@@ -26,8 +27,10 @@
/* Per spec, the maximum COS register number is 128. */
#define MAX_COS_REG_NUM 128
-#define PSR_SOCKET_L3_CAT 0
-#define PSR_SOCKET_L3_CDP 1
+enum psr_feat_type {
+ PSR_SOCKET_L3_CAT = 0,
+ PSR_SOCKET_L3_CDP,
+};
/* CAT/CDP HW info data structure. */
struct psr_cat_hw_info {
@@ -35,16 +38,25 @@ struct psr_cat_hw_info {
unsigned int cos_max;
};
-struct psr_cat_socket_info {
- /* bit 1~0: [01]->L3 CAT-only, [10]->L3 CDP */
- unsigned int feat_mask;
- struct psr_cat_hw_info l3_info;
+struct feat_node {
+ /* Which feature it is. */
+ enum psr_feat_type feature;
+ /* Feature HW info. */
+ struct psr_cat_hw_info info;
/*
* Store the values of COS registers:
* CAT uses 1 entry for one COS ID;
* CDP uses 2 entries for one COS ID and DATA is the first one.
*/
uint64_t cos_reg_val[MAX_COS_REG_NUM];
+ struct list_head list;
+};
+
+struct psr_cat_socket_info {
+ /* bit 1~0: [01]->L3 CAT-only, [10]->L3 CDP */
+ unsigned int feat_mask;
+ unsigned int nr_feat;
+ struct list_head feat_list;
/*
* Every entry of cos_ref is the reference count of a COS register.
* One entry of cos_ref corresponds to one COS ID.
@@ -56,18 +68,18 @@ struct psr_cat_socket_info {
/*
* get_data - get DATA COS register value from input COS ID.
- * @info: the struct psr_cat_socket_info pointer.
+ * @feat: the feature list entry.
* @cos: the COS ID.
*/
-#define get_cdp_data(info, cos) \
- info->cos_reg_val[cos * 2]
+#define get_cdp_data(feat, cos) \
+ feat->cos_reg_val[cos * 2]
/*
* get_cdp_code - get CODE COS register value from input COS ID.
- * @info: the struct psr_cat_socket_info pointer.
+ * @feat: the feature list entry.
* @cos: the COS ID.
*/
-#define get_cdp_code(info, cos) \
- info->cos_reg_val[cos * 2 + 1]
+#define get_cdp_code(feat, cos) \
+ feat->cos_reg_val[cos * 2 + 1]
struct psr_assoc {
uint64_t val;
@@ -84,6 +96,43 @@ static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_NUM - 1;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
+/* Feature list entry of feature L3 CAT/CDP. */
+static struct feat_node *feat_l3;
+
+static struct feat_node *get_feat_l3(struct psr_cat_socket_info *info)
+{
+ struct feat_node *feat_tmp;
+
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ if ( feat_tmp->feature == PSR_SOCKET_L3_CAT ||
+ feat_tmp->feature == PSR_SOCKET_L3_CDP )
+ return feat_tmp;
+
+ return NULL;
+}
+
+static void free_feature(struct psr_cat_socket_info *info)
+{
+ struct feat_node *feat_tmp;
+
+ if ( !info )
+ return;
+
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ {
+ clear_bit(feat_tmp->feature, &info->feat_mask);
+ list_del(&feat_tmp->list);
+ xfree(feat_tmp);
+ }
+
+ /* Free feature which are not added into feat_list. */
+ if ( feat_l3 )
+ {
+ xfree(feat_l3);
+ feat_l3 = NULL;
+ }
+}
+
static unsigned int get_socket_cpu(unsigned int socket)
{
if ( likely(socket < nr_sockets) )
@@ -240,10 +289,16 @@ static inline void psr_assoc_init(void)
if ( cat_socket_info )
{
unsigned int socket = cpu_to_socket(smp_processor_id());
+ struct psr_cat_socket_info *info = cat_socket_info + socket;
+ struct feat_node *feat_tmp;
+
+ feat_tmp = get_feat_l3(info);
+ if ( !feat_tmp )
+ return;
- if ( cat_socket_info[socket].feat_mask )
+ if ( info->feat_mask )
psra->cos_mask = ((1ull << get_count_order(
- cat_socket_info[socket].l3_info.cos_max)) - 1) << 32;
+ feat_tmp->info.cos_max)) - 1) << 32;
}
if ( psr_cmt_enabled() || psra->cos_mask )
@@ -303,12 +358,17 @@ int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
uint32_t *cos_max, uint32_t *flags)
{
struct psr_cat_socket_info *info = get_cat_socket_info(socket);
+ struct feat_node *feat_tmp;
if ( IS_ERR(info) )
return PTR_ERR(info);
- *cbm_len = info->l3_info.cbm_len;
- *cos_max = info->l3_info.cos_max;
+ feat_tmp = get_feat_l3(info);
+ if ( !feat_tmp )
+ return -ENOENT;
+
+ *cbm_len = feat_tmp->info.cbm_len;
+ *cos_max = feat_tmp->info.cos_max;
*flags = 0;
if ( cdp_is_enabled(socket) )
@@ -323,30 +383,35 @@ int psr_get_l3_cbm(struct domain *d, unsigned int socket,
struct psr_cat_socket_info *info = get_cat_socket_info(socket);
bool_t cdp_enabled = cdp_is_enabled(socket);
unsigned int cos = d->arch.psr_cos_ids[socket];
+ struct feat_node *feat_tmp;
if ( IS_ERR(info) )
return PTR_ERR(info);
+ feat_tmp = get_feat_l3(info);
+ if ( !feat_tmp )
+ return -ENOENT;
+
switch ( type )
{
case PSR_CBM_TYPE_L3:
if ( cdp_enabled )
return -EXDEV;
- *cbm = info->cos_reg_val[cos];
+ *cbm = feat_tmp->cos_reg_val[cos];
break;
case PSR_CBM_TYPE_L3_CODE:
if ( !cdp_enabled )
- *cbm = info->cos_reg_val[cos];
+ *cbm = feat_tmp->cos_reg_val[cos];
else
- *cbm = get_cdp_code(info, cos);
+ *cbm = get_cdp_code(feat_tmp, cos);
break;
case PSR_CBM_TYPE_L3_DATA:
if ( !cdp_enabled )
- *cbm = info->cos_reg_val[cos];
+ *cbm = feat_tmp->cos_reg_val[cos];
else
- *cbm = get_cdp_data(info, cos);
+ *cbm = get_cdp_data(feat_tmp, cos);
break;
default:
@@ -425,7 +490,7 @@ static int write_l3_cbm(unsigned int socket, unsigned int cos,
return 0;
}
-static int find_cos(struct psr_cat_socket_info *info, unsigned int *ref,
+static int find_cos(struct feat_node *feat, unsigned int *ref,
unsigned int cos_max,
uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
{
@@ -434,9 +499,9 @@ static int find_cos(struct psr_cat_socket_info *info, unsigned int *ref,
for ( cos = 0; cos <= cos_max; cos++ )
{
if ( (ref[cos] || cos == 0) &&
- ((!cdp_enabled && info->cos_reg_val[cos] == cbm_code) ||
- (cdp_enabled && get_cdp_code(info, cos) == cbm_code &&
- get_cdp_data(info, cos) == cbm_data)) )
+ ((!cdp_enabled && feat->cos_reg_val[cos] == cbm_code) ||
+ (cdp_enabled && get_cdp_code(feat, cos) == cbm_code &&
+ get_cdp_data(feat, cos) == cbm_data)) )
return cos;
}
@@ -469,18 +534,23 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
bool_t cdp_enabled = cdp_is_enabled(socket);
struct psr_cat_socket_info *info = get_cat_socket_info(socket);
unsigned int *ref;
+ struct feat_node *feat_tmp;
if ( IS_ERR(info) )
return PTR_ERR(info);
- if ( !psr_check_cbm(info->l3_info.cbm_len, cbm) )
+ feat_tmp = get_feat_l3(info);
+ if ( !feat_tmp )
+ return -ENOENT;
+
+ if ( !psr_check_cbm(feat_tmp->info.cbm_len, cbm) )
return -EINVAL;
if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
type == PSR_CBM_TYPE_L3_DATA) )
return -ENXIO;
- cos_max = info->l3_info.cos_max;
+ cos_max = feat_tmp->info.cos_max;
old_cos = d->arch.psr_cos_ids[socket];
ref = info->cos_ref;
@@ -493,11 +563,11 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
case PSR_CBM_TYPE_L3_CODE:
cbm_code = cbm;
- cbm_data = get_cdp_data(info, old_cos);
+ cbm_data = get_cdp_data(feat_tmp, old_cos);
break;
case PSR_CBM_TYPE_L3_DATA:
- cbm_code = get_cdp_code(info, old_cos);
+ cbm_code = get_cdp_code(feat_tmp, old_cos);
cbm_data = cbm;
break;
@@ -507,7 +577,7 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
}
spin_lock(&info->ref_lock);
- cos = find_cos(info, ref, cos_max, cbm_code, cbm_data, cdp_enabled);
+ cos = find_cos(feat_tmp, ref, cos_max, cbm_code, cbm_data, cdp_enabled);
if ( cos >= 0 )
{
if ( cos == old_cos )
@@ -527,9 +597,9 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
/* We try to avoid writing MSR. */
if ( (cdp_enabled &&
- (get_cdp_code(info, cos) != cbm_code ||
- get_cdp_data(info, cos) != cbm_data)) ||
- (!cdp_enabled && info->cos_reg_val[cos] != cbm_code) )
+ (get_cdp_code(feat_tmp, cos) != cbm_code ||
+ get_cdp_data(feat_tmp, cos) != cbm_data)) ||
+ (!cdp_enabled && feat_tmp->cos_reg_val[cos] != cbm_code) )
{
ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
if ( ret )
@@ -537,8 +607,8 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
spin_unlock(&info->ref_lock);
return ret;
}
- get_cdp_code(info, cos) = cbm_code;
- get_cdp_data(info, cos) = cbm_data;
+ get_cdp_code(feat_tmp, cos) = cbm_code;
+ get_cdp_data(feat_tmp, cos) = cbm_data;
}
}
@@ -599,7 +669,10 @@ static int cat_cpu_prepare(unsigned int cpu)
if ( !cat_socket_info )
return 0;
- /* Keep this function for future usage. */
+ /* Malloc memory for the global feature head here. */
+ if ( feat_l3 == NULL &&
+ (feat_l3 = xzalloc(struct feat_node)) == NULL )
+ return -ENOMEM;
return 0;
}
@@ -612,6 +685,7 @@ static void cat_cpu_init(void)
unsigned int cpu = smp_processor_id();
uint64_t val;
const struct cpuinfo_x86 *c = cpu_data + cpu;
+ struct feat_node *feat_tmp;
if ( !cpu_has(c, X86_FEATURE_PQE) || c->cpuid_level < PSR_CPUID_LEVEL_CAT )
return;
@@ -624,12 +698,15 @@ static void cat_cpu_init(void)
cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
if ( ebx & PSR_RESOURCE_TYPE_L3 )
{
+ feat_tmp = feat_l3;
+ feat_l3 = NULL;
+
cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
- info->l3_info.cbm_len = (eax & 0x1f) + 1;
- info->l3_info.cos_max = min(opt_cos_max, edx & 0xffff);
+ feat_tmp->info.cbm_len = (eax & 0x1f) + 1;
+ feat_tmp->info.cos_max = min(opt_cos_max, edx & 0xffff);
/* cos=0 is reserved as default cbm(all ones). */
- info->cos_reg_val[0] = (1ull << info->l3_info.cbm_len) - 1;
+ feat_tmp->cos_reg_val[0] = (1ull << feat_tmp->info.cbm_len) - 1;
spin_lock_init(&info->ref_lock);
@@ -637,26 +714,31 @@ static void cat_cpu_init(void)
!test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) )
{
/* CODE */
- get_cdp_code(info, 0) = (1ull << info->l3_info.cbm_len) - 1;
+ get_cdp_code(feat_tmp, 0) = (1ull << feat_tmp->info.cbm_len) - 1;
/* DATA */
- get_cdp_data(info, 0) = (1ull << info->l3_info.cbm_len) - 1;
+ get_cdp_data(feat_tmp, 0) = (1ull << feat_tmp->info.cbm_len) - 1;
/* We only write mask1 since mask0 is always all ones by default. */
- wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->l3_info.cbm_len) - 1);
+ wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << feat_tmp->info.cbm_len) - 1);
rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
/* Cut half of cos_max when CDP is enabled. */
- info->l3_info.cos_max >>= 1;
+ feat_tmp->info.cos_max >>= 1;
__set_bit(PSR_SOCKET_L3_CDP, &info->feat_mask);
- }
- else
+ } else {
+ feat_tmp->feature = PSR_SOCKET_L3_CAT;
__set_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
+ }
+
+ info->nr_feat++;
+ /* Add this feature into list. */
+ list_add_tail(&feat_tmp->list, &info->feat_list);
printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
- socket, info->l3_info.cos_max, info->l3_info.cbm_len,
+ socket, feat_tmp->info.cos_max, feat_tmp->info.cbm_len,
cdp_is_enabled(socket) ? "on" : "off");
}
}
@@ -669,21 +751,25 @@ static void cat_cpu_fini(unsigned int cpu)
{
struct psr_cat_socket_info *info = cat_socket_info + socket;
- if ( cdp_is_enabled(socket) )
- clear_bit(PSR_SOCKET_L3_CDP, &info->feat_mask);
- else
- clear_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
+ free_feature(info);
}
}
static void __init psr_cat_free(void)
{
+ unsigned int i;
+
+ for ( i = 0; i < nr_sockets; i++ )
+ free_feature(&cat_socket_info[i]);
+
xfree(cat_socket_info);
cat_socket_info = NULL;
}
static void __init init_psr_cat(void)
{
+ unsigned int i;
+
if ( opt_cos_max < 1 )
{
printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
@@ -693,7 +779,13 @@ static void __init init_psr_cat(void)
cat_socket_info = xzalloc_array(struct psr_cat_socket_info, nr_sockets);
if ( !cat_socket_info )
+ {
printk(XENLOG_INFO "Fail to alloc socket_info!\n");
+ return;
+ }
+
+ for ( i = 0; i < nr_sockets; i++ )
+ INIT_LIST_HEAD(&cat_socket_info[i].feat_list);
}
static int psr_cpu_prepare(unsigned int cpu)
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v3 07/15] x86: refactor psr: Implement feature operations structure.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (5 preceding siblings ...)
2016-10-25 3:40 ` [PATCH v3 06/15] x86: refactor psr: Create feature list Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-10-25 3:40 ` [PATCH v3 08/15] x86: refactor psr: Implement get hw info callback function Yi Sun
` (8 subsequent siblings)
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
To handle all features in a universal way, we need abstract the
common operations of all features and register different callback
functions for differnet features. The feature specific behaviors
should be encapsulated into these callback functions.
This patch defines 'struct feat_ops' to maintain features' callback
functions. Also implement the L3 CAT/CDP init callback function to
show how this mechanism work.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 123 ++++++++++++++++++++++++++++++++++++-----------------
1 file changed, 84 insertions(+), 39 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 38a64f0..750278c 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -32,6 +32,21 @@ enum psr_feat_type {
PSR_SOCKET_L3_CDP,
};
+struct feat_node;
+struct psr_cat_socket_info;
+
+/* Every feature enabled MUST implement such ops and callback functions. */
+struct feat_ops {
+ /*
+ * init_feature is used in cpu initialization process to do feature
+ * specific initialization works.
+ */
+ void (*init_feature)(unsigned int eax, unsigned int ebx,
+ unsigned int ecx, unsigned int edx,
+ struct feat_node *feat,
+ struct psr_cat_socket_info *info);
+};
+
/* CAT/CDP HW info data structure. */
struct psr_cat_hw_info {
unsigned int cbm_len;
@@ -41,6 +56,8 @@ struct psr_cat_hw_info {
struct feat_node {
/* Which feature it is. */
enum psr_feat_type feature;
+ /* Feature operation callback functions. */
+ struct feat_ops ops;
/* Feature HW info. */
struct psr_cat_hw_info info;
/*
@@ -133,6 +150,69 @@ static void free_feature(struct psr_cat_socket_info *info)
}
}
+/* L3 CAT/CDP callback functions implementation. */
+static void l3_cat_init_feature(unsigned int eax, unsigned int ebx,
+ unsigned int ecx, unsigned int edx,
+ struct feat_node *feat,
+ struct psr_cat_socket_info *info)
+{
+ struct psr_cat_hw_info l3_cat;
+ unsigned int socket;
+ uint64_t val;
+
+ /* No valid value so do not enable feature. */
+ if ( !eax || !edx )
+ return;
+
+ l3_cat.cbm_len = (eax & 0x1f) + 1;
+ l3_cat.cos_max = min(opt_cos_max, edx & 0xffff);
+
+ /* cos=0 is reserved as default cbm(all ones). */
+ feat->cos_reg_val[0] = (1ull << l3_cat.cbm_len) - 1;
+
+ if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
+ !test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) )
+ {
+ /* CODE */
+ get_cdp_code(feat, 0) =
+ (1ull << l3_cat.cbm_len) - 1;
+ /* DATA */
+ get_cdp_data(feat, 0) =
+ (1ull << l3_cat.cbm_len) - 1;
+
+ /* We only write mask1 since mask0 is always all ones by default. */
+ wrmsrl(MSR_IA32_PSR_L3_MASK(1),
+ (1ull << l3_cat.cbm_len) - 1);
+ rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
+ wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
+
+ /* Cut half of cos_max when CDP is enabled. */
+ l3_cat.cos_max >>= 1;
+
+ feat->feature = PSR_SOCKET_L3_CDP;
+ __set_bit(PSR_SOCKET_L3_CDP, &info->feat_mask);
+ } else {
+ feat->feature = PSR_SOCKET_L3_CAT;
+ __set_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
+ }
+
+ feat->info = l3_cat;
+
+ info->nr_feat++;
+
+ /* Add this feature into list. */
+ list_add_tail(&feat->list, &info->feat_list);
+
+ socket = cpu_to_socket(smp_processor_id());
+ printk(XENLOG_INFO "L3 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
+ socket, feat->info.cos_max, feat->info.cbm_len,
+ test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) ? "on" : "off");
+}
+
+struct feat_ops l3_cat_ops = {
+ .init_feature = l3_cat_init_feature,
+};
+
static unsigned int get_socket_cpu(unsigned int socket)
{
if ( likely(socket < nr_sockets) )
@@ -683,7 +763,6 @@ static void cat_cpu_init(void)
struct psr_cat_socket_info *info;
unsigned int socket;
unsigned int cpu = smp_processor_id();
- uint64_t val;
const struct cpuinfo_x86 *c = cpu_data + cpu;
struct feat_node *feat_tmp;
@@ -695,6 +774,8 @@ static void cat_cpu_init(void)
if ( info->feat_mask )
return;
+ spin_lock_init(&info->ref_lock);
+
cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
if ( ebx & PSR_RESOURCE_TYPE_L3 )
{
@@ -702,44 +783,8 @@ static void cat_cpu_init(void)
feat_l3 = NULL;
cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
- feat_tmp->info.cbm_len = (eax & 0x1f) + 1;
- feat_tmp->info.cos_max = min(opt_cos_max, edx & 0xffff);
-
- /* cos=0 is reserved as default cbm(all ones). */
- feat_tmp->cos_reg_val[0] = (1ull << feat_tmp->info.cbm_len) - 1;
-
- spin_lock_init(&info->ref_lock);
-
- if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
- !test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) )
- {
- /* CODE */
- get_cdp_code(feat_tmp, 0) = (1ull << feat_tmp->info.cbm_len) - 1;
- /* DATA */
- get_cdp_data(feat_tmp, 0) = (1ull << feat_tmp->info.cbm_len) - 1;
-
- /* We only write mask1 since mask0 is always all ones by default. */
- wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << feat_tmp->info.cbm_len) - 1);
-
- rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
- wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
-
- /* Cut half of cos_max when CDP is enabled. */
- feat_tmp->info.cos_max >>= 1;
-
- __set_bit(PSR_SOCKET_L3_CDP, &info->feat_mask);
- } else {
- feat_tmp->feature = PSR_SOCKET_L3_CAT;
- __set_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
- }
-
- info->nr_feat++;
- /* Add this feature into list. */
- list_add_tail(&feat_tmp->list, &info->feat_list);
-
- printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
- socket, feat_tmp->info.cos_max, feat_tmp->info.cbm_len,
- cdp_is_enabled(socket) ? "on" : "off");
+ feat_tmp->ops = l3_cat_ops;
+ feat_tmp->ops.init_feature(eax, ebx, ecx, edx, feat_tmp, info);
}
}
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v3 08/15] x86: refactor psr: Implement get hw info callback function
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (6 preceding siblings ...)
2016-10-25 3:40 ` [PATCH v3 07/15] x86: refactor psr: Implement feature operations structure Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-10-25 3:40 ` [PATCH v3 09/15] x86: refactor psr: Implement get value " Yi Sun
` (7 subsequent siblings)
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
Get feature HW info is an interface provided by psr.c. This
can be abstracted as an operation.
This patch defines 'get_feat_info' callback function to get
the feature HW info and implement the callback function for
L3 CAT/CDP.
It also modifies the sysctl interface to make it general.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 55 ++++++++++++++++++++++++++++++++++++-----------
xen/arch/x86/sysctl.c | 17 ++++++++++-----
xen/include/asm-x86/psr.h | 4 ++--
3 files changed, 56 insertions(+), 20 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 750278c..acd180c 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -27,6 +27,10 @@
/* Per spec, the maximum COS register number is 128. */
#define MAX_COS_REG_NUM 128
+#define CBM_LEN 0
+#define COS_MAX 1
+#define CDP_FLAG 2
+
enum psr_feat_type {
PSR_SOCKET_L3_CAT = 0,
PSR_SOCKET_L3_CDP,
@@ -45,6 +49,9 @@ struct feat_ops {
unsigned int ecx, unsigned int edx,
struct feat_node *feat,
struct psr_cat_socket_info *info);
+ /* get_feat_info is used to get feature HW info. */
+ bool (*get_feat_info)(const struct feat_node *feat, enum cbm_type type,
+ uint32_t dat[], uint32_t array_len);
};
/* CAT/CDP HW info data structure. */
@@ -209,8 +216,37 @@ static void l3_cat_init_feature(unsigned int eax, unsigned int ebx,
test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) ? "on" : "off");
}
+static bool l3_cat_get_feat_info(const struct feat_node *feat,
+ enum cbm_type type,
+ uint32_t dat[], uint32_t array_len)
+{
+ if ( !dat || 3 > array_len )
+ return false;
+
+ switch ( type ) {
+ case PSR_CBM_TYPE_L3:
+ case PSR_CBM_TYPE_L3_DATA:
+ case PSR_CBM_TYPE_L3_CODE:
+ if ( feat->feature == PSR_SOCKET_L3_CDP )
+ dat[CDP_FLAG] |= XEN_SYSCTL_PSR_CAT_L3_CDP;
+ else
+ dat[CDP_FLAG] = 0;
+ break;
+
+ default:
+ /* This feature is not requested feature. */
+ return false;
+ }
+
+ dat[CBM_LEN] = feat->info.cbm_len;
+ dat[COS_MAX] = feat->info.cos_max;
+
+ return true;
+}
+
struct feat_ops l3_cat_ops = {
.init_feature = l3_cat_init_feature,
+ .get_feat_info = l3_cat_get_feat_info,
};
static unsigned int get_socket_cpu(unsigned int socket)
@@ -434,8 +470,8 @@ static inline bool_t cdp_is_enabled(unsigned int socket)
return test_bit(PSR_SOCKET_L3_CDP, &cat_socket_info[socket].feat_mask);
}
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags)
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t dat[], uint32_t array_len)
{
struct psr_cat_socket_info *info = get_cat_socket_info(socket);
struct feat_node *feat_tmp;
@@ -443,18 +479,11 @@ int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
if ( IS_ERR(info) )
return PTR_ERR(info);
- feat_tmp = get_feat_l3(info);
- if ( !feat_tmp )
- return -ENOENT;
-
- *cbm_len = feat_tmp->info.cbm_len;
- *cos_max = feat_tmp->info.cos_max;
-
- *flags = 0;
- if ( cdp_is_enabled(socket) )
- *flags |= XEN_SYSCTL_PSR_CAT_L3_CDP;
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ if ( feat_tmp->ops.get_feat_info(feat_tmp, type, dat, array_len) )
+ return 0;
- return 0;
+ return -ENOENT;
}
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 14e7dc7..e82adec 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -83,6 +83,9 @@ void arch_do_physinfo(xen_sysctl_physinfo_t *pi)
pi->capabilities |= XEN_SYSCTL_PHYSCAP_hvm_directio;
}
+#define CBM_LEN 0
+#define COS_MAX 1
+#define CDP_FLAG 2
long arch_do_sysctl(
struct xen_sysctl *sysctl, XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
{
@@ -176,15 +179,19 @@ long arch_do_sysctl(
switch ( sysctl->u.psr_cat_op.cmd )
{
case XEN_SYSCTL_PSR_CAT_get_l3_info:
- ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
- &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
- &sysctl->u.psr_cat_op.u.l3_info.cos_max,
- &sysctl->u.psr_cat_op.u.l3_info.flags);
+ {
+ uint32_t dat[3];
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L3, dat, 3);
+
+ sysctl->u.psr_cat_op.u.l3_info.cbm_len = dat[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l3_info.cos_max = dat[COS_MAX];
+ sysctl->u.psr_cat_op.u.l3_info.flags = dat[CDP_FLAG];
if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
ret = -EFAULT;
break;
-
+ }
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 57f47e9..eccfa75 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -63,8 +63,8 @@ int psr_alloc_rmid(struct domain *d);
void psr_free_rmid(struct domain *d);
void psr_ctxt_switch_to(struct domain *d);
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
- uint32_t *cos_max, uint32_t *flags);
+int psr_get_info(unsigned int socket, enum cbm_type type,
+ uint32_t dat[], uint32_t array_len);
int psr_get_l3_cbm(struct domain *d, unsigned int socket,
uint64_t *cbm, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v3 09/15] x86: refactor psr: Implement get value callback function.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (7 preceding siblings ...)
2016-10-25 3:40 ` [PATCH v3 08/15] x86: refactor psr: Implement get hw info callback function Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-10-25 3:40 ` [PATCH v3 10/15] x86: refactor psr: Implement function to get the max cos_max Yi Sun
` (6 subsequent siblings)
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
Get feature COS register value is an interface provided by psr.c.
It can be abstracted as another operation.
This patch adds 'get_val' callback function to get the COS register
value of the feature and implement the callback function for L3
CAT/CDP.
It also modifies the domctl interface to make it general.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/domctl.c | 18 +++++------
xen/arch/x86/psr.c | 79 ++++++++++++++++++++++++++++-------------------
xen/include/asm-x86/psr.h | 4 +--
3 files changed, 59 insertions(+), 42 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 2a2fe04..3d7fc34 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1404,23 +1404,23 @@ long arch_do_domctl(
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3);
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_CODE);
copyback = 1;
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
- ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
- &domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_DATA);
copyback = 1;
break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index acd180c..b8bb09b 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -52,6 +52,9 @@ struct feat_ops {
/* get_feat_info is used to get feature HW info. */
bool (*get_feat_info)(const struct feat_node *feat, enum cbm_type type,
uint32_t dat[], uint32_t array_len);
+ /* get_val is used to get feature COS register value. */
+ int (*get_val)(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val);
};
/* CAT/CDP HW info data structure. */
@@ -244,9 +247,42 @@ static bool l3_cat_get_feat_info(const struct feat_node *feat,
return true;
}
+static int l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val)
+{
+ if ( cos > feat->info.cos_max )
+ /* Use default value. */
+ cos = 0;
+
+ switch ( type ) {
+ /* CDP */
+ case PSR_CBM_TYPE_L3_DATA:
+ case PSR_CBM_TYPE_L3_CODE:
+ if ( feat->feature != PSR_SOCKET_L3_CDP )
+ return -ENXIO;
+
+ if ( type == PSR_CBM_TYPE_L3_DATA )
+ *val = get_cdp_data(feat, cos);
+ else
+ *val = get_cdp_code(feat, cos);
+ break;
+
+ /* CAT */
+ case PSR_CBM_TYPE_L3:
+ *val = feat->cos_reg_val[cos];
+ break;
+
+ default:
+ return 0;
+ }
+
+ return 1;
+}
+
struct feat_ops l3_cat_ops = {
.init_feature = l3_cat_init_feature,
.get_feat_info = l3_cat_get_feat_info,
+ .get_val = l3_cat_get_val,
};
static unsigned int get_socket_cpu(unsigned int socket)
@@ -486,45 +522,26 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
return -ENOENT;
}
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type)
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint64_t *val, enum cbm_type type)
{
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
- bool_t cdp_enabled = cdp_is_enabled(socket);
+ const struct psr_cat_socket_info *info = get_cat_socket_info(socket);
unsigned int cos = d->arch.psr_cos_ids[socket];
- struct feat_node *feat_tmp;
+ const struct feat_node *feat_tmp;
+ int ret;
if ( IS_ERR(info) )
return PTR_ERR(info);
- feat_tmp = get_feat_l3(info);
- if ( !feat_tmp )
- return -ENOENT;
-
- switch ( type )
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
{
- case PSR_CBM_TYPE_L3:
- if ( cdp_enabled )
- return -EXDEV;
- *cbm = feat_tmp->cos_reg_val[cos];
- break;
+ ret = feat_tmp->ops.get_val(feat_tmp, cos, type, val);
+ if ( ret < 0 )
+ return ret;
- case PSR_CBM_TYPE_L3_CODE:
- if ( !cdp_enabled )
- *cbm = feat_tmp->cos_reg_val[cos];
- else
- *cbm = get_cdp_code(feat_tmp, cos);
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- if ( !cdp_enabled )
- *cbm = feat_tmp->cos_reg_val[cos];
- else
- *cbm = get_cdp_data(feat_tmp, cos);
- break;
-
- default:
- ASSERT_UNREACHABLE();
+ if ( ret > 0 )
+ /* Found */
+ break;
}
return 0;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index eccfa75..f2872a2 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -65,8 +65,8 @@ void psr_ctxt_switch_to(struct domain *d);
int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t dat[], uint32_t array_len);
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t *cbm, enum cbm_type type);
+int psr_get_val(struct domain *d, unsigned int socket,
+ uint64_t *val, enum cbm_type type);
int psr_set_l3_cbm(struct domain *d, unsigned int socket,
uint64_t cbm, enum cbm_type type);
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v3 10/15] x86: refactor psr: Implement function to get the max cos_max.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (8 preceding siblings ...)
2016-10-25 3:40 ` [PATCH v3 09/15] x86: refactor psr: Implement get value " Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-10-25 3:40 ` [PATCH v3 11/15] x86: refactor psr: Implement set value callback function Yi Sun
` (5 subsequent siblings)
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
When set ASSOC register, we need a cos_mask to calculate the COS
ID to set. The cos_mask is calculated by cos_max. When supporting
more than one feature, the cos_max to calculate cos_mask should be
the max one of all features.
This patch implements 'get_max_cos_max' function and the corresponding
callback function to get the max cos_max of all features.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 31 +++++++++++++++++++++++--------
1 file changed, 23 insertions(+), 8 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index b8bb09b..fdf4e3a 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -55,6 +55,8 @@ struct feat_ops {
/* get_val is used to get feature COS register value. */
int (*get_val)(const struct feat_node *feat, unsigned int cos,
enum cbm_type type, uint64_t *val);
+ /* get_max_cos_max is used to get feature's cos_max. */
+ unsigned int (*get_max_cos_max)(const struct feat_node *feat);
};
/* CAT/CDP HW info data structure. */
@@ -279,10 +281,16 @@ static int l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
return 1;
}
+static unsigned int l3_cat_get_max_cos_max(const struct feat_node *feat)
+{
+ return feat->info.cos_max;
+}
+
struct feat_ops l3_cat_ops = {
.init_feature = l3_cat_init_feature,
.get_feat_info = l3_cat_get_feat_info,
.get_val = l3_cat_get_val,
+ .get_max_cos_max = l3_cat_get_max_cos_max,
};
static unsigned int get_socket_cpu(unsigned int socket)
@@ -434,6 +442,18 @@ void psr_free_rmid(struct domain *d)
d->arch.psr_rmid = 0;
}
+static inline unsigned int get_max_cos_max(
+ const struct psr_cat_socket_info *info)
+{
+ const struct feat_node *feat_tmp;
+ unsigned int cos_max = 0;
+
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ cos_max = max(feat_tmp->ops.get_max_cos_max(feat_tmp), cos_max);
+
+ return cos_max;
+}
+
static inline void psr_assoc_init(void)
{
struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -441,16 +461,11 @@ static inline void psr_assoc_init(void)
if ( cat_socket_info )
{
unsigned int socket = cpu_to_socket(smp_processor_id());
- struct psr_cat_socket_info *info = cat_socket_info + socket;
- struct feat_node *feat_tmp;
-
- feat_tmp = get_feat_l3(info);
- if ( !feat_tmp )
- return;
+ const struct psr_cat_socket_info *info = cat_socket_info + socket;
+ unsigned int cos_max = get_max_cos_max(info);
if ( info->feat_mask )
- psra->cos_mask = ((1ull << get_count_order(
- feat_tmp->info.cos_max)) - 1) << 32;
+ psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) << 32;
}
if ( psr_cmt_enabled() || psra->cos_mask )
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v3 11/15] x86: refactor psr: Implement set value callback function.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (9 preceding siblings ...)
2016-10-25 3:40 ` [PATCH v3 10/15] x86: refactor psr: Implement function to get the max cos_max Yi Sun
@ 2016-10-25 3:40 ` Yi Sun
2016-10-25 3:41 ` [PATCH v3 12/15] x86: Implement L2 CAT in psr.c Yi Sun
` (4 subsequent siblings)
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:40 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
This patch modifies the set value interface provided by psr.c.
It makes it be general and change the logic in the function to
make it be compatible for supporting multiple features. It
includes below steps:
1. Assemble a value array to store all features current value
in it and replace the current value of the feature which is
being set to the new input value.
2. Find if there is already a COS ID on which all features'
values are same as the array. Then, we can reuse this COS
ID.
3. If fail to find, we need allocate a new COS ID.
4. Write all features MSRs corresponding to the COS ID.
So, some functions are abstracted and the callback functions
of L3 CAT/CDP are implemented.
Furthermore, the domctl interface is also modified to make it
general.
Here is an example to understand the process. The CPU supports
two featuers, e.g. L3 CAT and L2 CAT. user wants to set L3 CAT
of Dom1 to 0x1ff. The old_cos of Dom1 is 0. L3 CAT is the first
element of feature list. The COS registers values are below at
this time.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | ... | ... | ... |
-------------------------------
L2 CAT | 0xff | ... | ... | ... |
-------------------------------
The value array assembled is:
val[0]: 0x1ff
val[1]: 0xff
It cannot find a matching COS so allocate COS 1 to store the
value set. The COS registers values are changed to below now.
-------------------------------
| COS 0 | COS 1 | COS 2 | ... |
-------------------------------
L3 CAT | 0x7ff | 0x1ff | ... | ... |
-------------------------------
L2 CAT | 0xff | 0xff | ... | ... |
-------------------------------
Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
of Dom2 is 0 too.
The val array assembled is:
val[0]: 0x1ff
val[1]: 0xff
So, it can find a matching COS, COS 1. Then, it can reuse COS 1
for Dom2.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/domctl.c | 18 +-
xen/arch/x86/psr.c | 679 +++++++++++++++++++++++++++++++++++++---------
xen/include/asm-x86/psr.h | 4 +-
3 files changed, 566 insertions(+), 135 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 3d7fc34..8e5502a 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1386,21 +1386,21 @@ long arch_do_domctl(
switch ( domctl->u.psr_cat_op.cmd )
{
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_CODE);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_CODE);
break;
case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA:
- ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
- domctl->u.psr_cat_op.data,
- PSR_CBM_TYPE_L3_DATA);
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L3_DATA);
break;
case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index fdf4e3a..32f899a 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -39,6 +39,17 @@ enum psr_feat_type {
struct feat_node;
struct psr_cat_socket_info;
+/*
+ * To add a new PSR feature, please follow below steps:
+ * 1. Implement feature specific callback functions listed in
+ * "struct feat_ops";
+ * 2. Implement a "struct feat_ops" instance for the feature and register
+ * the feature callback functions into it;
+ * 3. Delcare feat_node instance for the feature and malloc memory for it
+ * in cat_cpu_prepare(). Correspondingly, free it in free_feature();
+ * 4. Add feature initialization codes in cat_cpu_init().
+ */
+
/* Every feature enabled MUST implement such ops and callback functions. */
struct feat_ops {
/*
@@ -57,6 +68,74 @@ struct feat_ops {
enum cbm_type type, uint64_t *val);
/* get_max_cos_max is used to get feature's cos_max. */
unsigned int (*get_max_cos_max)(const struct feat_node *feat);
+ /*
+ * get_cos_num is used to get the COS registers amount used by the
+ * feature for one setting, e.g. CDP uses 2 COSs but CAT uses 1.
+ */
+ unsigned int (*get_cos_num)(const struct feat_node *feat);
+ /*
+ * get_old_val and set_new_val are a pair of functions called together.
+ * The caller will traverse all features in the list and call both
+ * functions for every feature to do below two things:
+ * 1. get old_cos register value of all supported features and
+ * 2. set the new value for the feature.
+ *
+ * All the values are set into value array according the traversal order,
+ * meaning the same order of feature list members.
+ *
+ * The return value is the amount of entries to skip in the value array
+ * or error.
+ * 1 - one entry in value array.
+ * 2 - two entries in value array, e.g. CDP uses two entries.
+ * negative - error.
+ */
+ int (*get_old_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos);
+ int (*set_new_val)(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos,
+ enum cbm_type type,
+ uint64_t m);
+ /*
+ * compare_val is used in set value process to compare if the
+ * input value array can match all the features' COS registers values
+ * according to input cos id.
+ *
+ * The return value is the amount of entries to skip in the value array
+ * or error.
+ * 1 - one entry in value array.
+ * 2 - two entries in value array, e.g. CDP uses two entries.
+ * negative - error.
+ */
+ int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
+ unsigned int cos, bool *found);
+ /*
+ * get_cos_max_from_type is used to get the cos_max value of the feature
+ * according to input type.
+ */
+ unsigned int (*get_cos_max_from_type)(const struct feat_node *feat,
+ enum cbm_type type);
+ /*
+ * exceeds_cos_max is used to check if the input cos id exceeds the
+ * feature's cos_max and if the input value is not the default one.
+ * Even if the associated cos exceeds the cos_max, HW can work with default
+ * value. That is the reason we need check if input value is default one.
+ * If both criteria are fulfilled, that means the input exceeds the
+ * range.
+ *
+ * The return value of the function means the number of the value array
+ * entries to skip or error.
+ * 1 - one entry in value array.
+ * 2 - two entries in value array, e.g. CDP uses two entries.
+ * 0 - error, exceed cos_max and the input value is not default.
+ */
+ unsigned int (*exceeds_cos_max)(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos);
+ /* write_msr is used to write out feature MSR register. */
+ int (*write_msr)(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat);
};
/* CAT/CDP HW info data structure. */
@@ -128,18 +207,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
/* Feature list entry of feature L3 CAT/CDP. */
static struct feat_node *feat_l3;
-static struct feat_node *get_feat_l3(struct psr_cat_socket_info *info)
-{
- struct feat_node *feat_tmp;
-
- list_for_each_entry(feat_tmp, &info->feat_list, list)
- if ( feat_tmp->feature == PSR_SOCKET_L3_CAT ||
- feat_tmp->feature == PSR_SOCKET_L3_CDP )
- return feat_tmp;
-
- return NULL;
-}
-
+/* Common functions. */
static void free_feature(struct psr_cat_socket_info *info)
{
struct feat_node *feat_tmp;
@@ -162,6 +230,29 @@ static void free_feature(struct psr_cat_socket_info *info)
}
}
+static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
+{
+ unsigned int first_bit, zero_bit;
+
+ /* Set bits should only in the range of [0, cbm_len). */
+ if ( cbm & (~0ull << cbm_len) )
+ return 0;
+
+ /* At least one bit need to be set. */
+ if ( cbm == 0 )
+ return 0;
+
+ first_bit = find_first_bit(&cbm, cbm_len);
+ zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
+
+ /* Set bits should be contiguous. */
+ if ( zero_bit < cbm_len &&
+ find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
+ return 0;
+
+ return 1;
+}
+
/* L3 CAT/CDP callback functions implementation. */
static void l3_cat_init_feature(unsigned int eax, unsigned int ebx,
unsigned int ecx, unsigned int edx,
@@ -286,11 +377,235 @@ static unsigned int l3_cat_get_max_cos_max(const struct feat_node *feat)
return feat->info.cos_max;
}
+static unsigned int l3_cat_get_cos_num(const struct feat_node *feat)
+{
+ if ( feat->feature == PSR_SOCKET_L3_CDP )
+ return 2;
+
+ return 1;
+}
+
+static int l3_cat_get_old_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos)
+{
+ if ( old_cos > feat->info.cos_max )
+ /* Use default value. */
+ old_cos = 0;
+
+ /* CDP */
+ if ( feat->feature == PSR_SOCKET_L3_CDP )
+ {
+ /* Data */
+ val[0] = get_cdp_data(feat, old_cos);
+ /* Code */
+ val[1] = get_cdp_code(feat, old_cos);
+
+ /* CDP uses two COSs, one for data, one for code. */
+ return 2;
+ }
+
+ /* CAT */
+ val[0] = feat->cos_reg_val[old_cos];
+
+ /* L3 CAT uses one COS. */
+ return 1;
+}
+
+static int l3_cat_set_new_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos,
+ enum cbm_type type,
+ uint64_t m)
+{
+ /* L3 CAT uses one COS. */
+ unsigned int ret = 1;
+
+ switch ( type ) {
+ /* CDP uses two COS, one for data, one for code. */
+ case PSR_CBM_TYPE_L3_DATA:
+ case PSR_CBM_TYPE_L3_CODE:
+ if ( feat->feature != PSR_SOCKET_L3_CDP )
+ return -ENXIO;
+
+ if ( !psr_check_cbm(feat->info.cbm_len, m) )
+ return -EINVAL;
+
+ if ( type == PSR_CBM_TYPE_L3_DATA )
+ val[0] = m;
+ else
+ val[1] = m;
+
+ ret= 2;
+ break;
+
+ /* CAT */
+ case PSR_CBM_TYPE_L3:
+ if ( !psr_check_cbm(feat->info.cbm_len, m) )
+ return -EINVAL;
+
+ val[0] = m;
+ break;
+
+ default:
+ if ( feat->feature == PSR_SOCKET_L3_CDP )
+ /* CDP uses two COS, one for data, one for code. */
+ ret = 2;
+ break;
+ }
+
+ return ret;
+}
+
+static int l3_cat_compare_val(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos, bool *found)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.cbm_len) - 1;
+
+ /* CDP */
+ if ( feat->feature == PSR_SOCKET_L3_CDP )
+ {
+ if ( cos > feat->info.cos_max )
+ {
+ if ( val[0] != l3_def_cbm ||
+ val[1] != l3_def_cbm )
+ {
+ *found = false;
+ return -ENOENT;
+ }
+ *found = true;
+ }
+ else
+ *found = (val[0] == get_cdp_data(feat, cos) &&
+ val[1] == get_cdp_code(feat, cos));
+
+ /* CDP uses two COS, one for data, one for code. */
+ return 2;
+ }
+
+ /* CAT */
+ if ( cos > feat->info.cos_max )
+ {
+ if ( val[0] != l3_def_cbm )
+ {
+ *found = false;
+ return -ENOENT;
+ }
+ *found = true;
+ }
+ else
+ *found = (val[0] == feat->cos_reg_val[cos]);
+
+ /* L3 CAT uses one COS. */
+ return 1;
+}
+
+static unsigned int l3_cat_get_cos_max_from_type(const struct feat_node *feat,
+ enum cbm_type type)
+{
+ if ( type != PSR_CBM_TYPE_L3_DATA &&
+ type != PSR_CBM_TYPE_L3_CODE &&
+ type != PSR_CBM_TYPE_L3 )
+ return 0;
+
+ return feat->info.cos_max;
+}
+
+static unsigned int l3_cat_exceeds_cos_max(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos)
+{
+ uint64_t l3_def_cbm;
+
+ l3_def_cbm = (1ull << feat->info.cbm_len) - 1;
+
+ /* CDP */
+ if ( feat->feature == PSR_SOCKET_L3_CDP )
+ {
+ if ( cos > feat->info.cos_max &&
+ (val[0] != l3_def_cbm || val[1] != l3_def_cbm) )
+ /*
+ * Exceed cos_max and value to set is not default,
+ * return error.
+ */
+ return 0;
+
+ /* CDP uses two COS, one for data, one for code. */
+ return 2;
+ }
+
+ /* CAT */
+ if ( cos > feat->info.cos_max &&
+ val[0] != l3_def_cbm )
+ /*
+ * Exceed cos_max and value to set is not default,
+ * return error.
+ */
+ return 0;
+
+ /* L3 CAT uses one COS. */
+ return 1;
+}
+
+static int l3_cat_write_msr(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat)
+{
+ /* CDP */
+ if ( feat->feature == PSR_SOCKET_L3_CDP )
+ {
+ /*
+ * If input cos is more than the cos_max of the feature, we should
+ * not set the value.
+ */
+ if ( cos > feat->info.cos_max )
+ /* CDP uses two COS, one for data, one for code. */
+ return 2;
+
+ /* Data */
+ if ( get_cdp_data(feat, cos) != val[0] )
+ {
+ get_cdp_data(feat, cos) = val[0];
+ wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(cos), val[0]);
+ }
+ /* Code */
+ if ( get_cdp_code(feat, cos) != val[1] )
+ {
+ get_cdp_code(feat, cos) = val[1];
+ wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(cos), val[1]);
+ }
+ /* CDP uses two COS, one for data, one for code. */
+ return 2;
+ }
+
+ /* CAT */
+ if ( cos > feat->info.cos_max )
+ /* L3 CAT uses one COS. */
+ return 1;
+
+ if ( feat->cos_reg_val[cos] != val[0] )
+ {
+ feat->cos_reg_val[cos] = val[0];
+ wrmsrl(MSR_IA32_PSR_L3_MASK(cos), val[0]);
+ }
+ /* L3 CAT uses one COS. */
+ return 1;
+}
+
struct feat_ops l3_cat_ops = {
.init_feature = l3_cat_init_feature,
.get_feat_info = l3_cat_get_feat_info,
.get_val = l3_cat_get_val,
.get_max_cos_max = l3_cat_get_max_cos_max,
+ .get_cos_num = l3_cat_get_cos_num,
+ .get_old_val = l3_cat_get_old_val,
+ .set_new_val = l3_cat_set_new_val,
+ .compare_val = l3_cat_compare_val,
+ .get_cos_max_from_type = l3_cat_get_cos_max_from_type,
+ .exceeds_cos_max = l3_cat_exceeds_cos_max,
+ .write_msr = l3_cat_write_msr,
};
static unsigned int get_socket_cpu(unsigned int socket)
@@ -516,11 +831,6 @@ static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
return cat_socket_info + socket;
}
-static inline bool_t cdp_is_enabled(unsigned int socket)
-{
- return test_bit(PSR_SOCKET_L3_CDP, &cat_socket_info[socket].feat_mask);
-}
-
int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t dat[], uint32_t array_len)
{
@@ -562,194 +872,314 @@ int psr_get_val(struct domain *d, unsigned int socket,
return 0;
}
-static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
-{
- unsigned int first_bit, zero_bit;
-
- /* Set bits should only in the range of [0, cbm_len). */
- if ( cbm & (~0ull << cbm_len) )
- return 0;
-
- /* At least one bit need to be set. */
- if ( cbm == 0 )
- return 0;
-
- first_bit = find_first_bit(&cbm, cbm_len);
- zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
-
- /* Set bits should be contiguous. */
- if ( zero_bit < cbm_len &&
- find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
- return 0;
-
- return 1;
-}
-
-struct cos_cbm_info
+struct cos_write_info
{
unsigned int cos;
- bool_t cdp;
- uint64_t cbm_code;
- uint64_t cbm_data;
+ struct list_head *feat_list;
+ const uint64_t *val;
};
-static void do_write_l3_cbm(void *data)
+static void do_write_psr_msr(void *data)
{
- struct cos_cbm_info *info = data;
+ struct cos_write_info *info = (struct cos_write_info *)data;
+ unsigned int cos = info->cos;
+ struct list_head *feat_list= info->feat_list;
+ const uint64_t *val = info->val;
+ struct feat_node *feat_tmp;
+ int ret;
+
+ if ( !feat_list )
+ return;
- if ( info->cdp )
+ list_for_each_entry(feat_tmp, feat_list, list)
{
- wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(info->cos), info->cbm_code);
- wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(info->cos), info->cbm_data);
+ ret = feat_tmp->ops.write_msr(cos, val, feat_tmp);
+ if ( ret <= 0)
+ return;
+
+ val += ret;
}
- else
- wrmsrl(MSR_IA32_PSR_L3_MASK(info->cos), info->cbm_code);
}
-static int write_l3_cbm(unsigned int socket, unsigned int cos,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp)
+static int write_psr_msr(unsigned int socket, unsigned int cos,
+ const uint64_t *val)
{
- struct cos_cbm_info info =
+ struct psr_cat_socket_info *info = get_cat_socket_info(socket);
+
+ struct cos_write_info data =
{
.cos = cos,
- .cbm_code = cbm_code,
- .cbm_data = cbm_data,
- .cdp = cdp,
+ .feat_list = &info->feat_list,
+ .val = val,
};
if ( socket == cpu_to_socket(smp_processor_id()) )
- do_write_l3_cbm(&info);
+ do_write_psr_msr(&data);
else
{
unsigned int cpu = get_socket_cpu(socket);
if ( cpu >= nr_cpu_ids )
return -ENOTSOCK;
- on_selected_cpus(cpumask_of(cpu), do_write_l3_cbm, &info, 1);
+ on_selected_cpus(cpumask_of(cpu), do_write_psr_msr, &data, 1);
}
return 0;
}
-static int find_cos(struct feat_node *feat, unsigned int *ref,
- unsigned int cos_max,
- uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
+static unsigned int get_cos_num(const struct psr_cat_socket_info *info)
+{
+ const struct feat_node *feat_tmp;
+ unsigned int num = 0;
+
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ num += feat_tmp->ops.get_cos_num(feat_tmp);
+
+ return num;
+}
+
+static int get_old_set_new(uint64_t *val,
+ uint32_t array_len,
+ const struct psr_cat_socket_info *info,
+ unsigned int old_cos,
+ enum cbm_type type,
+ uint64_t m)
+{
+ const struct feat_node *feat_tmp;
+ int ret;
+ uint64_t *val_tmp = val;
+
+ if ( !val )
+ return -EINVAL;
+
+ /* Get all features current values according to old_cos. */
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ {
+ /* value getting order is same as feature list */
+ ret = feat_tmp->ops.get_old_val(val_tmp, feat_tmp, old_cos);
+
+ val_tmp += ret;
+ if ( val_tmp - val > array_len)
+ return -EINVAL;
+ }
+
+ /* Set new value into array according to feature's position in array. */
+ val_tmp = val;
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ {
+ /* value setting order is same as feature list */
+ ret = feat_tmp->ops.set_new_val(val_tmp, feat_tmp, old_cos, type, m);
+ if ( ret < 0 )
+ return ret;
+
+ val_tmp += ret;
+ if ( val_tmp - val > array_len)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int find_cos(const uint64_t *val, uint32_t array_len,
+ enum cbm_type type,
+ const struct psr_cat_socket_info *info)
{
unsigned int cos;
+ const unsigned int *ref = info->cos_ref;
+ const struct feat_node *feat_tmp;
+ const uint64_t *val_tmp = val;
+ int ret;
+ bool found = false;
+ unsigned int cos_max = 0;
+
+ /* cos_max is the one of the feature which is being set. */
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ {
+ cos_max = feat_tmp->ops.get_cos_max_from_type(feat_tmp, type);
+ if ( cos_max > 0 )
+ break;
+ }
for ( cos = 0; cos <= cos_max; cos++ )
{
- if ( (ref[cos] || cos == 0) &&
- ((!cdp_enabled && feat->cos_reg_val[cos] == cbm_code) ||
- (cdp_enabled && get_cdp_code(feat, cos) == cbm_code &&
- get_cdp_data(feat, cos) == cbm_data)) )
+ if ( cos && !ref[cos] )
+ continue;
+
+ /* Not found, need find again from beginning. */
+ val_tmp = val;
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ {
+ /*
+ * Compare value according to feature list order.
+ * We must follow this order because value array is assembled
+ * as this order in get_old_set_new().
+ */
+ ret = feat_tmp->ops.compare_val(val_tmp, feat_tmp, cos, &found);
+ if ( ret < 0 )
+ return ret;
+
+ /* If fail to match, go to next cos to compare. */
+ if ( !found )
+ break;
+
+ val_tmp += ret;
+ if ( val_tmp - val > array_len )
+ return -EINVAL;
+ }
+
+ /*
+ * With this cos id, every entry of value array can match. This cos
+ * is what we find.
+ */
+ if ( found )
return cos;
}
return -ENOENT;
}
-static int pick_avail_cos(unsigned int *ref, unsigned int cos_max,
- unsigned int old_cos)
+static bool exceeds_cos_max(const uint64_t *val,
+ uint32_t array_len,
+ const struct psr_cat_socket_info *info,
+ unsigned int cos)
+{
+ unsigned int ret;
+ const uint64_t *val_tmp = val;
+ const struct feat_node *feat_tmp;
+
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ {
+ ret = feat_tmp->ops.exceeds_cos_max(val_tmp, feat_tmp, cos);
+ if ( !ret )
+ return false;
+
+ val_tmp += ret;
+ if ( val_tmp - val > array_len )
+ return false;
+ }
+
+ return true;
+}
+
+static int alloc_new_cos(const struct psr_cat_socket_info *info,
+ const uint64_t *val, uint32_t array_len,
+ unsigned int old_cos,
+ enum cbm_type type)
{
unsigned int cos;
+ unsigned int cos_max = 0;
+ const struct feat_node *feat_tmp;
+ const unsigned int *ref = info->cos_ref;
+
+ /*
+ * cos_max is the one of the feature which is being set.
+ */
+ list_for_each_entry(feat_tmp, &info->feat_list, list)
+ {
+ cos_max = feat_tmp->ops.get_cos_max_from_type(feat_tmp, type);
+ if ( cos_max > 0 )
+ break;
+ }
+
+ if ( !cos_max )
+ return -ENOENT;
/* If old cos is referred only by the domain, then use it. */
- if ( ref[old_cos] == 1 && old_cos != 0 )
- return old_cos;
+ if ( ref[old_cos] == 1 && old_cos )
+ if ( exceeds_cos_max(val, array_len, info, old_cos) )
+ return old_cos;
/* Find an unused one other than cos0. */
for ( cos = 1; cos <= cos_max; cos++ )
- if ( ref[cos] == 0 )
+ /*
+ * ref is 0 means this COS is not used by other domain and
+ * can be used for current setting.
+ */
+ if ( !ref[cos] )
+ {
+ if ( !exceeds_cos_max(val, array_len, info, cos) )
+ return -ENOENT;
+
return cos;
+ }
return -ENOENT;
}
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type)
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t val, enum cbm_type type)
{
- unsigned int old_cos, cos_max;
+ unsigned int old_cos;
int cos, ret;
- uint64_t cbm_data, cbm_code;
- bool_t cdp_enabled = cdp_is_enabled(socket);
- struct psr_cat_socket_info *info = get_cat_socket_info(socket);
unsigned int *ref;
- struct feat_node *feat_tmp;
+ uint64_t *val_array;
+ struct psr_cat_socket_info *info = get_cat_socket_info(socket);
+ uint32_t array_len;
if ( IS_ERR(info) )
return PTR_ERR(info);
- feat_tmp = get_feat_l3(info);
- if ( !feat_tmp )
- return -ENOENT;
-
- if ( !psr_check_cbm(feat_tmp->info.cbm_len, cbm) )
- return -EINVAL;
-
- if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
- type == PSR_CBM_TYPE_L3_DATA) )
- return -ENXIO;
-
- cos_max = feat_tmp->info.cos_max;
old_cos = d->arch.psr_cos_ids[socket];
ref = info->cos_ref;
- switch ( type )
- {
- case PSR_CBM_TYPE_L3:
- cbm_code = cbm;
- cbm_data = cbm;
- break;
-
- case PSR_CBM_TYPE_L3_CODE:
- cbm_code = cbm;
- cbm_data = get_cdp_data(feat_tmp, old_cos);
- break;
-
- case PSR_CBM_TYPE_L3_DATA:
- cbm_code = get_cdp_code(feat_tmp, old_cos);
- cbm_data = cbm;
- break;
+ /*
+ * Assemle a value array to store all featues cos_reg_val[old_cos].
+ * And, set the input val into array according to the feature's
+ * position in array.
+ */
+ array_len = get_cos_num((const struct psr_cat_socket_info *)info);
+ val_array = xzalloc_array(uint64_t, array_len);
+ if ( !val_array )
+ return -ENOMEM;
- default:
- ASSERT_UNREACHABLE();
- return -EINVAL;
+ if ( (ret = get_old_set_new(val_array, array_len,
+ (const struct psr_cat_socket_info *)info,
+ old_cos, type, val)) != 0 )
+ {
+ xfree(val_array);
+ return ret;
}
+ /*
+ * Lock here to make sure the ref is not changed during find and
+ * write process.
+ */
spin_lock(&info->ref_lock);
- cos = find_cos(feat_tmp, ref, cos_max, cbm_code, cbm_data, cdp_enabled);
+ /*
+ * Try to find if there is already a COS ID on which all features' values
+ * are same as the array. Then, we can reuse this COS ID.
+ */
+ cos = find_cos((const uint64_t *)val_array, array_len, type,
+ (const struct psr_cat_socket_info *)info);
if ( cos >= 0 )
{
if ( cos == old_cos )
{
spin_unlock(&info->ref_lock);
+ xfree(val_array);
return 0;
}
}
else
{
- cos = pick_avail_cos(ref, cos_max, old_cos);
+ /* If fail to find, we need allocate a new COS ID. */
+ cos = alloc_new_cos((const struct psr_cat_socket_info *)info,
+ (const uint64_t *)val_array, array_len,
+ old_cos, type);
if ( cos < 0 )
{
spin_unlock(&info->ref_lock);
+ xfree(val_array);
return cos;
}
- /* We try to avoid writing MSR. */
- if ( (cdp_enabled &&
- (get_cdp_code(feat_tmp, cos) != cbm_code ||
- get_cdp_data(feat_tmp, cos) != cbm_data)) ||
- (!cdp_enabled && feat_tmp->cos_reg_val[cos] != cbm_code) )
+ /* Write all features MSRs according to the COS ID. */
+ ret = write_psr_msr(socket, cos, (const uint64_t *)val_array);
+ if ( ret )
{
- ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
- if ( ret )
- {
- spin_unlock(&info->ref_lock);
- return ret;
- }
- get_cdp_code(feat_tmp, cos) = cbm_code;
- get_cdp_data(feat_tmp, cos) = cbm_data;
+ spin_unlock(&info->ref_lock);
+ xfree(val_array);
+ return ret;
}
}
@@ -758,6 +1188,7 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
spin_unlock(&info->ref_lock);
d->arch.psr_cos_ids[socket] = cos;
+ xfree(val_array);
return 0;
}
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index f2872a2..17ee6f3 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -67,8 +67,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
uint32_t dat[], uint32_t array_len);
int psr_get_val(struct domain *d, unsigned int socket,
uint64_t *val, enum cbm_type type);
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
- uint64_t cbm, enum cbm_type type);
+int psr_set_val(struct domain *d, unsigned int socket,
+ uint64_t val, enum cbm_type type);
int psr_domain_init(struct domain *d);
void psr_domain_free(struct domain *d);
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v3 12/15] x86: Implement L2 CAT in psr.c.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (10 preceding siblings ...)
2016-10-25 3:40 ` [PATCH v3 11/15] x86: refactor psr: Implement set value callback function Yi Sun
@ 2016-10-25 3:41 ` Yi Sun
2016-10-25 3:41 ` [PATCH v3 13/15] x86: Add L2 CAT interfaces in domctl Yi Sun
` (3 subsequent siblings)
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:41 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
Enable L2 CAT (Cache Allocation Technology) feature support in
psr.c.
- Implement 'struct feat_ops' callback functions for L2 CAT.
- Initialize L2 CAT feature and add it into feature list to
enable L2 CAT in psr.c.
- Free resources when CPU dead or cancelled.
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/psr.c | 230 +++++++++++++++++++++++++++++++++++++++-
xen/include/asm-x86/msr-index.h | 1 +
xen/include/asm-x86/psr.h | 2 +
3 files changed, 229 insertions(+), 4 deletions(-)
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 32f899a..c6f57c0 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -31,9 +31,13 @@
#define COS_MAX 1
#define CDP_FLAG 2
+#define CAT_CBM_LEN_MASK 0x1f
+#define CAT_COS_MAX_MASK 0xffff
+
enum psr_feat_type {
PSR_SOCKET_L3_CAT = 0,
PSR_SOCKET_L3_CDP,
+ PSR_SOCKET_L2_CAT,
};
struct feat_node;
@@ -161,7 +165,10 @@ struct feat_node {
};
struct psr_cat_socket_info {
- /* bit 1~0: [01]->L3 CAT-only, [10]->L3 CDP */
+ /*
+ * bit 1~0: [01]->L3 CAT-only, [10]->L3 CDP
+ * bit 2: L2 CAT
+ */
unsigned int feat_mask;
unsigned int nr_feat;
struct list_head feat_list;
@@ -204,8 +211,9 @@ static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_NUM - 1;
static uint64_t rmid_mask;
static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
-/* Feature list entry of feature L3 CAT/CDP. */
+/* Declare feature list entry. */
static struct feat_node *feat_l3;
+static struct feat_node *feat_l2;
/* Common functions. */
static void free_feature(struct psr_cat_socket_info *info)
@@ -228,6 +236,12 @@ static void free_feature(struct psr_cat_socket_info *info)
xfree(feat_l3);
feat_l3 = NULL;
}
+
+ if ( feat_l2 )
+ {
+ xfree(feat_l2);
+ feat_l2 = NULL;
+ }
}
static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
@@ -253,6 +267,194 @@ static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
return 1;
}
+/* L2 CAT callback functions implementation. */
+static void l2_cat_init_feature(unsigned int eax, unsigned int ebx,
+ unsigned int ecx, unsigned int edx,
+ struct feat_node *feat,
+ struct psr_cat_socket_info *info)
+{
+ struct psr_cat_hw_info l2_cat;
+ unsigned int socket;
+
+ /* No valid values so do not enable the feature. */
+ if ( !eax || !edx )
+ return;
+
+ l2_cat.cbm_len = (eax & CAT_CBM_LEN_MASK) + 1;
+ l2_cat.cos_max = min(opt_cos_max, edx & CAT_COS_MAX_MASK);
+
+ /* cos=0 is reserved as default cbm(all ones). */
+ feat->cos_reg_val[0] = (1ull << l2_cat.cbm_len) - 1;
+
+ feat->feature = PSR_SOCKET_L2_CAT;
+ __set_bit(PSR_SOCKET_L2_CAT, &info->feat_mask);
+
+ feat->info = l2_cat;
+
+ info->nr_feat++;
+
+ /* Add this feature into list. */
+ list_add_tail(&feat->list, &info->feat_list);
+
+ socket = cpu_to_socket(smp_processor_id());
+ printk(XENLOG_INFO "L2 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u.\n",
+ socket, feat->info.cos_max, feat->info.cbm_len);
+}
+
+static bool l2_cat_get_feat_info(const struct feat_node *feat,
+ enum cbm_type type,
+ uint32_t dat[], uint32_t array_len)
+{
+ if ( type != PSR_CBM_TYPE_L2 || !dat || 2 > array_len )
+ return false;
+
+ dat[CBM_LEN] = feat->info.cbm_len;
+ dat[COS_MAX] = feat->info.cos_max;
+
+ return true;
+}
+
+static int l2_cat_get_val(const struct feat_node *feat, unsigned int cos,
+ enum cbm_type type, uint64_t *val)
+{
+ if ( type != PSR_CBM_TYPE_L2 )
+ return 0;
+
+ if ( cos > feat->info.cos_max )
+ cos = 0;
+
+ /* L2 CAT */
+ *val = feat->cos_reg_val[cos];
+
+ return 1;
+}
+
+static unsigned int l2_cat_get_max_cos_max(const struct feat_node *feat)
+{
+ return feat->info.cos_max;
+}
+
+static unsigned int l2_cat_get_cos_num(const struct feat_node *feat)
+{
+ /* L2 CAT uses one COS. */
+ return 1;
+}
+
+static int l2_cat_get_old_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos)
+{
+ if ( old_cos > feat->info.cos_max )
+ /* Use default value. */
+ old_cos = 0;
+
+ val[0] = feat->cos_reg_val[old_cos];
+
+ /* L2 CAT uses one COS. */
+ return 1;
+}
+
+static int l2_cat_set_new_val(uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int old_cos,
+ enum cbm_type type,
+ uint64_t m)
+{
+ if ( type == PSR_CBM_TYPE_L2 )
+ {
+ if ( !psr_check_cbm(feat->info.cbm_len, m) )
+ return -EINVAL;
+
+ val[0] = m;
+ }
+
+ /* L2 CAT uses one COS. */
+ return 1;
+}
+
+static int l2_cat_compare_val(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos, bool *found)
+{
+ uint64_t l2_def_cbm;
+
+ l2_def_cbm = (1ull << feat->info.cbm_len) - 1;
+
+ /* L2 CAT */
+ if ( cos > feat->info.cos_max )
+ {
+ if ( val[0] != l2_def_cbm )
+ {
+ *found = false;
+ return -ENOENT;
+ }
+ *found = true;
+ }
+ else
+ *found = (val[0] == feat->cos_reg_val[cos]);
+
+ /* L2 CAT uses one COS. */
+ return 1;
+}
+
+static unsigned int l2_cat_get_cos_max_from_type(const struct feat_node *feat,
+ enum cbm_type type)
+{
+ if ( type != PSR_CBM_TYPE_L2 )
+ return 0;
+
+ return feat->info.cos_max;
+}
+
+static unsigned int l2_cat_exceeds_cos_max(const uint64_t val[],
+ const struct feat_node *feat,
+ unsigned int cos)
+{
+ uint64_t l2_def_cbm;
+
+ l2_def_cbm = (1ull << feat->info.cbm_len) - 1;
+
+ /* L2 CAT */
+ if ( cos > feat->info.cos_max &&
+ val[0] != l2_def_cbm )
+ /*
+ * Exceed cos_max and value to set is not default,
+ * return error.
+ */
+ return 0;
+
+ /* L2 CAT uses one COS. */
+ return 1;
+}
+
+static int l2_cat_write_msr(unsigned int cos, const uint64_t val[],
+ struct feat_node *feat)
+{
+ /* L2 CAT */
+ if ( cos > feat->info.cos_max )
+ return 1;
+
+ feat->cos_reg_val[cos] = val[0];
+ wrmsrl(MSR_IA32_PSR_L2_MASK(cos), val[0]);
+
+ /* L2 CAT uses one COS. */
+ return 1;
+}
+
+struct feat_ops l2_cat_ops = {
+ .init_feature = l2_cat_init_feature,
+ .get_feat_info = l2_cat_get_feat_info,
+ .get_val = l2_cat_get_val,
+ .get_max_cos_max = l2_cat_get_max_cos_max,
+ .get_cos_num = l2_cat_get_cos_num,
+ .get_old_val = l2_cat_get_old_val,
+ .set_new_val = l2_cat_set_new_val,
+ .compare_val = l2_cat_compare_val,
+ .get_cos_max_from_type = l2_cat_get_cos_max_from_type,
+ .exceeds_cos_max = l2_cat_exceeds_cos_max,
+ .write_msr = l2_cat_write_msr,
+};
+
/* L3 CAT/CDP callback functions implementation. */
static void l3_cat_init_feature(unsigned int eax, unsigned int ebx,
unsigned int ecx, unsigned int edx,
@@ -267,8 +469,8 @@ static void l3_cat_init_feature(unsigned int eax, unsigned int ebx,
if ( !eax || !edx )
return;
- l3_cat.cbm_len = (eax & 0x1f) + 1;
- l3_cat.cos_max = min(opt_cos_max, edx & 0xffff);
+ l3_cat.cbm_len = (eax & CAT_CBM_LEN_MASK) + 1;
+ l3_cat.cos_max = min(opt_cos_max, edx & CAT_COS_MAX_MASK);
/* cos=0 is reserved as default cbm(all ones). */
feat->cos_reg_val[0] = (1ull << l3_cat.cbm_len) - 1;
@@ -1246,6 +1448,14 @@ static int cat_cpu_prepare(unsigned int cpu)
(feat_l3 = xzalloc(struct feat_node)) == NULL )
return -ENOMEM;
+ if ( feat_l2 == NULL &&
+ (feat_l2 = xzalloc(struct feat_node)) == NULL )
+ {
+ xfree(feat_l3);
+ feat_l3 = NULL;
+ return -ENOMEM;
+ }
+
return 0;
}
@@ -1278,6 +1488,18 @@ static void cat_cpu_init(void)
feat_tmp->ops = l3_cat_ops;
feat_tmp->ops.init_feature(eax, ebx, ecx, edx, feat_tmp, info);
}
+
+ cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
+ if ( ebx & PSR_RESOURCE_TYPE_L2 )
+ {
+ feat_tmp = feat_l2;
+ feat_l2 = NULL;
+
+ /* Initialize L2 CAT according to CPUID. */
+ cpuid_count(PSR_CPUID_LEVEL_CAT, 2, &eax, &ebx, &ecx, &edx);
+ feat_tmp->ops = l2_cat_ops;
+ feat_tmp->ops.init_feature(eax, ebx, ecx, edx, feat_tmp, info);
+ }
}
static void cat_cpu_fini(unsigned int cpu)
diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h
index 98dbff1..a41e63a 100644
--- a/xen/include/asm-x86/msr-index.h
+++ b/xen/include/asm-x86/msr-index.h
@@ -343,6 +343,7 @@
#define MSR_IA32_PSR_L3_MASK(n) (0x00000c90 + (n))
#define MSR_IA32_PSR_L3_MASK_CODE(n) (0x00000c90 + (n) * 2 + 1)
#define MSR_IA32_PSR_L3_MASK_DATA(n) (0x00000c90 + (n) * 2)
+#define MSR_IA32_PSR_L2_MASK(n) (0x00000d10 + (n))
/* Intel Model 6 */
#define MSR_P6_PERFCTR(n) (0x000000c1 + (n))
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 17ee6f3..37ad185 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -23,6 +23,7 @@
/* Resource Type Enumeration */
#define PSR_RESOURCE_TYPE_L3 0x2
+#define PSR_RESOURCE_TYPE_L2 0x4
/* L3 Monitoring Features */
#define PSR_CMT_L3_OCCUPANCY 0x1
@@ -50,6 +51,7 @@ enum cbm_type {
PSR_CBM_TYPE_L3,
PSR_CBM_TYPE_L3_CODE,
PSR_CBM_TYPE_L3_DATA,
+ PSR_CBM_TYPE_L2,
};
extern struct psr_cmt *psr_cmt;
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v3 13/15] x86: Add L2 CAT interfaces in domctl.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (11 preceding siblings ...)
2016-10-25 3:41 ` [PATCH v3 12/15] x86: Implement L2 CAT in psr.c Yi Sun
@ 2016-10-25 3:41 ` Yi Sun
2016-10-25 3:41 ` [PATCH v3 14/15] x86: Add L2 CAT interfaces in sysctl Yi Sun
` (2 subsequent siblings)
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:41 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
This patch adds L2 CAT interfaces to get/set CBM in domctl.
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/domctl.c | 13 +++++++++++++
xen/include/public/domctl.h | 2 ++
2 files changed, 15 insertions(+)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 8e5502a..81f05d7 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1424,6 +1424,19 @@ long arch_do_domctl(
copyback = 1;
break;
+ case XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM:
+ ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+ domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L2);
+ break;
+
+ case XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM:
+ ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+ &domctl->u.psr_cat_op.data,
+ PSR_CBM_TYPE_L2);
+ copyback = 1;
+ break;
+
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 177319d..17c4ab4 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1137,6 +1137,8 @@ struct xen_domctl_psr_cat_op {
#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4
#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM 6
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM 7
uint32_t cmd; /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
uint32_t target; /* IN */
uint64_t data; /* IN/OUT */
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v3 14/15] x86: Add L2 CAT interfaces in sysctl.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (12 preceding siblings ...)
2016-10-25 3:41 ` [PATCH v3 13/15] x86: Add L2 CAT interfaces in domctl Yi Sun
@ 2016-10-25 3:41 ` Yi Sun
2016-10-25 3:41 ` [PATCH v3 15/15] tools & docs: add L2 CAT support in tools and docs Yi Sun
2016-11-09 1:28 ` [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:41 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
This patch adds L2 CAT interface to get HW info in sysctl.
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
xen/arch/x86/sysctl.c | 13 +++++++++++++
xen/include/public/sysctl.h | 6 ++++++
2 files changed, 19 insertions(+)
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index e82adec..a66f049 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -192,6 +192,19 @@ long arch_do_sysctl(
ret = -EFAULT;
break;
}
+ case XEN_SYSCTL_PSR_CAT_get_l2_info:
+ {
+ uint32_t dat[2];
+ ret = psr_get_info(sysctl->u.psr_cat_op.target,
+ PSR_CBM_TYPE_L2,
+ dat, 2);
+ sysctl->u.psr_cat_op.u.l2_info.cbm_len = dat[CBM_LEN];
+ sysctl->u.psr_cat_op.u.l2_info.cos_max = dat[COS_MAX];
+
+ if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
+ ret = -EFAULT;
+ break;
+ }
default:
ret = -EOPNOTSUPP;
break;
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 418f4bb..0a0c9f7 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -745,6 +745,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
#define XEN_SYSCTL_PSR_CAT_get_l3_info 0
+#define XEN_SYSCTL_PSR_CAT_get_l2_info 1
struct xen_sysctl_psr_cat_op {
uint32_t cmd; /* IN: XEN_SYSCTL_PSR_CAT_* */
uint32_t target; /* IN */
@@ -755,6 +756,11 @@ struct xen_sysctl_psr_cat_op {
#define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0)
uint32_t flags; /* OUT: CAT flags */
} l3_info;
+
+ struct {
+ uint32_t cbm_len; /* OUT: CBM length */
+ uint32_t cos_max; /* OUT: Maximum COS */
+ } l2_info;
} u;
};
typedef struct xen_sysctl_psr_cat_op xen_sysctl_psr_cat_op_t;
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* [PATCH v3 15/15] tools & docs: add L2 CAT support in tools and docs.
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (13 preceding siblings ...)
2016-10-25 3:41 ` [PATCH v3 14/15] x86: Add L2 CAT interfaces in sysctl Yi Sun
@ 2016-10-25 3:41 ` Yi Sun
2016-11-09 1:28 ` [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
15 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-10-25 3:41 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
This patch is the xl/xc changes to support Intel L2 CAT
(Cache Allocation Technology).
The new level option is introduced to original CAT setting
command in order to set CBM for specified level CAT.
- 'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
info.
- 'xl psr-cat-cbm-set' is updated to set cache capacity
bitmasks(CBM) for a domain according to input cache level.
- 'xl psr-cat-show' is updated to show CBM of a domain
according to input cache level.
Examples:
root@:~$ xl psr-hwinfo --cat
Cache Allocation Technology (CAT): L2
Socket ID : 0
Maximum COS : 3
CBM length : 8
Default CBM : 0xff
root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
root@:~$ xl psr-cat-show -l2 1
Socket ID : 0
Default CBM : 0xff
ID NAME CBM
1 ubuntu14 0x7f
Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
docs/man/xl.pod.1.in | 25 ++++-
docs/misc/xl-psr.markdown | 10 +-
tools/libxc/include/xenctrl.h | 7 +-
tools/libxc/xc_psr.c | 46 ++++++---
tools/libxl/libxl.h | 11 ++-
tools/libxl/libxl_psr.c | 10 +-
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 220 +++++++++++++++++++++++++++++++++---------
tools/libxl/xl_cmdtable.c | 4 +-
9 files changed, 260 insertions(+), 74 deletions(-)
diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 8e2aa5b..2c41ea7 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -1701,6 +1701,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
a domain is attached to the monitoring service, monitoring data can be shown
for any of these monitoring types.
+There is no cache monitoring and memory bandwidth monitoring on L2 cache so
+far.
+
=over 4
=item B<psr-cmt-attach> [I<domain-id>]
@@ -1725,7 +1728,7 @@ monitor types are:
Intel Broadwell and later server platforms offer capabilities to configure and
make use of the Cache Allocation Technology (CAT) mechanisms, which enable more
-cache resources (i.e. L3 cache) to be made available for high priority
+cache resources (i.e. L3/L2 cache) to be made available for high priority
applications. In the Xen implementation, CAT is used to control cache allocation
on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
(CBM) for the domain.
@@ -1735,7 +1738,7 @@ Intel Broadwell and later server platforms also offer Code/Data Prioritization
applications. CDP is used on a per VM basis in the Xen implementation. To
specify code or data CBM for the domain, CDP feature must be enabled and CBM
type options need to be specified when setting CBM, and the type options (code
-and data) are mutually exclusive.
+and data) are mutually exclusive. There is no CDP support on L2 so far.
=over 4
@@ -1752,6 +1755,11 @@ B<OPTIONS>
Specify the socket to process, otherwise all sockets are processed.
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
=item B<-c>, B<--code>
Set code CBM when CDP is enabled.
@@ -1762,10 +1770,21 @@ Set data CBM when CDP is enabled.
=back
-=item B<psr-cat-show> [I<domain-id>]
+=item B<psr-cat-show> [I<OPTIONS>] [I<domain-id>]
Show CAT settings for a certain domain or all domains.
+B<OPTIONS>
+
+=over 4
+
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
+=back
+
=back
=head1 IGNORED FOR COMPATIBILITY WITH XM
diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
index c3c1e8e..bd2b6bd 100644
--- a/docs/misc/xl-psr.markdown
+++ b/docs/misc/xl-psr.markdown
@@ -70,7 +70,7 @@ total-mem-bandwidth instead of cache-occupancy). E.g. after a `xl psr-cmt-attach
Cache Allocation Technology (CAT) is a new feature available on Intel
Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
-partition cache allocation (i.e. L3 cache) based on application priority or
+partition cache allocation (i.e. L3/L2 cache) based on application priority or
Class of Service (COS). Each COS is configured using capacity bitmasks (CBM)
which represent cache capacity and indicate the degree of overlap and
isolation between classes. System cache resource is divided into numbers of
@@ -119,13 +119,19 @@ A cbm is valid only when:
In a multi-socket system, the same cbm will be set on each socket by default.
Per socket cbm can be specified with the `--socket SOCKET` option.
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
+
Setting the CBM may not be successful if insufficient COS is available. In
such case unused COS(es) may be freed by setting CBM of all related domains to
its default value(all-ones).
Per domain CBM settings can be shown by:
-`xl psr-cat-show`
+`xl psr-cat-show [OPTIONS] <domid>`
+
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
## Code and Data Prioritization (CDP)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 2c83544..8ee068f 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2602,6 +2602,7 @@ enum xc_psr_cat_type {
XC_PSR_CAT_L3_CBM = 1,
XC_PSR_CAT_L3_CBM_CODE = 2,
XC_PSR_CAT_L3_CBM_DATA = 3,
+ XC_PSR_CAT_L2_CBM = 4,
};
typedef enum xc_psr_cat_type xc_psr_cat_type;
@@ -2626,9 +2627,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
xc_psr_cat_type type, uint32_t target,
uint64_t *data);
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled);
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len,
+ bool *cdp_enabled);
int xc_get_cpu_levelling_caps(xc_interface *xch, uint32_t *caps);
int xc_get_cpu_featureset(xc_interface *xch, uint32_t index,
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 43b3286..d5055db 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -266,6 +266,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
@@ -299,6 +302,9 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
case XC_PSR_CAT_L3_CBM_DATA:
cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
break;
+ case XC_PSR_CAT_L2_CBM:
+ cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM;
+ break;
default:
errno = EINVAL;
return -1;
@@ -317,24 +323,40 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
return rc;
}
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
- uint32_t *cos_max, uint32_t *cbm_len,
- bool *cdp_enabled)
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, int lvl,
+ uint32_t *cos_max, uint32_t *cbm_len,
+ bool *cdp_enabled)
{
- int rc;
+ int rc = -1;
DECLARE_SYSCTL;
sysctl.cmd = XEN_SYSCTL_psr_cat_op;
- sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
sysctl.u.psr_cat_op.target = socket;
- rc = xc_sysctl(xch, &sysctl);
- if ( !rc )
- {
- *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
- *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
- *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
- XEN_SYSCTL_PSR_CAT_L3_CDP;
+ switch ( lvl ) {
+ case 2:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l2_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.l2_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.l2_info.cbm_len;
+ }
+ break;
+ case 3:
+ sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
+ rc = xc_sysctl(xch, &sysctl);
+ if ( !rc )
+ {
+ *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
+ *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
+ *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
+ XEN_SYSCTL_PSR_CAT_L3_CDP;
+ }
+ break;
+ default:
+ errno = EOPNOTSUPP;
+ return rc;
}
return rc;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index acbf476..026014a 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -907,6 +907,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, const libxl_mac *src);
#endif
/*
+ * LIBXL_HAVE_PSR_GEN
+ *
+ * If this is defined, the General Interfaces for PSR features are supported.
+ */
+#define LIBXL_HAVE_PSR_GEN 1
+
+/*
* LIBXL_HAVE_PCITOPOLOGY
*
* If this is defined, then interface to query hypervisor about PCI device
@@ -2159,8 +2166,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
* On success, the function returns an array of elements in 'info',
* and the length in 'nr'.
*/
-int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
- int *nr);
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl);
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr);
#endif
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index 786183c..ec2042b 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -352,8 +352,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
return rc;
}
-int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
- int *nr)
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+ int *nr, unsigned int lvl)
{
GC_INIT(ctx);
int rc;
@@ -380,8 +380,8 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
libxl_for_each_set_bit(socketid, socketmap) {
ptr[i].id = socketid;
- if (xc_psr_cat_get_l3_info(ctx->xch, socketid, &ptr[i].cos_max,
- &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
+ if (xc_psr_cat_get_info(ctx->xch, socketid, lvl, &ptr[i].cos_max,
+ &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
libxl__psr_cat_log_err_msg(gc, errno);
rc = ERROR_FAIL;
free(ptr);
@@ -400,7 +400,7 @@ out:
void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr)
{
- int i;
+ unsigned int i;
for (i = 0; i < nr; i++)
libxl_psr_cat_info_dispose(&list[i]);
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a32c751..944b7d4 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -898,6 +898,7 @@ libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
(1, "L3_CBM"),
(2, "L3_CBM_CODE"),
(3, "L3_CBM_DATA"),
+ (4, "L2_CBM"),
])
libxl_psr_cat_info = Struct("psr_cat_info", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index bb5afb8..ca3122d 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9332,18 +9332,128 @@ int main_psr_cmt_show(int argc, char **argv)
#endif
#ifdef LIBXL_HAVE_PSR_CAT
-static int psr_cat_hwinfo(void)
+static void psr_cat_print_one_domain_cbm_type(uint32_t domid, uint32_t socketid,
+ libxl_psr_cbm_type type)
+{
+ uint64_t cbm;
+
+ if (!libxl_psr_cat_get_cbm(ctx, domid, type, socketid, &cbm))
+ printf("%#16"PRIx64, cbm);
+ else
+ printf("%16s", "error");
+}
+
+static int psr_l2_cat_hwinfo(void)
{
int rc;
- int i, nr;
+ unsigned int i;
+ int nr;
+ libxl_psr_cat_info *info;
+
+ printf("Cache Allocation Technology (CAT): L2\n");
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 2);
+ if (rc) {
+ fprintf(stderr, "Failed to get l2 cat info\n");
+ return rc;
+ }
+
+ for (i = 0; i < nr; i++) {
+ /* There is no CMT on L2 cache so far. */
+ printf("%-16s: %u\n", "Socket ID", info[i].id);
+ printf("%-16s: %u\n", "Maximum COS", info[i].cos_max);
+ printf("%-16s: %u\n", "CBM length", info[i].cbm_len);
+ printf("%-16s: %#llx\n", "Default CBM",
+ (1ull << info[i].cbm_len) - 1);
+ }
+
+ libxl_psr_cat_info_list_free(info, nr);
+ return rc;
+}
+
+static void psr_l2_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid)
+{
+ char *domain_name;
+
+ domain_name = libxl_domid_to_name(ctx, domid);
+ printf("%5d%25s", domid, domain_name);
+ free(domain_name);
+
+ psr_cat_print_one_domain_cbm_type(domid, socketid,
+ LIBXL_PSR_CBM_TYPE_L2_CBM);
+
+ printf("\n");
+}
+
+static int psr_l2_cat_print_domain_cbm(uint32_t domid, uint32_t socketid)
+{
+ unsigned int i;
+ int nr_domains;
+ libxl_dominfo *list;
+
+ if (domid != INVALID_DOMID) {
+ psr_l2_cat_print_one_domain_cbm(domid, socketid);
+ return 0;
+ }
+
+ if (!(list = libxl_list_domain(ctx, &nr_domains))) {
+ fprintf(stderr, "Failed to get domain list for cbm display\n");
+ return ERROR_FAIL;
+ }
+
+ for (i = 0; i < nr_domains; i++)
+ psr_l2_cat_print_one_domain_cbm(list[i].domid, socketid);
+ libxl_dominfo_list_free(list, nr_domains);
+
+ return 0;
+}
+
+static int psr_l2_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
+{
+ printf("%-16s: %u\n", "Socket ID", info->id);
+ printf("%-16s: %#llx\n", "Default CBM", (1ull << info->cbm_len) - 1);
+ printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
+
+ return psr_l2_cat_print_domain_cbm(domid, info->id);
+}
+
+static int psr_l2_cat_show(uint32_t domid)
+{
+ unsigned int i;
+ int nr;
+ int rc;
+ libxl_psr_cat_info *info;
+
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 2);
+ if (rc) {
+ fprintf(stderr, "Failed to get l2 cat info\n");
+ return rc;
+ }
+
+ for (i = 0; i < nr; i++) {
+ rc = psr_l2_cat_print_socket(domid, info + i);
+ if (rc)
+ goto out;
+ }
+
+out:
+ libxl_psr_cat_info_list_free(info, nr);
+ return rc;
+}
+
+static int psr_l3_cat_hwinfo(void)
+{
+ int rc;
+ unsigned int i;
+ int nr;
uint32_t l3_cache_size;
libxl_psr_cat_info *info;
- printf("Cache Allocation Technology (CAT):\n");
+ printf("Cache Allocation Technology (CAT): L3\n");
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ fprintf(stderr, "Failed to get l3 cat info\n");
return rc;
}
@@ -9369,19 +9479,8 @@ out:
return rc;
}
-static void psr_cat_print_one_domain_cbm_type(uint32_t domid, uint32_t socketid,
- libxl_psr_cbm_type type)
-{
- uint64_t cbm;
-
- if (!libxl_psr_cat_get_cbm(ctx, domid, type, socketid, &cbm))
- printf("%#16"PRIx64, cbm);
- else
- printf("%16s", "error");
-}
-
-static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+static void psr_l3_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
+ bool cdp_enabled)
{
char *domain_name;
@@ -9402,14 +9501,15 @@ static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
printf("\n");
}
-static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
- bool cdp_enabled)
+static int psr_l3_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
+ bool cdp_enabled)
{
- int i, nr_domains;
+ unsigned int i;
+ int nr_domains;
libxl_dominfo *list;
if (domid != INVALID_DOMID) {
- psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
+ psr_l3_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
return 0;
}
@@ -9419,13 +9519,13 @@ static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
}
for (i = 0; i < nr_domains; i++)
- psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
+ psr_l3_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
libxl_dominfo_list_free(list, nr_domains);
return 0;
}
-static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
+static int psr_l3_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
{
int rc;
uint32_t l3_cache_size;
@@ -9445,23 +9545,24 @@ static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
else
printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
- return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled);
+ return psr_l3_cat_print_domain_cbm(domid, info->id, info->cdp_enabled);
}
-static int psr_cat_show(uint32_t domid)
+static int psr_l3_cat_show(uint32_t domid)
{
- int i, nr;
+ unsigned int i;
+ int nr;
int rc;
libxl_psr_cat_info *info;
- rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
+ rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
if (rc) {
- fprintf(stderr, "Failed to get cat info\n");
+ fprintf(stderr, "Failed to get l3 cat info\n");
return rc;
}
for (i = 0; i < nr; i++) {
- rc = psr_cat_print_socket(domid, info + i);
+ rc = psr_l3_cat_print_socket(domid, info + i);
if (rc)
goto out;
}
@@ -9482,19 +9583,21 @@ int main_psr_cat_cbm_set(int argc, char **argv)
char *value;
libxl_string_list socket_list;
unsigned long start, end;
- int i, j, len;
+ unsigned int i, j, len;
+ unsigned int lvl = 3;
static struct option opts[] = {
{"socket", 1, 0, 's'},
{"data", 0, 0, 'd'},
{"code", 0, 0, 'c'},
+ {"level", 1, 0, 'l'},
COMMON_LONG_OPTS
};
libxl_socket_bitmap_alloc(ctx, &target_map, 0);
libxl_bitmap_set_none(&target_map);
- SWITCH_FOREACH_OPT(opt, "s:cd", opts, "psr-cat-cbm-set", 2) {
+ SWITCH_FOREACH_OPT(opt, "s:l:cd", opts, "psr-cat-cbm-set", 2) {
case 's':
trim(isspace, optarg, &value);
split_string_into_string_list(value, ",", &socket_list);
@@ -9514,17 +9617,24 @@ int main_psr_cat_cbm_set(int argc, char **argv)
case 'c':
opt_code = 1;
break;
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
- if (opt_data && opt_code) {
- fprintf(stderr, "Cannot handle -c and -d at the same time\n");
- return -1;
- } else if (opt_data) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
- } else if (opt_code) {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
- } else {
- type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ if (lvl == 2)
+ type = LIBXL_PSR_CBM_TYPE_L2_CBM;
+ else if (lvl == 3) {
+ if (opt_data && opt_code) {
+ fprintf(stderr, "Cannot handle -c and -d at the same time\n");
+ return ERROR_FAIL;
+ } else if (opt_data) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
+ } else if (opt_code) {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+ } else {
+ type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+ }
}
if (libxl_bitmap_is_empty(&target_map))
@@ -9546,11 +9656,20 @@ int main_psr_cat_cbm_set(int argc, char **argv)
int main_psr_cat_show(int argc, char **argv)
{
- int opt;
+ int opt = 0;
uint32_t domid;
+ unsigned int lvl = 3;
+ int rc;
- SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cat-show", 0) {
- /* No options */
+ static struct option opts[] = {
+ {"level", 1, 0, 'l'},
+ COMMON_LONG_OPTS
+ };
+
+ SWITCH_FOREACH_OPT(opt, "l:", opts, "psr-cat-show", 0) {
+ case 'l':
+ lvl = atoi(optarg);
+ break;
}
if (optind >= argc)
@@ -9562,7 +9681,12 @@ int main_psr_cat_show(int argc, char **argv)
return 2;
}
- return psr_cat_show(domid);
+ if (lvl == 3)
+ rc = psr_l3_cat_show(domid);
+ else if (lvl == 2)
+ rc = psr_l2_cat_show(domid);
+
+ return rc;
}
int main_psr_hwinfo(int argc, char **argv)
@@ -9588,7 +9712,11 @@ int main_psr_hwinfo(int argc, char **argv)
ret = psr_cmt_hwinfo();
if (!ret && (all || cat))
- ret = psr_cat_hwinfo();
+ ret = psr_l3_cat_hwinfo();
+
+ /* L2 CAT is independent of CMT and L3 CAT */
+ if (all || cat)
+ ret = psr_l2_cat_hwinfo();
return ret;
}
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index 588d5d9..32c3ee5 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -550,13 +550,15 @@ struct cmd_spec cmd_table[] = {
"Set cache capacity bitmasks(CBM) for a domain",
"[options] <Domain> <CBM>",
"-s <socket> Specify the socket to process, otherwise all sockets are processed\n"
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
"-c Set code CBM if CDP is supported\n"
"-d Set data CBM if CDP is supported\n"
},
{ "psr-cat-show",
&main_psr_cat_show, 0, 1,
"Show Cache Allocation Technology information",
- "<Domain>",
+ "[options] <Domain>",
+ "-l <level> Specify the cache level to process, otherwise L3 cache is processed\n"
},
#endif
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 42+ messages in thread* Re: [PATCH v3 00/15] Enable L2 Cache Allocation Technology
2016-10-25 3:40 [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
` (14 preceding siblings ...)
2016-10-25 3:41 ` [PATCH v3 15/15] tools & docs: add L2 CAT support in tools and docs Yi Sun
@ 2016-11-09 1:28 ` Yi Sun
2016-11-09 8:37 ` Jan Beulich
15 siblings, 1 reply; 42+ messages in thread
From: Yi Sun @ 2016-11-09 1:28 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich, chao.p.peng
Hi, all,
Any comment or suggestion to this patch set? That would be very appreciated.
Thank you!
BRs,
Sun Yi
On 16-10-25 11:40:48, Yi Sun wrote:
> Hi all,
>
> We plan to bring a new PSR (Platform Shared Resource) feature called
> Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
>
> Besides the L2 CAT implementaion, we refactor the psr.c to make it more
> flexible to add new features and fulfill the principle, open for extension
> but closed for modification. We abstract the general operations of all
> features and encapsulate them into a structure. Then, the development
> of new feature is simple to mainly implement these callback functions.
>
> The patch set can be found at:
> https://github.com/yisun-git/xen_l2_cat.git xen_l2_v3_3
>
> ---
> Changes since v2:
> * Add design document into docs/.
> * Split "refactor psr" patch to several small patches.
> * Function and variables names are changed to express accurately.
> * Revert some unnecessary changes.
> * Fix issues according to comments provided by reviewers.
>
> --
> 2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread* Re: [PATCH v3 00/15] Enable L2 Cache Allocation Technology
2016-11-09 1:28 ` [PATCH v3 00/15] Enable L2 Cache Allocation Technology Yi Sun
@ 2016-11-09 8:37 ` Jan Beulich
2016-11-10 1:56 ` Yi Sun
0 siblings, 1 reply; 42+ messages in thread
From: Jan Beulich @ 2016-11-09 8:37 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, chao.p.peng, xen-devel
>>> On 09.11.16 at 02:28, <yi.y.sun@linux.intel.com> wrote:
> Any comment or suggestion to this patch set? That would be very appreciated.
Please be assured that this has not been forgotten, but there are
more important things to deal with, so it may take some more time
to get to this. That's both because this clearly is for 4.9 only, and
because it affecting Atoms only for now its relatively low priority
only anyway (to date Atoms aren't a primary target for Xen afaict).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH v3 00/15] Enable L2 Cache Allocation Technology
2016-11-09 8:37 ` Jan Beulich
@ 2016-11-10 1:56 ` Yi Sun
0 siblings, 0 replies; 42+ messages in thread
From: Yi Sun @ 2016-11-10 1:56 UTC (permalink / raw)
To: Jan Beulich
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, xen-devel, chao.p.peng
On 16-11-09 01:37:55, Jan Beulich wrote:
> >>> On 09.11.16 at 02:28, <yi.y.sun@linux.intel.com> wrote:
> > Any comment or suggestion to this patch set? That would be very appreciated.
>
> Please be assured that this has not been forgotten, but there are
> more important things to deal with, so it may take some more time
> to get to this. That's both because this clearly is for 4.9 only, and
> because it affecting Atoms only for now its relatively low priority
> only anyway (to date Atoms aren't a primary target for Xen afaict).
>
> Jan
>
Thank you! This patch set refactors the psr.c. It is the base of other
new features implementation. So, it would be better to get them merged
ASAP. Of course, I can understand you are busy working on more important
things. Please review the patches once you are free. Thanks!
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 42+ messages in thread