All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c
@ 2017-01-19  6:01 Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document Yi Sun
                   ` (23 more replies)
  0 siblings, 24 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

Hi all,

We plan to bring a new PSR (Platform Shared Resource) feature called
Intel L2 Cache Allocation Technology (L2 CAT) to Xen.

Besides the L2 CAT implementaion, we refactor the psr.c to make it more
flexible to add new features and fulfill the principle, open for extension
but closed for modification. We abstract the general operations of all
features and encapsulate them into a structure. Then, the development
of new feature is simple to mainly implement these callback functions.

The patch set can be found at:
https://github.com/yisun-git/xen_l2_cat_v5.git l2_cat_v5

v5:
- patch 3: address Jan's comments.
    - explain CDP more in commit message.
    - remove exact SDM chapter number but only keep title.
    - remove init_feature from callback function ops structure.
- patch 4: address Jan's comments.
    - modify commit message beacuse of code changes.
    - add 'struct cpuid_leaf_regs' to save cpu registers value to reduce
      parameters of init_feature function.
    - modify comments to make them accurate.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
    - use 'list_for_each_entry_safe' when free features.
    - do not delete 'feat_l3_cat' to make it can be reused when cpu online.
    - use 'current_cpu_data'.
    - clear 'X86_FEATURE_PQE' if cpuid_level is not right.
    - Print socket info when 'opt_cpu_info' is true.
    - remove 'cpu_prepare_work' function and move contents of it into
      'psr_cpu_prepare'.
- patch 5: address Jan's comments.
    - define macro 'PSR_ASSOC_REG_POS' to replace integer 32.
    - rename 'l3_cat_get_max_cos_max' to 'l3_cat_get_cos_max'.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
- patch 6: address Jan's comments.
    - add function 'psr_cbm_type_to_feat_type' to covert 'cbm_type' to
      'psr_feat_type'. This is part of codes to move type check out from
      callback functions.
    - remove type check from feature callback functions.
    - rename 'dat[]' to 'data[]'
    - check if feature type match in caller of feature callback function.
- patch 7: address Jan's comments.
    - rename 'dat[]' to 'data[]'
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
    - check if feature type match in caller of feature callback function.
- patch 8: address Jan's comments.
    - modify commit message because of function name change.
    - change 'alloc_new_cos' to 'pick_avail_cos' to make name accurate.
    - divide 'get_old_set_new' to two functions, 'assemble_val_array' and
      'set_new_val_to_array'.
    - check feature type when entering 'psr_set_val'.
    - remove cast.
    - use ASSERT to check ref.
    - rename 'dat[]' to 'data[]'
- patch 9: address Jan's comments.
    - modify comments according to changes of codes.
    - change 'bool_t' to 'bool'.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
- patch 10: address Jan's comments.
    - modify commit message to provide exact patch name to continue from.
    - remove 'get_cos_max_from_type' because it can be replaced by
      'get_cos_max'.
    - move type check out from callback functions to caller.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
    - modify comments according to changes of codes.
- patch 11: address Jan's comments.
    - modify commit message to provide exact patch name to continue from.
    - change 'exceeds_cos_max' to 'fits_cos_max' to be accurate.
    - modify comments according to changes of codes.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - move type check out from callback functions to caller.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
- patch 12: address Jan's comments.
    - modify commit message to provide exact patch name to continue from.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - move type check out from callback functions to caller.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
- patch 13: address Jan's comments.
    - remove 'feat_l3_cdp' free in 'free_feature'.
    - Encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
    - Print socket info when 'opt_cpu_info' is true.
    - rename 'l3_cdp_get_max_cos_max' to 'l3_cdp_get_cos_max'.
    - rename 'dat[]' to 'data[]'
    - move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
- patch 14: address Jan's comments.
    - rename 'dat[]' to 'data[]'
    - remove type check in callback function.
- patch 15: address Jan's comments.
    - remove type check in callback function.
- patch 16: address Jan's comments.
    - remove type check in callback function.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - remove 'l3_cdp_get_cos_max_from_type'.
    - rename 'l3_cdp_exceeds_cos_max' to 'l3_cdp_fits_cos_max'.
- patch 17: address Jan's comments.
    - remove 'feat_l2_cat' free in 'free_feature'.
    - Encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
    - Print socket info when 'opt_cpu_info' is true.
    - rename 'l2_cat_get_max_cos_max' to 'l2_cat_get_cos_max'.
    - rename 'dat[]' to 'data[]'
    - move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
- patch 18: address Jan's comments.
    - rename 'dat[]' to 'data[]'
    - remove type check in callback function.
- patch 19: address Jan's comments.
    - remove type check in callback function.
- patch 20: address Jan's comments.
    - remove type check in callback function.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - remove 'l2_cat_get_cos_max_from_type'.
    - rename 'l2_cat_exceeds_cos_max' to 'l2_cat_fits_cos_max'.
- patch 21: address Wei and Jan's comments.
    - modify commit message to remove error log.
    - replace unnecessary 'return' to 'break'.
    - restore 'libxl_psr_cat_get_l3_info' to keep interface backward compatible
      but change codes in it to call new function to get hw info.
    - add 'L2_CBM' into 'psr_cbm_type' because it is interface change which
      should be in same patch with new 'LIBXL_HAVE_' macro.
    - addjust logs sentence to make unnecessary error logs not show.
- patch 22: address Wei and Jan's comments.
    - remove 'L2_CBM' in idl because it has been moved to patch 21.

Yi Sun (24):
  docs: create L2 Cache Allocation Technology (CAT) feature document
  x86: refactor psr: remove L3 CAT/CDP codes.
  x86: refactor psr: implement main data structures.
  x86: refactor psr: implement CPU init and free flow.
  x86: refactor psr: implement Domain init/free and schedule flows.
  x86: refactor psr: implement get hw info flow.
  x86: refactor psr: implement get value flow.
  x86: refactor psr: set value: implement framework.
  x86: refactor psr: set value: assemble features value array.
  x86: refactor psr: set value: implement cos finding flow.
  x86: refactor psr: set value: implement cos id picking flow.
  x86: refactor psr: set value: implement write msr flow.
  x86: refactor psr: implement CPU init and free flow for CDP.
  x86: refactor psr: implement get hw info flow for CDP.
  x86: refactor psr: implement get value flow for CDP.
  x86: refactor psr: implement set value callback functions for CDP.
  x86: L2 CAT: implement CPU init and free flow.
  x86: L2 CAT: implement get hw info flow.
  x86: L2 CAT: implement get value flow.
  x86: L2 CAT: implement set value flow.
  tools: L2 CAT: support get HW info for L2 CAT.
  tools: L2 CAT: support show cbm for L2 CAT.
  tools: L2 CAT: support set cbm for L2 CAT.
  docs: add L2 CAT description in docs.

 docs/features/intel_psr_l2_cat.pandoc |  347 +++++++
 docs/man/xl.pod.1.in                  |   25 +-
 docs/misc/xl-psr.markdown             |   10 +-
 tools/libxc/include/xenctrl.h         |    7 +-
 tools/libxc/xc_psr.c                  |   46 +-
 tools/libxl/libxl.h                   |    9 +
 tools/libxl/libxl_psr.c               |   19 +-
 tools/libxl/libxl_types.idl           |    1 +
 tools/libxl/xl_cmdimpl.c              |  162 ++--
 tools/libxl/xl_cmdtable.c             |    4 +-
 xen/arch/x86/domctl.c                 |   49 +-
 xen/arch/x86/psr.c                    | 1593 +++++++++++++++++++++++++++------
 xen/arch/x86/sysctl.c                 |   45 +-
 xen/include/asm-x86/msr-index.h       |    1 +
 xen/include/asm-x86/psr.h             |   19 +-
 xen/include/public/domctl.h           |    2 +
 xen/include/public/sysctl.h           |    6 +
 17 files changed, 1951 insertions(+), 394 deletions(-)
 create mode 100644 docs/features/intel_psr_l2_cat.pandoc

-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-20  9:39   ` Tian, Kevin
  2017-01-30 18:10   ` Konrad Rzeszutek Wilk
  2017-01-19  6:01 ` [PATCH RESEND v5 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
                   ` (22 subsequent siblings)
  23 siblings, 2 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch creates L2 CAT feature document in doc/features/.
It describes details of L2 CAT.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
 docs/features/intel_psr_l2_cat.pandoc | 347 ++++++++++++++++++++++++++++++++++
 1 file changed, 347 insertions(+)
 create mode 100644 docs/features/intel_psr_l2_cat.pandoc

diff --git a/docs/features/intel_psr_l2_cat.pandoc b/docs/features/intel_psr_l2_cat.pandoc
new file mode 100644
index 0000000..77bd61f
--- /dev/null
+++ b/docs/features/intel_psr_l2_cat.pandoc
@@ -0,0 +1,347 @@
+% Intel L2 Cache Allocation Technology (L2 CAT) Feature
+% Revision 1.0
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+         Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+   Component(s): Hypervisor, toolstack
+
+       Hardware: Atom codename Goldmont and beyond CPUs
+---------------- ----------------------------------------------------
+
+# Overview
+
+L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
+CPU's shared L2 cache based on application priority or Class of Service
+(COS). Each CLOS is configured using capacity bitmasks (CBM) which
+represent cache capacity and indicate the degree of overlap and
+isolation between classes. Once L2 CAT is configured, the processor
+allows access to portions of L2 cache according to the established
+class of service.
+
+## Terminology
+
+* CAT         Cache Allocation Technology
+* CBM         Capacity BitMasks
+* CDP         Code and Data Prioritization
+* COS/CLOS    Class of Service
+* MSRs        Machine Specific Registers
+* PSR         Intel Platform Shared Resource
+* VMM         Virtual Machine Monitor
+
+# User details
+
+* Feature Enabling:
+
+  Add "psr=cat" to boot line parameter to enable all supported level CAT
+  features.
+
+* xl interfaces:
+
+  1. `psr-cat-show [OPTIONS] domain-id`:
+
+     Show domain L2 or L3 CAT CBM.
+
+     New option `-l` is added.
+     `-l2`: Show cbm for L2 cache.
+     `-l3`: Show cbm for L3 cache.
+
+     If neither `-l2` nor `-l3` is given, show both of them. If any one
+     is not supported, will print error info.
+
+  2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
+
+     Set domain L2 or L3 CBM.
+
+     New option `-l` is added.
+     `-l2`: Specify cbm for L2 cache.
+     `-l3`: Specify cbm for L3 cache.
+
+     If neither `-l2` nor `-l3` is given, level 3 is the default option.
+
+  3. `psr-hwinfo [OPTIONS]`:
+
+     Show L2 & L3 CAT HW informations on every socket.
+
+# Technical details
+
+L2 CAT is a member of Intel PSR features and part of CAT, it shares
+some base PSR infrastructure in Xen.
+
+## Hardware perspective
+
+L2 CAT defines a new range MSRs to assign different L2 cache access
+patterns which are known as CBMs, each CBM is associated with a COS.
+
+```
+
+                        +----------------------------+----------------+
+   IA32_PQR_ASSOC       | MSR (per socket)           |    Address     |
+ +----+---+-------+     +----------------------------+----------------+
+ |    |COS|       |     | IA32_L2_QOS_MASK_0         |     0xD10      |
+ +----+---+-------+     +----------------------------+----------------+
+        └-------------> | ...                        |  ...           |
+                        +----------------------------+----------------+
+                        | IA32_L2_QOS_MASK_n         | 0xD10+n (n<64) |
+                        +----------------------------+----------------+
+```
+
+When context switch happens, the COS of VCPU is written to per-thread
+MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
+according to the corresponding CBM.
+
+## The relationship between L2 CAT and L3 CAT/CDP
+
+L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
+while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.
+
+L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
+the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
+patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of
+L2 CAT must be continuous too.
+
+N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
+associate register `IA32_PQR_ASSOC`, which means one COS associates to a
+pair of L2 CBM and L3 CBM.
+
+Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
+other PSR features in future). In some cases, a VM is permitted to have a
+COS that is beyond one (or more) of PSR features but within the others.
+For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
+L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
+COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
+limit) since COS 9 is beyond the max COS (8) of L2 CAT.
+
+## Design Overview
+
+* Core COS/CBM association
+
+  When enforcing L2 CAT, all cores of domains have the same default
+  COS (COS0) which associated to the fully open CBM (all ones bitmask)
+  to access all L2 cache. The default COS is used only in hypervisor
+  and is transparent to tool stack and user.
+
+  System administrator can change PSR allocation policy at runtime by
+  tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds
+  to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP
+  is enabled, one COS corresponds to a 3-tuple, like [L2 CBM,
+  L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled,
+  things would be easier, one COS corresponds to one L2 CBM.
+
+* VCPU schedule
+
+  This part reuses L3 CAT COS infrastructure.
+
+* Multi-sockets
+
+  Different sockets may have different L2 CAT capability (e.g. max COS)
+  although it is consistent on the same socket. So the capability of
+  per-socket L2 CAT is specified.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+  1. Boot line parameter "psr=cat" now will enable L2 CAT and L3
+     CAT if hardware supported.
+
+  2. SYSCTL:
+          - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
+
+  3. DOMCTL:
+          - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
+          - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
+
+* xl interfaces:
+
+  1. psr-cat-show -l2 domain-id
+          Show L2 cbm for a domain.
+          => XEN_SYSCTL_PSR_CAT_get_l2_info /
+             XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
+
+  2. psr-mba-set -l2 domain-id cbm
+          Set L2 cbm for a domain.
+          => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
+
+  3. psr-hwinfo
+          Show PSR HW information, including L2 CAT
+          => XEN_SYSCTL_PSR_CAT_get_l2_info
+
+* Key data structure:
+
+   1. Feature HW info
+
+      ```
+      struct psr_cat_hw_info {
+          unsigned int cbm_len;
+          unsigned int cos_max;
+      };
+      ```
+
+      - Member `cbm_len`
+
+        `cbm_len` is one of the hardware info of CAT.
+
+      - Member `cos_max`
+
+        `cos_max` is one of the hardware info of CAT.
+
+   2. Feature list node
+
+      ```
+      struct feat_node {
+          enum psr_feat_type feature;
+          struct feat_ops ops;
+          struct psr_cat_hw_info info;
+          uint64_t cos_reg_val[MAX_COS_REG_NUM];
+          struct list_head list;
+      };
+      ```
+
+      When a PSR enforcement feature is enabled, it will be added into a
+      feature list. The head of the list is created in psr initialization.
+
+      - Member `feature`
+
+        `feature` is an integer number, to indicate which feature the list entry
+        corresponds to.
+
+      - Member `ops`
+
+        `ops` maintains a callback function list of the feature. It will be introduced
+        in details later.
+
+      - Member `info`
+
+        `info` maintains the feature HW information which can be got through
+        psr_hwinfo command.
+
+      - Member `cos_reg_val`
+
+        `cos_reg_val` is an array to maintain the value set in all COS registers of
+        the feature.
+
+   3. Per-socket PSR features information structure
+
+      ```
+      struct psr_cat_socket_info {
+          unsigned int feat_mask;
+          unsigned int nr_feat;
+          struct list_head feat_list;
+          unsigned int cos_ref[MAX_COS_REG_NUM];
+          spinlock_t ref_lock;
+      };
+      ```
+
+      We collect all PSR allocation features information of a socket in
+      this `struct psr_cat_socket_info`.
+
+      - Member `feat_mask`
+
+        `feat_mask` is a bitmap, to indicate which feature is enabled on
+        current socket. We define `feat_mask` bitmap as:
+
+        bit 0~1: L3 CAT status, [01] stands for L3 CAT only and [10]
+                 stands for L3 CDP is enalbed.
+
+        bit 2: L2 CAT status.
+
+      - Member `cos_ref`
+
+        `cos_ref` is an array which maintains the reference of one COS.
+        If the COS is used by one domain, the reference will increase one.
+        If a domain releases the COS, the reference will decrease one. The
+        array is indexed by COS.
+
+   4. Feature operation functions structure
+
+      ```
+      struct feat_ops {
+          void (*init_feature)(unsigned int eax, unsigned int ebx,
+                               unsigned int ecx, unsigned int edx,
+                               struct feat_node *feat,
+                               struct psr_cat_socket_info *info);
+          int (*get_feat_info)(const struct feat_node *feat, enum cbm_type type,
+                               uint32_t dat[], uint32_t array_len);
+          int (*get_val)(const struct feat_node *feat, unsigned int cos,
+                         enum cbm_type type, uint64_t *val);
+          unsigned int (*get_max_cos_max)(const struct feat_node *feat);
+          unsigned int (*get_cos_num)(const struct feat_node *feat);
+          int (*get_old_val)(uint64_t val[],
+                             const struct feat_node *feat,
+                             unsigned int old_cos);
+          int (*set_new_val)(uint64_t val[],
+                             const struct feat_node *feat,
+                             unsigned int old_cos,
+                             enum cbm_type type,
+                             uint64_t m);
+          int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
+                             unsigned int cos, bool *found);
+          unsigned int (*get_cos_max_from_type)(const struct feat_node *feat,
+                                                enum cbm_type type);
+          unsigned int (*exceeds_cos_max)(const uint64_t val[],
+                                          const struct feat_node *feat,
+                                          unsigned int cos);
+          int (*write_msr)(unsigned int cos, const uint64_t val[],
+                           struct feat_node *feat);
+      };
+      ```
+
+      We abstract above callback functions to encapsulate the feature specific
+      behaviors into them. Then, it is easy to add a new feature. We just need:
+          1) Implement such ops and callback functions for every feature.
+          2) Register the ops into `struct feat_node`.
+          3) Add the feature into feature list during CPU initialization.
+
+# Limitations
+
+L2 CAT can only work on HW which enables it(check by CPUID). So far, there
+is no HW enables both L2 CAT and L3 CAT/CDP. But SW implementation has considered
+such scenario to enable both L2 CAT and L3 CAT/CDP.
+
+# Testing
+
+L2 CAT uses same xl interfaces as L3 CAT/CDP. So, we can execute these
+commands to verify L2 CAT and L3 CAT/CDP on different HWs support them.
+
+For example:
+    root@:~$ xl psr-hwinfo --cat
+    Cache Allocation Technology (CAT): L2
+    Socket ID       : 0
+    Maximum COS     : 3
+    CBM length      : 8
+    Default CBM     : 0xff
+
+    root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
+
+    root@:~$ xl psr-cat-show -l2 1
+    Socket ID       : 0
+    Default CBM     : 0xff
+       ID                     NAME             CBM
+        1                 ubuntu14            0x7f
+
+# Areas for improvement
+
+N/A
+
+# Known issues
+
+N/A
+
+# References
+
+"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
+2016-08-12 1.0      Xen 4.9  Design document written
+---------- -------- -------- -------------------------------------------
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 02/24] x86: refactor psr: remove L3 CAT/CDP codes.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-30 22:05   ` Konrad Rzeszutek Wilk
  2017-01-19  6:01 ` [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures Yi Sun
                   ` (21 subsequent siblings)
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

The current cache allocation codes in psr.c do not consider
future features addition and are not friendly to extend.

To make psr.c be more flexible to add new features and fulfill
the program principle, open for extension but closed for
modification, we have to refactor the psr.c:
1. Analyze cache allocation features and abstract general data
   structures.
2. Analyze the init and all other functions flow, abstract all
   steps that different features may have different implementations.
   Make these steps be callback functions and register feature
   specific fuctions. Then, the main processes will not be changed
   when introducing a new feature.

Because the quantity of refactor codes is big and the logics are
changed a lot, it will cause reviewers confused if just change
old codes. Reviewers have to understand both old codes and new
implementations. After review iterations from V1 to V3, Jan has
proposed to remove all old cache allocation codes firstly, then
implement new codes step by step. This will help to make codes
be more easily reviewable.

There is no construction without destruction. So, this patch
removes all current L3 CAT/CDP codes in psr.c. The following
patches will introduce the new mechanism.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
 xen/arch/x86/psr.c | 470 +----------------------------------------------------
 1 file changed, 5 insertions(+), 465 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0b5073c..96a8589 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -23,24 +23,6 @@
 #define PSR_CAT        (1<<1)
 #define PSR_CDP        (1<<2)
 
-struct psr_cat_cbm {
-    union {
-        uint64_t cbm;
-        struct {
-            uint64_t code;
-            uint64_t data;
-        };
-    };
-    unsigned int ref;
-};
-
-struct psr_cat_socket_info {
-    unsigned int cbm_len;
-    unsigned int cos_max;
-    struct psr_cat_cbm *cos_to_cbm;
-    spinlock_t cbm_lock;
-};
-
 struct psr_assoc {
     uint64_t val;
     uint64_t cos_mask;
@@ -48,26 +30,11 @@ struct psr_assoc {
 
 struct psr_cmt *__read_mostly psr_cmt;
 
-static unsigned long *__read_mostly cat_socket_enable;
-static struct psr_cat_socket_info *__read_mostly cat_socket_info;
-static unsigned long *__read_mostly cdp_socket_enable;
-
 static unsigned int opt_psr;
 static unsigned int __initdata opt_rmid_max = 255;
-static unsigned int __read_mostly opt_cos_max = 255;
 static uint64_t rmid_mask;
 static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
 
-static struct psr_cat_cbm *temp_cos_to_cbm;
-
-static unsigned int get_socket_cpu(unsigned int socket)
-{
-    if ( likely(socket < nr_sockets) )
-        return cpumask_any(socket_cpumask[socket]);
-
-    return nr_cpu_ids;
-}
-
 static void __init parse_psr_bool(char *s, char *value, char *feature,
                                   unsigned int mask)
 {
@@ -107,9 +74,6 @@ static void __init parse_psr_param(char *s)
         if ( val_str && !strcmp(s, "rmid_max") )
             opt_rmid_max = simple_strtoul(val_str, NULL, 0);
 
-        if ( val_str && !strcmp(s, "cos_max") )
-            opt_cos_max = simple_strtoul(val_str, NULL, 0);
-
         s = ss + 1;
     } while ( ss );
 }
@@ -213,16 +177,7 @@ static inline void psr_assoc_init(void)
 {
     struct psr_assoc *psra = &this_cpu(psr_assoc);
 
-    if ( cat_socket_info )
-    {
-        unsigned int socket = cpu_to_socket(smp_processor_id());
-
-        if ( test_bit(socket, cat_socket_enable) )
-            psra->cos_mask = ((1ull << get_count_order(
-                             cat_socket_info[socket].cos_max)) - 1) << 32;
-    }
-
-    if ( psr_cmt_enabled() || psra->cos_mask )
+    if ( psr_cmt_enabled() )
         rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
 }
 
@@ -231,12 +186,6 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
     *reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
 }
 
-static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
-                                 uint64_t cos_mask)
-{
-    *reg = (*reg & ~cos_mask) | (((uint64_t)cos << 32) & cos_mask);
-}
-
 void psr_ctxt_switch_to(struct domain *d)
 {
     struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -245,459 +194,54 @@ void psr_ctxt_switch_to(struct domain *d)
     if ( psr_cmt_enabled() )
         psr_assoc_rmid(&reg, d->arch.psr_rmid);
 
-    if ( psra->cos_mask )
-        psr_assoc_cos(&reg, d->arch.psr_cos_ids ?
-                      d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
-                      0, psra->cos_mask);
-
     if ( reg != psra->val )
     {
         wrmsrl(MSR_IA32_PSR_ASSOC, reg);
         psra->val = reg;
     }
 }
-static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
-{
-    if ( !cat_socket_info )
-        return ERR_PTR(-ENODEV);
-
-    if ( socket >= nr_sockets )
-        return ERR_PTR(-ENOTSOCK);
-
-    if ( !test_bit(socket, cat_socket_enable) )
-        return ERR_PTR(-ENOENT);
-
-    return cat_socket_info + socket;
-}
-
-static inline bool_t cdp_is_enabled(unsigned int socket)
-{
-    return cdp_socket_enable && test_bit(socket, cdp_socket_enable);
-}
 
 int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
                         uint32_t *cos_max, uint32_t *flags)
 {
-    struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
-    if ( IS_ERR(info) )
-        return PTR_ERR(info);
-
-    *cbm_len = info->cbm_len;
-    *cos_max = info->cos_max;
-
-    *flags = 0;
-    if ( cdp_is_enabled(socket) )
-        *flags |= XEN_SYSCTL_PSR_CAT_L3_CDP;
-
     return 0;
 }
 
 int psr_get_l3_cbm(struct domain *d, unsigned int socket,
                    uint64_t *cbm, enum cbm_type type)
 {
-    struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-    bool_t cdp_enabled = cdp_is_enabled(socket);
-
-    if ( IS_ERR(info) )
-        return PTR_ERR(info);
-
-    switch ( type )
-    {
-    case PSR_CBM_TYPE_L3:
-        if ( cdp_enabled )
-            return -EXDEV;
-        *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
-        break;
-
-    case PSR_CBM_TYPE_L3_CODE:
-        if ( !cdp_enabled )
-            *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
-        else
-            *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].code;
-        break;
-
-    case PSR_CBM_TYPE_L3_DATA:
-        if ( !cdp_enabled )
-            *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
-        else
-            *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].data;
-        break;
-
-    default:
-        ASSERT_UNREACHABLE();
-    }
-
-    return 0;
-}
-
-static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
-{
-    unsigned int first_bit, zero_bit;
-
-    /* Set bits should only in the range of [0, cbm_len). */
-    if ( cbm & (~0ull << cbm_len) )
-        return 0;
-
-    /* At least one bit need to be set. */
-    if ( cbm == 0 )
-        return 0;
-
-    first_bit = find_first_bit(&cbm, cbm_len);
-    zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
-
-    /* Set bits should be contiguous. */
-    if ( zero_bit < cbm_len &&
-         find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
-        return 0;
-
-    return 1;
-}
-
-struct cos_cbm_info
-{
-    unsigned int cos;
-    bool_t cdp;
-    uint64_t cbm_code;
-    uint64_t cbm_data;
-};
-
-static void do_write_l3_cbm(void *data)
-{
-    struct cos_cbm_info *info = data;
-
-    if ( info->cdp )
-    {
-        wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(info->cos), info->cbm_code);
-        wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(info->cos), info->cbm_data);
-    }
-    else
-        wrmsrl(MSR_IA32_PSR_L3_MASK(info->cos), info->cbm_code);
-}
-
-static int write_l3_cbm(unsigned int socket, unsigned int cos,
-                        uint64_t cbm_code, uint64_t cbm_data, bool_t cdp)
-{
-    struct cos_cbm_info info =
-    {
-        .cos = cos,
-        .cbm_code = cbm_code,
-        .cbm_data = cbm_data,
-        .cdp = cdp,
-    };
-
-    if ( socket == cpu_to_socket(smp_processor_id()) )
-        do_write_l3_cbm(&info);
-    else
-    {
-        unsigned int cpu = get_socket_cpu(socket);
-
-        if ( cpu >= nr_cpu_ids )
-            return -ENOTSOCK;
-        on_selected_cpus(cpumask_of(cpu), do_write_l3_cbm, &info, 1);
-    }
-
     return 0;
 }
 
-static int find_cos(struct psr_cat_cbm *map, unsigned int cos_max,
-                    uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
-{
-    unsigned int cos;
-
-    for ( cos = 0; cos <= cos_max; cos++ )
-    {
-        if ( (map[cos].ref || cos == 0) &&
-             ((!cdp_enabled && map[cos].cbm == cbm_code) ||
-              (cdp_enabled && map[cos].code == cbm_code &&
-                              map[cos].data == cbm_data)) )
-            return cos;
-    }
-
-    return -ENOENT;
-}
-
-static int pick_avail_cos(struct psr_cat_cbm *map, unsigned int cos_max,
-                          unsigned int old_cos)
-{
-    unsigned int cos;
-
-    /* If old cos is referred only by the domain, then use it. */
-    if ( map[old_cos].ref == 1 && old_cos != 0 )
-        return old_cos;
-
-    /* Find an unused one other than cos0. */
-    for ( cos = 1; cos <= cos_max; cos++ )
-        if ( map[cos].ref == 0 )
-            return cos;
-
-    return -ENOENT;
-}
-
 int psr_set_l3_cbm(struct domain *d, unsigned int socket,
                    uint64_t cbm, enum cbm_type type)
 {
-    unsigned int old_cos, cos_max;
-    int cos, ret;
-    uint64_t cbm_data, cbm_code;
-    bool_t cdp_enabled = cdp_is_enabled(socket);
-    struct psr_cat_cbm *map;
-    struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
-    if ( IS_ERR(info) )
-        return PTR_ERR(info);
-
-    if ( !psr_check_cbm(info->cbm_len, cbm) )
-        return -EINVAL;
-
-    if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
-                          type == PSR_CBM_TYPE_L3_DATA) )
-        return -ENXIO;
-
-    cos_max = info->cos_max;
-    old_cos = d->arch.psr_cos_ids[socket];
-    map = info->cos_to_cbm;
-
-    switch ( type )
-    {
-    case PSR_CBM_TYPE_L3:
-        cbm_code = cbm;
-        cbm_data = cbm;
-        break;
-
-    case PSR_CBM_TYPE_L3_CODE:
-        cbm_code = cbm;
-        cbm_data = map[old_cos].data;
-        break;
-
-    case PSR_CBM_TYPE_L3_DATA:
-        cbm_code = map[old_cos].code;
-        cbm_data = cbm;
-        break;
-
-    default:
-        ASSERT_UNREACHABLE();
-        return -EINVAL;
-    }
-
-    spin_lock(&info->cbm_lock);
-    cos = find_cos(map, cos_max, cbm_code, cbm_data, cdp_enabled);
-    if ( cos >= 0 )
-    {
-        if ( cos == old_cos )
-        {
-            spin_unlock(&info->cbm_lock);
-            return 0;
-        }
-    }
-    else
-    {
-        cos = pick_avail_cos(map, cos_max, old_cos);
-        if ( cos < 0 )
-        {
-            spin_unlock(&info->cbm_lock);
-            return cos;
-        }
-
-        /* We try to avoid writing MSR. */
-        if ( (cdp_enabled &&
-             (map[cos].code != cbm_code || map[cos].data != cbm_data)) ||
-             (!cdp_enabled && map[cos].cbm != cbm_code) )
-        {
-            ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
-            if ( ret )
-            {
-                spin_unlock(&info->cbm_lock);
-                return ret;
-            }
-            map[cos].code = cbm_code;
-            map[cos].data = cbm_data;
-        }
-    }
-
-    map[cos].ref++;
-    map[old_cos].ref--;
-    spin_unlock(&info->cbm_lock);
-
-    d->arch.psr_cos_ids[socket] = cos;
-
     return 0;
 }
 
-/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
-static void psr_free_cos(struct domain *d)
-{
-    unsigned int socket;
-    unsigned int cos;
-    struct psr_cat_socket_info *info;
-
-    if( !d->arch.psr_cos_ids )
-        return;
-
-    for_each_set_bit(socket, cat_socket_enable, nr_sockets)
-    {
-        if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
-            continue;
-
-        info = cat_socket_info + socket;
-        spin_lock(&info->cbm_lock);
-        info->cos_to_cbm[cos].ref--;
-        spin_unlock(&info->cbm_lock);
-    }
-
-    xfree(d->arch.psr_cos_ids);
-    d->arch.psr_cos_ids = NULL;
-}
-
 int psr_domain_init(struct domain *d)
 {
-    if ( cat_socket_info )
-    {
-        d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
-        if ( !d->arch.psr_cos_ids )
-            return -ENOMEM;
-    }
-
     return 0;
 }
 
 void psr_domain_free(struct domain *d)
 {
     psr_free_rmid(d);
-    psr_free_cos(d);
-}
-
-static int cat_cpu_prepare(unsigned int cpu)
-{
-    if ( !cat_socket_info )
-        return 0;
-
-    if ( temp_cos_to_cbm == NULL &&
-         (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
-                                          opt_cos_max + 1UL)) == NULL )
-        return -ENOMEM;
-
-    return 0;
-}
-
-static void cat_cpu_init(void)
-{
-    unsigned int eax, ebx, ecx, edx;
-    struct psr_cat_socket_info *info;
-    unsigned int socket;
-    unsigned int cpu = smp_processor_id();
-    uint64_t val;
-    const struct cpuinfo_x86 *c = cpu_data + cpu;
-
-    if ( !cpu_has(c, X86_FEATURE_PQE) || c->cpuid_level < PSR_CPUID_LEVEL_CAT )
-        return;
-
-    socket = cpu_to_socket(cpu);
-    if ( test_bit(socket, cat_socket_enable) )
-        return;
-
-    cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
-    if ( ebx & PSR_RESOURCE_TYPE_L3 )
-    {
-        cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
-        info = cat_socket_info + socket;
-        info->cbm_len = (eax & 0x1f) + 1;
-        info->cos_max = min(opt_cos_max, edx & 0xffff);
-
-        info->cos_to_cbm = temp_cos_to_cbm;
-        temp_cos_to_cbm = NULL;
-        /* cos=0 is reserved as default cbm(all ones). */
-        info->cos_to_cbm[0].cbm = (1ull << info->cbm_len) - 1;
-
-        spin_lock_init(&info->cbm_lock);
-
-        set_bit(socket, cat_socket_enable);
-
-        if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
-             cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
-        {
-            info->cos_to_cbm[0].code = (1ull << info->cbm_len) - 1;
-            info->cos_to_cbm[0].data = (1ull << info->cbm_len) - 1;
-
-            /* We only write mask1 since mask0 is always all ones by default. */
-            wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->cbm_len) - 1);
-
-            rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
-            wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
-
-            /* Cut half of cos_max when CDP is enabled. */
-            info->cos_max >>= 1;
-
-            set_bit(socket, cdp_socket_enable);
-        }
-        printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
-               socket, info->cos_max, info->cbm_len,
-               cdp_is_enabled(socket) ? "on" : "off");
-    }
-}
-
-static void cat_cpu_fini(unsigned int cpu)
-{
-    unsigned int socket = cpu_to_socket(cpu);
-
-    if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
-    {
-        struct psr_cat_socket_info *info = cat_socket_info + socket;
-
-        if ( info->cos_to_cbm )
-        {
-            xfree(info->cos_to_cbm);
-            info->cos_to_cbm = NULL;
-        }
-
-        if ( cdp_is_enabled(socket) )
-            clear_bit(socket, cdp_socket_enable);
-
-        clear_bit(socket, cat_socket_enable);
-    }
-}
-
-static void __init psr_cat_free(void)
-{
-    xfree(cat_socket_enable);
-    cat_socket_enable = NULL;
-    xfree(cat_socket_info);
-    cat_socket_info = NULL;
-}
-
-static void __init init_psr_cat(void)
-{
-    if ( opt_cos_max < 1 )
-    {
-        printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
-        return;
-    }
-
-    cat_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
-    cat_socket_info = xzalloc_array(struct psr_cat_socket_info, nr_sockets);
-    cdp_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
-
-    if ( !cat_socket_enable || !cat_socket_info )
-        psr_cat_free();
 }
 
 static int psr_cpu_prepare(unsigned int cpu)
 {
-    return cat_cpu_prepare(cpu);
+    return 0;
 }
 
 static void psr_cpu_init(void)
 {
-    if ( cat_socket_info )
-        cat_cpu_init();
-
     psr_assoc_init();
 }
 
 static void psr_cpu_fini(unsigned int cpu)
 {
-    if ( cat_socket_info )
-        cat_cpu_fini(cpu);
+    return;
 }
 
 static int cpu_callback(
@@ -738,14 +282,10 @@ static int __init psr_presmp_init(void)
     if ( (opt_psr & PSR_CMT) && opt_rmid_max )
         init_psr_cmt(opt_rmid_max);
 
-    if ( opt_psr & PSR_CAT )
-        init_psr_cat();
-
-    if ( psr_cpu_prepare(0) )
-        psr_cat_free();
+    psr_cpu_prepare(0);
 
     psr_cpu_init();
-    if ( psr_cmt_enabled() || cat_socket_info )
+    if ( psr_cmt_enabled() )
         register_cpu_notifier(&cpu_nfb);
 
     return 0;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-30 22:20   ` Konrad Rzeszutek Wilk
  2017-01-19  6:01 ` [PATCH RESEND v5 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
                   ` (20 subsequent siblings)
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

To construct an extendible framework, we need analyze PSR features
and abstract the common things and feature specific things. Then,
encapsulate them into different data structures.

By analyzing PSR features, we can get below map.
                +------+------+------+
      --------->| Dom0 | Dom1 | ...  |
      |         +------+------+------+
      |            |
      |Dom ID      | cos_id of domain
      |            V
      |        +-----------------------------------------------------------------------------+
User --------->| PSR                                                                         |
     Socket ID |  +--------------+---------------+---------------+                           |
               |  | Socket0 Info | Socket 1 Info |    ...        |                           |
               |  +--------------+---------------+---------------+                           |
               |    |                   cos_id=0               cos_id=1          ...         |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |->Ref   : |         ref 0         |         ref 1         | ...       | |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |->L3 CAT: |         cos 0         |         cos 1         | ...       | |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |->L2 CAT: |         cos 0         |         cos 1         | ...       | |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |          +-----------+-----------+-----------+-----------+-----------+ |
               |    |->CDP   : | cos0 code | cos0 data | cos1 code | cos1 data | ...       | |
               |               +-----------+-----------+-----------+-----------+-----------+ |
               +-----------------------------------------------------------------------------+

So, we need define a socket info data structure, 'struct
psr_socket_info' to manage information per socket. It contains a
reference count array according to COS ID and a feature list to
manage all features enabled. Every entry of the reference count
array is used to record how many domains are using the COS registers
according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
Dom1 uses COS_ID=1 registers of both features to save CBM values, like
below.
        +-------+-------+-------+-----+
        | COS 0 | COS 1 | COS 2 | ... |
        +-------+-------+-------+-----+
L3 CAT  | 0x7ff | 0x1ff | ...   | ... |
        +-------+-------+-------+-----+
L2 CAT  | 0xff  | 0xff  | ...   | ... |
        +-------+-------+-------+-----+

If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
That means, both Dom1 and Dom2 use same COS registers(ID=1) to save same
L3/L2 values. So, the value ref[1] is 2 which means 2 domains are using
COS_ID 1.

To manage a feature, we need define a feature node data structure,
'struct feat_node', to manage feature's specific HW info, its callback
functions (all feature's specific behaviors are encapsulated into these
callback functions), and an array of all COS registers values of this
feature.

CDP is a special feature which uses two entries of the array
for one COS ID. So, the number of CDP COS registers is the half of L3
CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
it is enabled. CDP uses the COS registers array as below.

                         +-----------+-----------+-----------+-----------+-----------+
CDP cos_reg_val[] index: |     0     |     1     |     2     |     3     |    ...    |
                         +-----------+-----------+-----------+-----------+-----------+
                  value: | cos0 code | cos0 data | cos1 code | cos1 data |    ...    |
                         +-----------+-----------+-----------+-----------+-----------+

For more details, please refer spec and codes.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - explain CDP more in commit message.
    - remove exact SDM chapter number but only keep title.
    - remove init_feature from callback function ops structure.
---
 xen/arch/x86/psr.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 104 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 96a8589..f7ff3fc 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -17,12 +17,116 @@
 #include <xen/cpu.h>
 #include <xen/err.h>
 #include <xen/sched.h>
+#include <xen/list.h>
 #include <asm/psr.h>
 
+/*
+ * Terminology:
+ * - CAT         Cache Allocation Technology
+ * - CBM         Capacity BitMasks
+ * - CDP         Code and Data Prioritization
+ * - COS/CLOS    Class of Service. Also mean COS registers.
+ * - COS_MAX     Max number of COS for the feature (minus 1)
+ * - MSRs        Machine Specific Registers
+ * - PSR         Intel Platform Shared Resource
+ */
+
 #define PSR_CMT        (1<<0)
 #define PSR_CAT        (1<<1)
 #define PSR_CDP        (1<<2)
 
+/*
+ * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
+ * the MSRs range from 0C90H through 0D0FH (inclusive), enables support for
+ * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
+ *
+ * The MSRs range from 0D10H through 0D4FH (inclusive), enables support for
+ * up to 64 L2 CAT COS. The COS_ID=[0,63].
+ *
+ * So, the maximum COS register count of one feature is 128.
+ */
+#define MAX_COS_REG_CNT  128
+
+/*
+ * PSR features are managed per socket. Below structure defines the members
+ * used to manage these features.
+ * feat_mask - Mask used to record features enabled on socket. There may be
+ *             some features enabled at same time.
+ * nr_feat   - Record how many features enabled.
+ * feat_list - A list used to manage all features enabled.
+ * cos_ref   - A reference count array to record how many domains are using the
+ *             COS_ID.
+ *             Every entry of cos_ref corresponds to one COS ID.
+ * ref_lock  - A lock to protect cos_ref.
+ */
+struct psr_socket_info {
+    /*
+     * bit 0:   L3 CAT
+     * bit 1:   L3 CDP
+     * bit 2:   L2 CAT
+     */
+    unsigned int feat_mask;
+    unsigned int nr_feat;
+    struct list_head feat_list;
+    unsigned int cos_ref[MAX_COS_REG_CNT];
+    spinlock_t ref_lock;
+};
+
+enum psr_feat_type {
+    PSR_SOCKET_L3_CAT = 0,
+    PSR_SOCKET_L3_CDP,
+    PSR_SOCKET_L2_CAT,
+};
+
+/* CAT/CDP HW info data structure. */
+struct psr_cat_hw_info {
+    unsigned int cbm_len;
+    unsigned int cos_max;
+};
+
+/* Encapsulate feature specific HW info here. */
+struct feat_hw_info {
+    union {
+        struct psr_cat_hw_info l3_cat_info;
+    };
+};
+
+struct feat_node;
+
+/*
+ * This structure defines feature operation callback functions. Every feature
+ * enabled MUST implement such callback functions and register them to ops.
+ *
+ * Feature specific behaviors will be encapsulated into these callback
+ * functions. Then, the main flows will not be changed when introducing a new
+ * feature.
+ */
+struct feat_ops {
+    /* get_cos_max is used to get feature's cos_max. */
+    unsigned int (*get_cos_max)(const struct feat_node *feat);
+};
+
+/*
+ * This structure represents one feature.
+ * feature     - Which feature it is.
+ * feat_ops    - Feature operation callback functions.
+ * info        - Feature HW info.
+ * cos_reg_val - Array to store the values of COS registers. One entry stores
+ *               the value of one COS register.
+ *               For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
+ *               For CDP, two entries correspond to one COS_ID. E.g.
+ *               COS_ID=0 corresponds to cos_reg_val[0] (Data) and
+ *               cos_reg_val[1] (Code).
+ * list        - Feature list.
+ */
+struct feat_node {
+    enum psr_feat_type feature;
+    struct feat_ops ops;
+    struct feat_hw_info info;
+    uint64_t cos_reg_val[MAX_COS_REG_CNT];
+    struct list_head list;
+};
+
 struct psr_assoc {
     uint64_t val;
     uint64_t cos_mask;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 04/24] x86: refactor psr: implement CPU init and free flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (2 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-31  2:44   ` Konrad Rzeszutek Wilk
  2017-01-19  6:01 ` [PATCH RESEND v5 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
                   ` (19 subsequent siblings)
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements the CPU init and free flow including L3 CAT
initialization and feature list free.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - modify commit message beacuse of code changes.
    - add 'struct cpuid_leaf_regs' to save cpu registers value to reduce
      parameters of init_feature function.
    - modify comments to make them accurate.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
    - use 'list_for_each_entry_safe' when free features.
    - do not delete 'feat_l3_cat' to make it can be reused when cpu online.
    - use 'current_cpu_data'.
    - clear 'X86_FEATURE_PQE' if cpuid_level is not right.
    - Print socket info when 'opt_cpu_info' is true.
    - remove 'cpu_prepare_work' function and move contents of it into
      'psr_cpu_prepare'.
---
 xen/arch/x86/psr.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 174 insertions(+), 2 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index f7ff3fc..e9dc07a 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -35,6 +35,9 @@
 #define PSR_CAT        (1<<1)
 #define PSR_CDP        (1<<2)
 
+#define CAT_CBM_LEN_MASK 0x1f
+#define CAT_COS_MAX_MASK 0xffff
+
 /*
  * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
  * the MSRs range from 0C90H through 0D0FH (inclusive), enables support for
@@ -127,6 +130,13 @@ struct feat_node {
     struct list_head list;
 };
 
+struct cpuid_leaf_regs {
+    unsigned int eax;
+    unsigned int ebx;
+    unsigned int ecx;
+    unsigned int edx;
+};
+
 struct psr_assoc {
     uint64_t val;
     uint64_t cos_mask;
@@ -134,11 +144,76 @@ struct psr_assoc {
 
 struct psr_cmt *__read_mostly psr_cmt;
 
+static struct psr_socket_info *__read_mostly socket_info;
+
 static unsigned int opt_psr;
 static unsigned int __initdata opt_rmid_max = 255;
+static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
 static uint64_t rmid_mask;
 static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
 
+/*
+ * Declare global feature list entry for every feature to facilitate the
+ * feature list creation. It will be allocated in psr_cpu_prepare() and
+ * inserted into feature list in cpu_init_work().
+ */
+static struct feat_node *feat_l3_cat;
+
+/* Common functions. */
+static void free_feature(struct psr_socket_info *info)
+{
+    struct feat_node *feat, *next;
+
+    if ( !info )
+        return;
+
+    list_for_each_entry_safe(feat, next, &info->feat_list, list)
+    {
+        clear_bit(feat->feature, &info->feat_mask);
+        list_del(&feat->list);
+        xfree(feat);
+    }
+}
+
+/* L3 CAT functions implementation. */
+static void l3_cat_init_feature(struct cpuid_leaf_regs regs,
+                                struct feat_node *feat,
+                                struct psr_socket_info *info)
+{
+    struct psr_cat_hw_info l3_cat;
+    unsigned int socket;
+
+    /* No valid value so do not enable feature. */
+    if ( !regs.eax || !regs.edx )
+        return;
+
+    l3_cat.cbm_len = (regs.eax & CAT_CBM_LEN_MASK) + 1;
+    l3_cat.cos_max = min(opt_cos_max, regs.edx & CAT_COS_MAX_MASK);
+
+    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+    feat->cos_reg_val[0] = (1ull << l3_cat.cbm_len) - 1;
+
+    feat->feature = PSR_SOCKET_L3_CAT;
+    __set_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);
+
+    feat->info.l3_cat_info = l3_cat;
+
+    info->nr_feat++;
+
+    /* Add this feature into list. */
+    list_add_tail(&feat->list, &info->feat_list);
+
+    socket = cpu_to_socket(smp_processor_id());
+    if ( opt_cpu_info )
+        printk(XENLOG_INFO
+           "L3 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+           socket, feat->info.l3_cat_info.cos_max,
+           feat->info.l3_cat_info.cbm_len);
+}
+
+static const struct feat_ops l3_cat_ops = {
+};
+
 static void __init parse_psr_bool(char *s, char *value, char *feature,
                                   unsigned int mask)
 {
@@ -178,6 +253,9 @@ static void __init parse_psr_param(char *s)
         if ( val_str && !strcmp(s, "rmid_max") )
             opt_rmid_max = simple_strtoul(val_str, NULL, 0);
 
+        if ( val_str && !strcmp(s, "cos_max") )
+            opt_cos_max = simple_strtoul(val_str, NULL, 0);
+
         s = ss + 1;
     } while ( ss );
 }
@@ -333,18 +411,108 @@ void psr_domain_free(struct domain *d)
     psr_free_rmid(d);
 }
 
+static void cpu_init_work(void)
+{
+    struct psr_socket_info *info;
+    unsigned int socket;
+    unsigned int cpu = smp_processor_id();
+    struct feat_node *feat;
+    struct cpuid_leaf_regs regs;
+
+    if ( !cpu_has(&current_cpu_data, X86_FEATURE_PQE) )
+        return;
+    else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
+    {
+        clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
+        return;
+    }
+
+    socket = cpu_to_socket(cpu);
+    info = socket_info + socket;
+    if ( info->feat_mask )
+        return;
+
+    INIT_LIST_HEAD(&info->feat_list);
+    spin_lock_init(&info->ref_lock);
+
+    cpuid_count(PSR_CPUID_LEVEL_CAT, 0,
+                &regs.eax, &regs.ebx, &regs.ecx, &regs.edx);
+    if ( regs.ebx & PSR_RESOURCE_TYPE_L3 )
+    {
+        cpuid_count(PSR_CPUID_LEVEL_CAT, 1,
+                    &regs.eax, &regs.ebx, &regs.ecx, &regs.edx);
+
+        feat = feat_l3_cat;
+        feat_l3_cat = NULL;
+        feat->ops = l3_cat_ops;
+
+        l3_cat_init_feature(regs, feat, info);
+    }
+}
+
+static void cpu_fini_work(unsigned int cpu)
+{
+    unsigned int socket = cpu_to_socket(cpu);
+
+    if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
+    {
+        free_feature(socket_info + socket);
+    }
+}
+
+static void __init init_psr(void)
+{
+    if ( opt_cos_max < 1 )
+    {
+        printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
+        return;
+    }
+
+    socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
+
+    if ( !socket_info )
+    {
+        printk(XENLOG_INFO "Fail to alloc socket_info!\n");
+        return;
+    }
+}
+
+static void __init psr_free(void)
+{
+    unsigned int i;
+
+    for ( i = 0; i < nr_sockets; i++ )
+        free_feature(&socket_info[i]);
+
+    xfree(socket_info);
+    socket_info = NULL;
+}
+
 static int psr_cpu_prepare(unsigned int cpu)
 {
+    if ( !socket_info )
+        return 0;
+
+    /* Malloc memory for the global feature head here. */
+    if ( feat_l3_cat == NULL &&
+         (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
+        return -ENOMEM;
+
     return 0;
 }
 
 static void psr_cpu_init(void)
 {
+    if ( socket_info )
+        cpu_init_work();
+
     psr_assoc_init();
 }
 
 static void psr_cpu_fini(unsigned int cpu)
 {
+    if ( socket_info )
+        cpu_fini_work(cpu);
     return;
 }
 
@@ -386,10 +554,14 @@ static int __init psr_presmp_init(void)
     if ( (opt_psr & PSR_CMT) && opt_rmid_max )
         init_psr_cmt(opt_rmid_max);
 
-    psr_cpu_prepare(0);
+    if ( opt_psr & PSR_CAT )
+        init_psr();
+
+    if ( psr_cpu_prepare(0) )
+        psr_free();
 
     psr_cpu_init();
-    if ( psr_cmt_enabled() )
+    if ( psr_cmt_enabled() || socket_info )
         register_cpu_notifier(&cpu_nfb);
 
     return 0;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 05/24] x86: refactor psr: implement Domain init/free and schedule flows.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (3 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-31 19:52   ` Konrad Rzeszutek Wilk
  2017-01-19  6:01 ` [PATCH RESEND v5 06/24] x86: refactor psr: implement get hw info flow Yi Sun
                   ` (18 subsequent siblings)
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements the Domain init/free and schedule flows.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - define macro 'PSR_ASSOC_REG_POS' to replace integer 32.
    - rename 'l3_cat_get_max_cos_max' to 'l3_cat_get_cos_max'.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
---
 xen/arch/x86/psr.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 61 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index e9dc07a..7f06235 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -50,6 +50,8 @@
  */
 #define MAX_COS_REG_CNT  128
 
+#define PSR_ASSOC_REG_POS 32
+
 /*
  * PSR features are managed per socket. Below structure defines the members
  * used to manage these features.
@@ -211,7 +213,13 @@ static void l3_cat_init_feature(struct cpuid_leaf_regs regs,
            feat->info.l3_cat_info.cbm_len);
 }
 
+static unsigned int l3_cat_get_cos_max(const struct feat_node *feat)
+{
+    return feat->info.l3_cat_info.cos_max;
+}
+
 static const struct feat_ops l3_cat_ops = {
+    .get_cos_max = l3_cat_get_cos_max,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -355,11 +363,33 @@ void psr_free_rmid(struct domain *d)
     d->arch.psr_rmid = 0;
 }
 
+static inline unsigned int get_max_cos_max(const struct psr_socket_info *info)
+{
+    const struct feat_node *feat;
+    unsigned int cos_max = 0;
+
+    list_for_each_entry(feat, &info->feat_list, list)
+        cos_max = max(feat->ops.get_cos_max(feat), cos_max);
+
+    return cos_max;
+}
+
 static inline void psr_assoc_init(void)
 {
     struct psr_assoc *psra = &this_cpu(psr_assoc);
 
-    if ( psr_cmt_enabled() )
+    if ( socket_info )
+    {
+        unsigned int socket = cpu_to_socket(smp_processor_id());
+        const struct psr_socket_info *info = socket_info + socket;
+        unsigned int cos_max = get_max_cos_max(info);
+
+        if ( info->feat_mask )
+            psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
+                              PSR_ASSOC_REG_POS;
+    }
+
+    if ( psr_cmt_enabled() || psra->cos_mask )
         rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
 }
 
@@ -368,6 +398,13 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
     *reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
 }
 
+static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
+                                 uint64_t cos_mask)
+{
+    *reg = (*reg & ~cos_mask) |
+            (((uint64_t)cos << PSR_ASSOC_REG_POS) & cos_mask);
+}
+
 void psr_ctxt_switch_to(struct domain *d)
 {
     struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -376,6 +413,11 @@ void psr_ctxt_switch_to(struct domain *d)
     if ( psr_cmt_enabled() )
         psr_assoc_rmid(&reg, d->arch.psr_rmid);
 
+    if ( psra->cos_mask )
+        psr_assoc_cos(&reg, d->arch.psr_cos_ids ?
+                      d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
+                      0, psra->cos_mask);
+
     if ( reg != psra->val )
     {
         wrmsrl(MSR_IA32_PSR_ASSOC, reg);
@@ -401,14 +443,32 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
     return 0;
 }
 
+/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
+static void psr_free_cos(struct domain *d)
+{
+    if( !d->arch.psr_cos_ids )
+        return;
+
+    xfree(d->arch.psr_cos_ids);
+    d->arch.psr_cos_ids = NULL;
+}
+
 int psr_domain_init(struct domain *d)
 {
+    if ( socket_info )
+    {
+        d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
+        if ( !d->arch.psr_cos_ids )
+            return -ENOMEM;
+    }
+
     return 0;
 }
 
 void psr_domain_free(struct domain *d)
 {
     psr_free_rmid(d);
+    psr_free_cos(d);
 }
 
 static void cpu_init_work(void)
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 06/24] x86: refactor psr: implement get hw info flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (4 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-31 20:17   ` Konrad Rzeszutek Wilk
  2017-01-19  6:01 ` [PATCH RESEND v5 07/24] x86: refactor psr: implement get value flow Yi Sun
                   ` (17 subsequent siblings)
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements get HW info flow including L3 CAT callback
function.

It also changes sysctl interface to make it more general.

With this patch, 'psr-hwinfo' can work for L3 CAT.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - add function 'psr_cbm_type_to_feat_type' to covert 'cbm_type' to
      'psr_feat_type'. This is part of codes to move type check out from
      callback functions.
    - remove type check from feature callback functions.
    - rename 'dat[]' to 'data[]'
    - check if feature type match in caller of feature callback function.
---
 xen/arch/x86/psr.c        | 73 +++++++++++++++++++++++++++++++++++++++++++++--
 xen/arch/x86/sysctl.c     | 14 +++++----
 xen/include/asm-x86/psr.h |  9 ++++--
 3 files changed, 86 insertions(+), 10 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 7f06235..319bfcc 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -109,6 +109,9 @@ struct feat_node;
 struct feat_ops {
     /* get_cos_max is used to get feature's cos_max. */
     unsigned int (*get_cos_max)(const struct feat_node *feat);
+    /* get_feat_info is used to get feature HW info. */
+    bool (*get_feat_info)(const struct feat_node *feat,
+                          uint32_t data[], unsigned int array_len);
 };
 
 /*
@@ -177,6 +180,23 @@ static void free_feature(struct psr_socket_info *info)
     }
 }
 
+static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
+{
+    enum psr_feat_type feat_type;
+
+    /* Judge if feature is enabled. */
+    switch ( type ) {
+    case PSR_CBM_TYPE_L3:
+        feat_type = PSR_SOCKET_L3_CAT;
+        break;
+    default:
+        feat_type = 0xFFFF;
+        break;
+    }
+
+    return feat_type;
+}
+
 /* L3 CAT functions implementation. */
 static void l3_cat_init_feature(struct cpuid_leaf_regs regs,
                                 struct feat_node *feat,
@@ -218,8 +238,22 @@ static unsigned int l3_cat_get_cos_max(const struct feat_node *feat)
     return feat->info.l3_cat_info.cos_max;
 }
 
+static bool l3_cat_get_feat_info(const struct feat_node *feat,
+                                 uint32_t data[], unsigned int array_len)
+{
+    if ( !data || 3 > array_len )
+        return false;
+
+    data[CBM_LEN] = feat->info.l3_cat_info.cbm_len;
+    data[COS_MAX] = feat->info.l3_cat_info.cos_max;
+    data[PSR_FLAG] = 0;
+
+    return true;
+}
+
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = l3_cat_get_cos_max,
+    .get_feat_info = l3_cat_get_feat_info,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -425,10 +459,43 @@ void psr_ctxt_switch_to(struct domain *d)
     }
 }
 
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
-                        uint32_t *cos_max, uint32_t *flags)
+static struct psr_socket_info *get_socket_info(unsigned int socket)
 {
-    return 0;
+    if ( !socket_info )
+        return ERR_PTR(-ENODEV);
+
+    if ( socket >= nr_sockets )
+        return ERR_PTR(-ENOTSOCK);
+
+    if ( !socket_info[socket].feat_mask )
+        return ERR_PTR(-ENOENT);
+
+    return socket_info + socket;
+}
+
+int psr_get_info(unsigned int socket, enum cbm_type type,
+                 uint32_t data[], unsigned int array_len)
+{
+    const struct psr_socket_info *info = get_socket_info(socket);
+    const struct feat_node *feat;
+    enum psr_feat_type feat_type;
+
+    if ( IS_ERR(info) )
+        return PTR_ERR(info);
+
+    feat_type = psr_cbm_type_to_feat_type(type);
+    list_for_each_entry(feat, &info->feat_list, list)
+    {
+        if ( feat->feature != feat_type )
+            continue;
+
+        if ( feat->ops.get_feat_info(feat, data, array_len) )
+            return 0;
+        else
+            return -EINVAL;
+    }
+
+    return -ENOENT;
 }
 
 int psr_get_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 14e7dc7..d90db78 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -176,15 +176,19 @@ long arch_do_sysctl(
         switch ( sysctl->u.psr_cat_op.cmd )
         {
         case XEN_SYSCTL_PSR_CAT_get_l3_info:
-            ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
-                                      &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
-                                      &sysctl->u.psr_cat_op.u.l3_info.cos_max,
-                                      &sysctl->u.psr_cat_op.u.l3_info.flags);
+        {
+            uint32_t data[3];
+            ret = psr_get_info(sysctl->u.psr_cat_op.target,
+                               PSR_CBM_TYPE_L3, data, 3);
+
+            sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
+            sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
+            sysctl->u.psr_cat_op.u.l3_info.flags   = data[PSR_FLAG];
 
             if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
                 ret = -EFAULT;
             break;
-
+        }
         default:
             ret = -EOPNOTSUPP;
             break;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 57f47e9..e3b18bc 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -33,6 +33,11 @@
 /* L3 CDP Enable bit*/
 #define PSR_L3_QOS_CDP_ENABLE_BIT       0x0
 
+/* Used by psr_get_info() */
+#define CBM_LEN  0
+#define COS_MAX  1
+#define PSR_FLAG 2
+
 struct psr_cmt_l3 {
     unsigned int features;
     unsigned int upscaling_factor;
@@ -63,8 +68,8 @@ int psr_alloc_rmid(struct domain *d);
 void psr_free_rmid(struct domain *d);
 void psr_ctxt_switch_to(struct domain *d);
 
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
-                        uint32_t *cos_max, uint32_t *flags);
+int psr_get_info(unsigned int socket, enum cbm_type type,
+                 uint32_t data[], unsigned int array_len);
 int psr_get_l3_cbm(struct domain *d, unsigned int socket,
                    uint64_t *cbm, enum cbm_type type);
 int psr_set_l3_cbm(struct domain *d, unsigned int socket,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 07/24] x86: refactor psr: implement get value flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (5 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 06/24] x86: refactor psr: implement get hw info flow Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-31 20:29   ` Konrad Rzeszutek Wilk
  2017-01-19  6:01 ` [PATCH RESEND v5 08/24] x86: refactor psr: set value: implement framework Yi Sun
                   ` (16 subsequent siblings)
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements get value flow including L3 CAT callback
function.

It also changes domctl interface to make it more general.

With this patch, 'psr-cat-show' can work for L3 CAT.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - rename 'dat[]' to 'data[]'
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
    - check if feature type match in caller of feature callback function.
---
 xen/arch/x86/domctl.c     | 18 +++++++++---------
 xen/arch/x86/psr.c        | 41 ++++++++++++++++++++++++++++++++++++++---
 xen/include/asm-x86/psr.h |  4 ++--
 3 files changed, 49 insertions(+), 14 deletions(-)

diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index ab141b1..11d2127 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1383,23 +1383,23 @@ long arch_do_domctl(
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
-            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 &domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3);
+            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+                              &domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3);
             copyback = 1;
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
-            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 &domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3_CODE);
+            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+                              &domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3_CODE);
             copyback = 1;
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
-            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 &domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3_DATA);
+            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+                              &domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3_DATA);
             copyback = 1;
             break;
 
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 319bfcc..3cbb60c 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -112,6 +112,9 @@ struct feat_ops {
     /* get_feat_info is used to get feature HW info. */
     bool (*get_feat_info)(const struct feat_node *feat,
                           uint32_t data[], unsigned int array_len);
+    /* get_val is used to get feature COS register value. */
+    bool (*get_val)(const struct feat_node *feat, unsigned int cos,
+                    enum cbm_type type, uint64_t *val);
 };
 
 /*
@@ -251,9 +254,22 @@ static bool l3_cat_get_feat_info(const struct feat_node *feat,
     return true;
 }
 
+static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
+                           enum cbm_type type, uint64_t *val)
+{
+    if ( cos > feat->info.l3_cat_info.cos_max )
+        /* Use default value. */
+        cos = 0;
+
+    *val =  feat->cos_reg_val[cos];
+
+    return true;
+}
+
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = l3_cat_get_cos_max,
     .get_feat_info = l3_cat_get_feat_info,
+    .get_val = l3_cat_get_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -498,10 +514,29 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
     return -ENOENT;
 }
 
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
-                   uint64_t *cbm, enum cbm_type type)
+int psr_get_val(struct domain *d, unsigned int socket,
+                uint64_t *val, enum cbm_type type)
 {
-    return 0;
+    const struct psr_socket_info *info = get_socket_info(socket);
+    unsigned int cos = d->arch.psr_cos_ids[socket];
+    const struct feat_node *feat;
+    enum psr_feat_type feat_type;
+
+    if ( IS_ERR(info) )
+        return PTR_ERR(info);
+
+    feat_type = psr_cbm_type_to_feat_type(type);
+    list_for_each_entry(feat, &info->feat_list, list)
+    {
+        if ( feat->feature != feat_type )
+            continue;
+
+        if ( feat->ops.get_val(feat, cos, type, val) )
+            /* Found */
+            return 0;
+    }
+
+    return -ENOENT;
 }
 
 int psr_set_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index e3b18bc..d50e359 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -70,8 +70,8 @@ void psr_ctxt_switch_to(struct domain *d);
 
 int psr_get_info(unsigned int socket, enum cbm_type type,
                  uint32_t data[], unsigned int array_len);
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
-                   uint64_t *cbm, enum cbm_type type);
+int psr_get_val(struct domain *d, unsigned int socket,
+                uint64_t *val, enum cbm_type type);
 int psr_set_l3_cbm(struct domain *d, unsigned int socket,
                    uint64_t cbm, enum cbm_type type);
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 08/24] x86: refactor psr: set value: implement framework.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (6 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 07/24] x86: refactor psr: implement get value flow Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
                   ` (15 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

As set value flow is the most complicated one in psr, it will be
divided to some patches to make things clearer. This patch
implements the set value framework to show a whole picture firstly.

It also changes domctl interface to make it more general.

To make the set value flow be general and can support multiple features
at same time, it includes below steps:
1. Get COS ID of current domain using.
2. Assemble a value array to store all features current value
   in it and replace the current value of the feature which is
   being set to the new input value.
3. Find if there is already a COS ID on which all features'
   values are same as the array. Then, we can reuse this COS
   ID.
4. If fail to find, we need pick an available COS ID. Only COS ID which ref
   is 0 or 1 can be picked.
5. Write all features MSRs according to the COS ID.
6. Update ref according to COS ID.
7. Save the COS ID into current domain's psr_cos_ids[socket] so that we
   can know which COS the domain is using on the socket.

So, some functions are abstracted and the callback functions will be
implemented in next patches.

Here is an example to understand the process. The CPU supports
two featuers, e.g. L3 CAT and L2 CAT. user wants to set L3 CAT
of Dom1 to 0x1ff.
1. Get the old_cos of Dom1 which is 0. L3 CAT is the first
element of feature list. The COS registers values are below at
this time.
        -------------------------------
        | COS 0 | COS 1 | COS 2 | ... |
        -------------------------------
L3 CAT  | 0x7ff | ...   | ...   | ... |
        -------------------------------
L2 CAT  | 0xff  | ...   | ...   | ... |
        -------------------------------

2. Assemble The value array to be:
val[0]: 0x1ff
val[1]: 0xff

3. It cannot find a matching COS.

4. Allocate COS 1 to store the value set.

5. Write the COS 1 registers. The COS registers values are
changed to below now.
        -------------------------------
        | COS 0 | COS 1 | COS 2 | ... |
        -------------------------------
L3 CAT  | 0x7ff | 0x1ff | ...   | ... |
        -------------------------------
L2 CAT  | 0xff  | 0xff  | ...   | ... |
        -------------------------------

6. The ref[1] is increased to 1 because Dom1 is using it now.

7. Save 1 to Dom1's psr_cos_ids[socket].

Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
of Dom2 is 0 too. Repeat above flow.

The val array assembled is:
val[0]: 0x1ff
val[1]: 0xff

So, it can find a matching COS, COS 1. Then, it can reuse COS 1
for Dom2.

The ref[1] is increased to 2 now because both Dom1 and Dom2 are
using this COS ID. Set 1 to Dom2's psr_cos_ids[socket].

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - modify commit message because of function name change.
    - change 'alloc_new_cos' to 'pick_avail_cos' to make name accurate.
    - divide 'get_old_set_new' to two functions, 'assemble_val_array' and
      'set_new_val_to_array'.
    - check feature type when entering 'psr_set_val'.
    - remove cast.
    - use ASSERT to check ref.
    - rename 'dat[]' to 'data[]'
---
 xen/arch/x86/domctl.c     |  18 ++---
 xen/arch/x86/psr.c        | 202 +++++++++++++++++++++++++++++++++++++++++++++-
 xen/include/asm-x86/psr.h |   4 +-
 3 files changed, 210 insertions(+), 14 deletions(-)

diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 11d2127..db56500 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1365,21 +1365,21 @@ long arch_do_domctl(
         switch ( domctl->u.psr_cat_op.cmd )
         {
         case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
-            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3);
+            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+                              domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3);
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE:
-            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3_CODE);
+            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+                              domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3_CODE);
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA:
-            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3_DATA);
+            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+                              domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3_DATA);
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 3cbb60c..050b0df 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -539,18 +539,214 @@ int psr_get_val(struct domain *d, unsigned int socket,
     return -ENOENT;
 }
 
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
-                   uint64_t cbm, enum cbm_type type)
+/* Set value functions */
+static unsigned int get_cos_num(const struct psr_socket_info *info)
 {
     return 0;
 }
 
+static int assemble_val_array(uint64_t *val,
+                              uint32_t array_len,
+                              const struct psr_socket_info *info,
+                              unsigned int old_cos)
+{
+    return -EINVAL;
+}
+
+static int set_new_val_to_array(uint64_t *val,
+                                uint32_t array_len,
+                                const struct psr_socket_info *info,
+                                enum psr_feat_type feat_type,
+                                enum cbm_type type,
+                                uint64_t m)
+{
+    return -EINVAL;
+}
+
+static int find_cos(const uint64_t *val, uint32_t array_len,
+                    enum psr_feat_type feat_type,
+                    const struct psr_socket_info *info)
+{
+    return -ENOENT;
+}
+
+static int pick_avail_cos(const struct psr_socket_info *info,
+                          const uint64_t *val, uint32_t array_len,
+                          unsigned int old_cos,
+                          enum psr_feat_type feat_type)
+{
+    return -ENOENT;
+}
+
+static int write_psr_msr(unsigned int socket, unsigned int cos,
+                         const uint64_t *val)
+{
+    return -ENOENT;
+}
+
+int psr_set_val(struct domain *d, unsigned int socket,
+                uint64_t val, enum cbm_type type)
+{
+    unsigned int old_cos;
+    int cos, ret;
+    unsigned int *ref;
+    uint64_t *val_array;
+    struct psr_socket_info *info = get_socket_info(socket);
+    uint32_t array_len;
+    enum psr_feat_type feat_type;
+
+    if ( IS_ERR(info) )
+        return PTR_ERR(info);
+
+    feat_type = psr_cbm_type_to_feat_type(type);
+    if ( !test_bit(feat_type, &info->feat_mask) )
+        return -ENOENT;
+
+    /*
+     * Step 0:
+     * old_cos means the COS ID current domain is using. By default, it is 0.
+     *
+     * For every COS ID, there is a reference count to record how many domains
+     * are using the COS register corresponding to this COS ID.
+     * - If ref[old_cos] is 0, that means this COS is not used by any domain.
+     * - If ref[old_cos] is 1, that means this COS is only used by current
+     *   domain.
+     * - If ref[old_cos] is more than 1, that mean multiple domains are using
+     *   this COS.
+     */
+    old_cos = d->arch.psr_cos_ids[socket];
+    if ( old_cos > MAX_COS_REG_CNT )
+        return -EOVERFLOW;
+
+    ref = info->cos_ref;
+
+    /*
+     * Step 1:
+     * Assemle a value array to store all featues cos_reg_val[old_cos].
+     * And, set the input val into array according to the feature's
+     * position in array.
+     */
+    array_len = get_cos_num(info);
+    val_array = xzalloc_array(uint64_t, array_len);
+    if ( !val_array )
+        return -ENOMEM;
+
+    if ( (ret = assemble_val_array(val_array, array_len, info, old_cos)) != 0 )
+    {
+        xfree(val_array);
+        return ret;
+    }
+
+    if ( (ret = set_new_val_to_array(val_array, array_len, info,
+                                     feat_type, type, val)) != 0 )
+    {
+        xfree(val_array);
+        return ret;
+    }
+
+    /*
+     * Lock here to make sure the ref is not changed during find and
+     * write process.
+     */
+    spin_lock(&info->ref_lock);
+
+    /*
+     * Step 2:
+     * Try to find if there is already a COS ID on which all features' values
+     * are same as the array. Then, we can reuse this COS ID.
+     */
+    cos = find_cos(val_array, array_len, feat_type, info);
+    if ( cos >= 0 )
+    {
+        if ( cos == old_cos )
+        {
+            spin_unlock(&info->ref_lock);
+            xfree(val_array);
+            return 0;
+        }
+    }
+    else
+    {
+        /*
+         * Step 3:
+         * If fail to find, we need allocate a new COS ID.
+         * If multiple domains are using same COS ID, its ref is more
+         * than 1. That means we cannot free this COS to make current domain
+         * use it. Because other domains are using the value saved in the COS.
+         * Unless the ref is changed to 1 (mean only current domain is using
+         * it), we cannot allocate the COS ID to current domain.
+         * So, only the COS ID which ref is 1 or 0 can be allocated.
+         */
+        cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
+        if ( cos < 0 )
+        {
+            spin_unlock(&info->ref_lock);
+            xfree(val_array);
+            return cos;
+        }
+
+        /*
+         * Step 4:
+         * Write all features MSRs according to the COS ID.
+         */
+        ret = write_psr_msr(socket, cos, val_array);
+        if ( ret )
+        {
+            spin_unlock(&info->ref_lock);
+            xfree(val_array);
+            return ret;
+        }
+    }
+
+    /*
+     * Step 5:
+     * Update ref according to COS ID.
+     */
+    ref[cos]++;
+    ASSERT(ref[cos] || cos == 0);
+    ref[old_cos]--;
+    spin_unlock(&info->ref_lock);
+
+    /*
+     * Step 6:
+     * Save the COS ID into current domain's psr_cos_ids[] so that we can know
+     * which COS the domain is using on the socket. One domain can only use
+     * one COS ID at same time on each socket.
+     */
+    d->arch.psr_cos_ids[socket] = cos;
+    xfree(val_array);
+
+    return 0;
+}
+
 /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
 static void psr_free_cos(struct domain *d)
 {
-    if( !d->arch.psr_cos_ids )
+    unsigned int socket, cos;
+
+    if ( !d->arch.psr_cos_ids )
         return;
 
+    /* Domain is free so its cos_ref should be decreased. */
+    for ( socket = 0; socket < nr_sockets; socket++ )
+    {
+        struct psr_socket_info *info;
+
+        /* cos 0 is default one which does not need be handled. */
+        if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
+            continue;
+
+        /*
+         * If domain uses other cos ids, all corresponding refs must have been
+         * increased 1 for this domain. So, we need decrease them.
+         */
+        info = socket_info + socket;
+        ASSERT(info->cos_ref[cos] || cos == 0);
+        spin_lock(&info->ref_lock);
+        info->cos_ref[cos]--;
+        spin_unlock(&info->ref_lock);
+    }
+
     xfree(d->arch.psr_cos_ids);
     d->arch.psr_cos_ids = NULL;
 }
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index d50e359..97214fe 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -72,8 +72,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
                  uint32_t data[], unsigned int array_len);
 int psr_get_val(struct domain *d, unsigned int socket,
                 uint64_t *val, enum cbm_type type);
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
-                   uint64_t cbm, enum cbm_type type);
+int psr_set_val(struct domain *d, unsigned int socket,
+                uint64_t val, enum cbm_type type);
 
 int psr_domain_init(struct domain *d);
 void psr_domain_free(struct domain *d);
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 09/24] x86: refactor psr: set value: assemble features value array.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (7 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 08/24] x86: refactor psr: set value: implement framework Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-31 20:57   ` Konrad Rzeszutek Wilk
  2017-01-19  6:01 ` [PATCH RESEND v5 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
                   ` (14 subsequent siblings)
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

Only can one COS ID be used by one domain at one time. That means all enabled
features' COS registers at this COS ID are valid for this domain at that time.

When user updates a feature's value, we need make sure all other features'
values are not affected. So, we firstly need assemble an array which contains
all features current values and replace the setting feature's value in array
to new value.

Then, we can try to find if there is a COS ID on which all features' COS
registers values are same as the array. If we can find, we just use this COS
ID. If fail to find, we need allocate a new COS ID.

This patch implements value array assembling flow.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - modify comments according to changes of codes.
    - change 'bool_t' to 'bool'.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
---
 xen/arch/x86/psr.c | 145 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 142 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 050b0df..7c6f2bf 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -115,6 +115,32 @@ struct feat_ops {
     /* get_val is used to get feature COS register value. */
     bool (*get_val)(const struct feat_node *feat, unsigned int cos,
                     enum cbm_type type, uint64_t *val);
+    /*
+     * get_cos_num is used to get the COS registers amount used by the
+     * feature for one setting, e.g. CDP uses 2 COSs but CAT uses 1.
+     */
+    unsigned int (*get_cos_num)(const struct feat_node *feat);
+    /*
+     * get_old_val and set_new_val are a pair of functions called in order.
+     * The caller will traverse all features in the list and call both
+     * functions for every feature to do below two things:
+     * 1. get old_cos register value of all supported features and
+     * 2. set the new value for the feature.
+     *
+     * All the values are set into value array according the traversal order,
+     * meaning the same order of feature list members.
+     *
+     * The return value meaning:
+     * 0 - success.
+     * negative - error.
+     */
+    int (*get_old_val)(uint64_t val[],
+                       const struct feat_node *feat,
+                       unsigned int old_cos);
+    int (*set_new_val)(uint64_t val[],
+                       const struct feat_node *feat,
+                       enum cbm_type type,
+                       uint64_t m);
 };
 
 /*
@@ -200,6 +226,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
     return feat_type;
 }
 
+static bool psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
+{
+    unsigned int first_bit, zero_bit;
+
+    /* Set bits should only in the range of [0, cbm_len). */
+    if ( cbm & (~0ull << cbm_len) )
+        return false;
+
+    /* At least one bit need to be set. */
+    if ( cbm == 0 )
+        return false;
+
+    first_bit = find_first_bit(&cbm, cbm_len);
+    zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
+
+    /* Set bits should be contiguous. */
+    if ( zero_bit < cbm_len &&
+         find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
+        return false;
+
+    return true;
+}
+
 /* L3 CAT functions implementation. */
 static void l3_cat_init_feature(struct cpuid_leaf_regs regs,
                                 struct feat_node *feat,
@@ -266,10 +315,45 @@ static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
     return true;
 }
 
+static unsigned int l3_cat_get_cos_num(const struct feat_node *feat)
+{
+    return 1;
+}
+
+static int l3_cat_get_old_val(uint64_t val[],
+                              const struct feat_node *feat,
+                              unsigned int old_cos)
+{
+    if ( old_cos > feat->info.l3_cat_info.cos_max )
+        /* Use default value. */
+        old_cos = 0;
+
+    /* CAT */
+    val[0] =  feat->cos_reg_val[old_cos];
+
+    return 0;
+}
+
+static int l3_cat_set_new_val(uint64_t val[],
+                              const struct feat_node *feat,
+                              enum cbm_type type,
+                              uint64_t m)
+{
+    if ( !psr_check_cbm(feat->info.l3_cat_info.cbm_len, m) )
+        return -EINVAL;
+
+    val[0] = m;
+
+    return 0;
+}
+
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = l3_cat_get_cos_max,
     .get_feat_info = l3_cat_get_feat_info,
     .get_val = l3_cat_get_val,
+    .get_cos_num = l3_cat_get_cos_num,
+    .get_old_val = l3_cat_get_old_val,
+    .set_new_val = l3_cat_set_new_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -542,7 +626,14 @@ int psr_get_val(struct domain *d, unsigned int socket,
 /* Set value functions */
 static unsigned int get_cos_num(const struct psr_socket_info *info)
 {
-    return 0;
+    const struct feat_node *feat_tmp;
+    unsigned int num = 0;
+
+    /* Get all features total amount. */
+    list_for_each_entry(feat_tmp, &info->feat_list, list)
+        num += feat_tmp->ops.get_cos_num(feat_tmp);
+
+    return num;
 }
 
 static int assemble_val_array(uint64_t *val,
@@ -550,7 +641,25 @@ static int assemble_val_array(uint64_t *val,
                               const struct psr_socket_info *info,
                               unsigned int old_cos)
 {
-    return -EINVAL;
+    const struct feat_node *feat;
+    int ret;
+    uint64_t *val_tmp = val;
+
+    if ( !val )
+        return -EINVAL;
+
+    /* Get all features current values according to old_cos. */
+    list_for_each_entry(feat, &info->feat_list, list)
+    {
+        /* value getting order is same as feature list */
+        ret = feat->ops.get_old_val(val_tmp, feat, old_cos);
+
+        val_tmp += feat->ops.get_cos_num(feat);
+        if ( val_tmp - val > array_len)
+            return -EINVAL;
+    }
+
+    return 0;
 }
 
 static int set_new_val_to_array(uint64_t *val,
@@ -560,7 +669,37 @@ static int set_new_val_to_array(uint64_t *val,
                                 enum cbm_type type,
                                 uint64_t m)
 {
-    return -EINVAL;
+    const struct feat_node *feat;
+    int ret;
+    uint64_t *val_tmp = val;
+
+    /* Set new value into array according to feature's position in array. */
+    list_for_each_entry(feat, &info->feat_list, list)
+    {
+        if ( feat->feature != feat_type )
+        {
+            val_tmp += feat->ops.get_cos_num(feat);
+            if ( val_tmp - val > array_len)
+                return -EINVAL;
+
+            continue;
+        }
+
+        /*
+         * Value setting position is same as feature list.
+         * Different features may have different setting behaviors, e.g. CDP
+         * has two values (DATA/CODE) which need us to save input value to
+         * different position in the array according to type, so we have to
+         * maintain a callback function.
+         */
+        ret = feat->ops.set_new_val(val_tmp, feat, type, m);
+        if ( ret )
+            return ret;
+        else
+            break;
+    }
+
+    return 0;
 }
 
 static int find_cos(const uint64_t *val, uint32_t array_len,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 10/24] x86: refactor psr: set value: implement cos finding flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (8 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
                   ` (13 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

Continue with patch:
'x86: refactor psr: set value: assemble features value array'

We can try to find if there is a COS ID on which all features' COS registers
values are same as the array assembled before.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - modify commit message to provide exact patch name to continue from.
    - remove 'get_cos_max_from_type' because it can be replaced by
      'get_cos_max'.
    - move type check out from callback functions to caller.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
    - modify comments according to changes of codes.
---
 xen/arch/x86/psr.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 7c6f2bf..8832e08 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -141,6 +141,19 @@ struct feat_ops {
                        const struct feat_node *feat,
                        enum cbm_type type,
                        uint64_t m);
+    /*
+     * compare_val is used in set value process to compare if the
+     * input value array can match all the features' COS registers values
+     * according to input cos id.
+     *
+     * The return value is the amount of entries to skip in the value array
+     * or error.
+     * 1 - one entry in value array.
+     * 2 - two entries in value array, e.g. CDP uses two entries.
+     * negative - error.
+     */
+    int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
+                        unsigned int cos, bool *found);
 };
 
 /*
@@ -347,6 +360,34 @@ static int l3_cat_set_new_val(uint64_t val[],
     return 0;
 }
 
+static int l3_cat_compare_val(const uint64_t val[],
+                              const struct feat_node *feat,
+                              unsigned int cos, bool *found)
+{
+    uint64_t l3_def_cbm;
+
+    l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
+
+    /*
+     * Different features' cos_max are different. If cos id of the feature
+     * being set exceeds other feature's cos_max, the val of other feature
+     * must be default value. HW supports such case.
+     */
+    if ( cos > feat->info.l3_cat_info.cos_max )
+    {
+        if ( val[0] != l3_def_cbm )
+        {
+            *found = false;
+            return -ENOENT;
+        }
+        *found = true;
+    }
+    else
+        *found = (val[0] == feat->cos_reg_val[cos]);
+
+    return 0;
+}
+
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = l3_cat_get_cos_max,
     .get_feat_info = l3_cat_get_feat_info,
@@ -354,6 +395,7 @@ static const struct feat_ops l3_cat_ops = {
     .get_cos_num = l3_cat_get_cos_num,
     .get_old_val = l3_cat_get_old_val,
     .set_new_val = l3_cat_set_new_val,
+    .compare_val = l3_cat_compare_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -706,6 +748,57 @@ static int find_cos(const uint64_t *val, uint32_t array_len,
                     enum psr_feat_type feat_type,
                     const struct psr_socket_info *info)
 {
+    unsigned int cos;
+    const unsigned int *ref = info->cos_ref;
+    const struct feat_node *feat;
+    const uint64_t *val_tmp = val;
+    int ret;
+    bool found = false;
+    unsigned int cos_max = 0;
+
+    /* cos_max is the one of the feature which is being set. */
+    list_for_each_entry(feat, &info->feat_list, list)
+    {
+        if ( feat->feature != feat_type )
+            continue;
+
+        cos_max = feat->ops.get_cos_max(feat);
+        if ( cos_max > 0 )
+            break;
+    }
+
+    for ( cos = 0; cos <= cos_max; cos++ )
+    {
+        if ( cos && !ref[cos] )
+            continue;
+
+        /* Not found, need find again from beginning. */
+        val_tmp = val;
+        list_for_each_entry(feat, &info->feat_list, list)
+        {
+            /*
+             * Compare value according to feature list order.
+             * We must follow this order because value array is assembled
+             * as this order in get_old_set_new().
+             */
+            ret = feat->ops.compare_val(val_tmp, feat, cos, &found);
+            if ( ret < 0 )
+                return ret;
+
+            /* If fail to match, go to next cos to compare. */
+            if ( !found )
+                break;
+
+            val_tmp += feat->ops.get_cos_num(feat);
+            if ( val_tmp - val > array_len )
+                return -EINVAL;
+        }
+
+        /* For this COS ID all entries in the values array did match. Use it. */
+        if ( found )
+            return cos;
+    }
+
     return -ENOENT;
 }
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 11/24] x86: refactor psr: set value: implement cos id picking flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (9 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 12/24] x86: refactor psr: set value: implement write msr flow Yi Sun
                   ` (12 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

Continue with previous patch:
'x86: refactor psr: set value: implement cos finding flow.'

If fail to find a COS ID, we need pick a new COS ID for domain. Only COS ID
that ref[COS_ID] is 1 or 0 can be picked to input a new set feature values.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - modify commit message to provide exact patch name to continue from.
    - change 'exceeds_cos_max' to 'fits_cos_max' to be accurate.
    - modify comments according to changes of codes.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - move type check out from callback functions to caller.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
---
 xen/arch/x86/psr.c | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 8832e08..c3e25bf 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -154,6 +154,17 @@ struct feat_ops {
      */
     int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
                         unsigned int cos, bool *found);
+    /*
+     * fits_cos_max is used to check if the input cos id exceeds the
+     * feature's cos_max and if the input value is not the default one.
+     * Even if the associated cos exceeds the cos_max, HW can work with default
+     * value. That is the reason we need check if input value is default one.
+     * If both criteria are fulfilled, that means the input exceeds the range.
+     * If not, that means the input fits the requirements.
+     */
+    bool (*fits_cos_max)(const uint64_t val[],
+                         const struct feat_node *feat,
+                         unsigned int cos);
 };
 
 /*
@@ -388,6 +399,25 @@ static int l3_cat_compare_val(const uint64_t val[],
     return 0;
 }
 
+static bool l3_cat_fits_cos_max(const uint64_t val[],
+                                const struct feat_node *feat,
+                                unsigned int cos)
+{
+    uint64_t l3_def_cbm;
+
+    l3_def_cbm = (1ull << feat->info.l3_cat_info.cbm_len) - 1;
+
+    if ( cos > feat->info.l3_cat_info.cos_max &&
+         val[0] != l3_def_cbm )
+            /*
+             * Exceed cos_max and value to set is not default,
+             * return error.
+             */
+            return false;
+
+    return true;
+}
+
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = l3_cat_get_cos_max,
     .get_feat_info = l3_cat_get_feat_info,
@@ -396,6 +426,7 @@ static const struct feat_ops l3_cat_ops = {
     .get_old_val = l3_cat_get_old_val,
     .set_new_val = l3_cat_set_new_val,
     .compare_val = l3_cat_compare_val,
+    .fits_cos_max = l3_cat_fits_cos_max,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -802,11 +833,79 @@ static int find_cos(const uint64_t *val, uint32_t array_len,
     return -ENOENT;
 }
 
+static bool fits_cos_max(const uint64_t *val,
+                         uint32_t array_len,
+                         const struct psr_socket_info *info,
+                         unsigned int cos)
+{
+    unsigned int ret;
+    const uint64_t *val_tmp = val;
+    const struct feat_node *feat;
+
+    list_for_each_entry(feat, &info->feat_list, list)
+    {
+        ret = feat->ops.fits_cos_max(val_tmp, feat, cos);
+        if ( !ret )
+            return false;
+
+        val_tmp += feat->ops.get_cos_num(feat);
+        if ( val_tmp - val > array_len )
+            return false;
+    }
+
+    return true;
+}
+
 static int pick_avail_cos(const struct psr_socket_info *info,
                           const uint64_t *val, uint32_t array_len,
                           unsigned int old_cos,
                           enum psr_feat_type feat_type)
 {
+    unsigned int cos;
+    unsigned int cos_max = 0;
+    const struct feat_node *feat;
+    const unsigned int *ref = info->cos_ref;
+
+    /*
+     * cos_max is the one of the feature which is being set.
+     */
+    list_for_each_entry(feat, &info->feat_list, list)
+    {
+        if ( feat->feature != feat_type )
+            continue;
+
+        cos_max = feat->ops.get_cos_max(feat);
+        if ( cos_max > 0 )
+            break;
+    }
+
+    if ( !cos_max )
+        return -ENOENT;
+
+    /*
+     * If old cos is referred only by the domain, then use it. And, we cannot
+     * use id 0 because it stores the default values.
+     */
+    if ( old_cos && ref[old_cos] == 1 &&
+         fits_cos_max(val, array_len, info, old_cos) )
+            return old_cos;
+
+    /* Find an unused one other than cos0. */
+    for ( cos = 1; cos <= cos_max; cos++ )
+    {
+        /*
+         * ref is 0 means this COS is not used by other domain and
+         * can be used for current setting.
+         */
+        if ( !ref[cos] )
+        {
+            if ( !fits_cos_max(val, array_len, info, cos) )
+                return -ENOENT;
+
+            return cos;
+        }
+    }
+
     return -ENOENT;
 }
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 12/24] x86: refactor psr: set value: implement write msr flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (10 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
                   ` (11 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

Continue with previous patch:
'x86: refactor psr: set value: implement cos id picking flow.'

We have got all features values and COS ID to set. Then, we write MSRs of all
features except the setting value is same as original value.

Till now, set value process is completed.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - modify commit message to provide exact patch name to continue from.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - move type check out from callback functions to caller.
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
---
 xen/arch/x86/psr.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 77 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index c3e25bf..b8d3c82 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -165,6 +165,9 @@ struct feat_ops {
     bool (*fits_cos_max)(const uint64_t val[],
                          const struct feat_node *feat,
                          unsigned int cos);
+    /* write_msr is used to write out feature MSR register. */
+    int (*write_msr)(unsigned int cos, const uint64_t val[],
+                     struct feat_node *feat);
 };
 
 /*
@@ -418,6 +421,21 @@ static bool l3_cat_fits_cos_max(const uint64_t val[],
     return true;
 }
 
+static int l3_cat_write_msr(unsigned int cos, const uint64_t val[],
+                            struct feat_node *feat)
+{
+    if ( cos > feat->info.l3_cat_info.cos_max )
+        return -EINVAL;
+
+    if ( feat->cos_reg_val[cos] != val[0] )
+    {
+        feat->cos_reg_val[cos] = val[0];
+        wrmsrl(MSR_IA32_PSR_L3_MASK(cos), val[0]);
+    }
+
+    return 0;
+}
+
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = l3_cat_get_cos_max,
     .get_feat_info = l3_cat_get_feat_info,
@@ -427,6 +445,7 @@ static const struct feat_ops l3_cat_ops = {
     .set_new_val = l3_cat_set_new_val,
     .compare_val = l3_cat_compare_val,
     .fits_cos_max = l3_cat_fits_cos_max,
+    .write_msr = l3_cat_write_msr,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -909,10 +928,67 @@ static int pick_avail_cos(const struct psr_socket_info *info,
     return -ENOENT;
 }
 
+static unsigned int get_socket_cpu(unsigned int socket)
+{
+    if ( likely(socket < nr_sockets) )
+        return cpumask_any(socket_cpumask[socket]);
+
+    return nr_cpu_ids;
+}
+
+struct cos_write_info
+{
+    unsigned int cos;
+    struct list_head *feat_list;
+    const uint64_t *val;
+};
+
+static void do_write_psr_msr(void *data)
+{
+    struct cos_write_info *info = (struct cos_write_info *)data;
+    unsigned int cos           = info->cos;
+    struct list_head *feat_list= info->feat_list;
+    const uint64_t *val        = info->val;
+    struct feat_node *feat;
+    int ret;
+
+    if ( !feat_list )
+        return;
+
+    /* We need set all features values into MSRs. */
+    list_for_each_entry(feat, feat_list, list)
+    {
+        ret = feat->ops.write_msr(cos, val, feat);
+        if ( ret < 0 )
+            return;
+
+        val += feat->ops.get_cos_num(feat);
+    }
+}
+
 static int write_psr_msr(unsigned int socket, unsigned int cos,
                          const uint64_t *val)
 {
-    return -ENOENT;
+    struct psr_socket_info *info = get_socket_info(socket);
+    struct cos_write_info data =
+    {
+        .cos = cos,
+        .feat_list = &info->feat_list,
+        .val = val,
+    };
+
+    if ( socket == cpu_to_socket(smp_processor_id()) )
+        do_write_psr_msr(&data);
+    else
+    {
+        unsigned int cpu = get_socket_cpu(socket);
+
+        if ( cpu >= nr_cpu_ids )
+            return -ENOTSOCK;
+        on_selected_cpus(cpumask_of(cpu), do_write_psr_msr, &data, 1);
+    }
+
+    return 0;
 }
 
 int psr_set_val(struct domain *d, unsigned int socket,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 13/24] x86: refactor psr: implement CPU init and free flow for CDP.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (11 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 12/24] x86: refactor psr: set value: implement write msr flow Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 14/24] x86: refactor psr: implement get hw info " Yi Sun
                   ` (10 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements the CPU init and free flow for CDP including L3 CDP
initialization callback function.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - remove 'feat_l3_cdp' free in 'free_feature'.
    - Encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
    - Print socket info when 'opt_cpu_info' is true.
    - rename 'l3_cdp_get_max_cos_max' to 'l3_cdp_get_cos_max'.
    - rename 'dat[]' to 'data[]'
    - move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
---
 xen/arch/x86/psr.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 93 insertions(+), 5 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index b8d3c82..a979128 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -93,6 +93,7 @@ struct psr_cat_hw_info {
 struct feat_hw_info {
     union {
         struct psr_cat_hw_info l3_cat_info;
+        struct psr_cat_hw_info l3_cdp_info;
     };
 };
 
@@ -197,6 +198,21 @@ struct cpuid_leaf_regs {
     unsigned int ecx;
     unsigned int edx;
 };
+/*
+ * get_data - get DATA COS register value from input COS ID.
+ * @feat:        the feature list entry.
+ * @cos:         the COS ID.
+ */
+#define get_cdp_data(feat, cos)                  \
+            ( feat->cos_reg_val[cos * 2] )
+
+/*
+ * get_cdp_code - get CODE COS register value from input COS ID.
+ * @feat:        the feature list entry.
+ * @cos:         the COS ID.
+ */
+#define get_cdp_code(feat, cos)                  \
+            ( feat->cos_reg_val[cos * 2 + 1] )
 
 struct psr_assoc {
     uint64_t val;
@@ -219,6 +235,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
  * inserted into feature list in cpu_init_work().
  */
 static struct feat_node *feat_l3_cat;
+static struct feat_node *feat_l3_cdp;
 
 /* Common functions. */
 static void free_feature(struct psr_socket_info *info)
@@ -448,6 +465,61 @@ static const struct feat_ops l3_cat_ops = {
     .write_msr = l3_cat_write_msr,
 };
 
+/* L3 CDP functions implementation. */
+static void l3_cdp_init_feature(struct cpuid_leaf_regs regs,
+                                struct feat_node *feat,
+                                struct psr_socket_info *info)
+{
+    struct psr_cat_hw_info l3_cdp;
+    unsigned int socket;
+    uint64_t val;
+
+    /* No valid value so do not enable feature. */
+    if ( !regs.eax || !regs.edx )
+        return;
+
+    l3_cdp.cbm_len = (regs.eax & CAT_CBM_LEN_MASK) + 1;
+    /* Cut half of cos_max when CDP is enabled. */
+    l3_cdp.cos_max = min(opt_cos_max, regs.edx & CAT_COS_MAX_MASK) >> 1;
+
+    /* cos=0 is reserved as default cbm(all ones). */
+    get_cdp_code(feat, 0) =
+                 (1ull << l3_cdp.cbm_len) - 1;
+    get_cdp_data(feat, 0) =
+                 (1ull << l3_cdp.cbm_len) - 1;
+
+    /* We only write mask1 since mask0 is always all ones by default. */
+    wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << l3_cdp.cbm_len) - 1);
+    rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
+    wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
+
+    feat->feature = PSR_SOCKET_L3_CDP;
+    __set_bit(PSR_SOCKET_L3_CDP, &info->feat_mask);
+
+    feat->info.l3_cdp_info = l3_cdp;
+
+    info->nr_feat++;
+
+    /* Add this feature into list. */
+    list_add_tail(&feat->list, &info->feat_list);
+
+    socket = cpu_to_socket(smp_processor_id());
+    if ( opt_cpu_info )
+        printk(XENLOG_INFO
+           "L3 CDP: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+           socket, feat->info.l3_cdp_info.cos_max,
+           feat->info.l3_cdp_info.cbm_len);
+}
+
+static unsigned int l3_cdp_get_cos_max(const struct feat_node *feat)
+{
+    return feat->info.l3_cdp_info.cos_max;
+}
+
+struct feat_ops l3_cdp_ops = {
+    .get_cos_max = l3_cdp_get_cos_max,
+};
+
 static void __init parse_psr_bool(char *s, char *value, char *feature,
                                   unsigned int mask)
 {
@@ -1207,11 +1279,19 @@ static void cpu_init_work(void)
         cpuid_count(PSR_CPUID_LEVEL_CAT, 1,
                     &regs.eax, &regs.ebx, &regs.ecx, &regs.edx);
 
-        feat = feat_l3_cat;
-        feat_l3_cat = NULL;
-        feat->ops = l3_cat_ops;
-
-        l3_cat_init_feature(regs, feat, info);
+        if ( (regs.ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
+             !test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) )
+        {
+            feat = feat_l3_cdp;
+            feat_l3_cdp = NULL;
+            feat->ops = l3_cdp_ops;
+            l3_cdp_init_feature(regs, feat, info);
+        } else {
+            feat = feat_l3_cat;
+            feat_l3_cat = NULL;
+            feat->ops = l3_cat_ops;
+            l3_cat_init_feature(regs, feat, info);
+        }
     }
 }
 
@@ -1263,6 +1343,14 @@ static int psr_cpu_prepare(unsigned int cpu)
          (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
         return -ENOMEM;
 
+    if ( feat_l3_cdp == NULL &&
+         (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
+    {
+        xfree(feat_l3_cat);
+        feat_l3_cat = NULL;
+        return -ENOMEM;
+    }
+
     return 0;
 }
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 14/24] x86: refactor psr: implement get hw info flow for CDP.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (12 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 15/24] x86: refactor psr: implement get value " Yi Sun
                   ` (9 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements get HW info flow for CDP including L3 CDP callback
function.

It also changes sysctl function to make it work for CDP.

With this patch, 'psr-hwinfo' can work for L3 CDP.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - rename 'dat[]' to 'data[]'
    - remove type check in callback function.
---
 xen/arch/x86/psr.c    | 18 ++++++++++++++++++
 xen/arch/x86/sysctl.c | 24 +++++++++++++++++++++---
 2 files changed, 39 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index a979128..b856761 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -262,6 +262,10 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
     case PSR_CBM_TYPE_L3:
         feat_type = PSR_SOCKET_L3_CAT;
         break;
+    case PSR_CBM_TYPE_L3_DATA:
+    case PSR_CBM_TYPE_L3_CODE:
+        feat_type = PSR_SOCKET_L3_CDP;
+        break;
     default:
         feat_type = 0xFFFF;
         break;
@@ -516,8 +520,22 @@ static unsigned int l3_cdp_get_cos_max(const struct feat_node *feat)
     return feat->info.l3_cdp_info.cos_max;
 }
 
+static bool l3_cdp_get_feat_info(const struct feat_node *feat,
+                                 uint32_t data[], uint32_t array_len)
+{
+    if ( !data || 3 > array_len )
+        return false;
+
+    data[CBM_LEN] = feat->info.l3_cdp_info.cbm_len;
+    data[COS_MAX] = feat->info.l3_cdp_info.cos_max;
+    data[PSR_FLAG] |= XEN_SYSCTL_PSR_CAT_L3_CDP;
+
+    return true;
+}
+
 struct feat_ops l3_cdp_ops = {
     .get_cos_max = l3_cdp_get_cos_max,
+    .get_feat_info = l3_cdp_get_feat_info,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index d90db78..a4c8cfe 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -181,9 +181,27 @@ long arch_do_sysctl(
             ret = psr_get_info(sysctl->u.psr_cat_op.target,
                                PSR_CBM_TYPE_L3, data, 3);
 
-            sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
-            sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
-            sysctl->u.psr_cat_op.u.l3_info.flags   = data[PSR_FLAG];
+            if ( !ret )
+            {
+                sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
+                sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
+                sysctl->u.psr_cat_op.u.l3_info.flags   = data[PSR_FLAG];
+            } else {
+                /*
+                 * Check if CDP is enabled.
+                 *
+                 * Per spec, L3 CAT and CDP cannot co-exist. So, we need replace
+                 * output values to CDP's if it is enabled.
+                 */
+                ret = psr_get_info(sysctl->u.psr_cat_op.target,
+                               PSR_CBM_TYPE_L3_CODE, data, 3);
+                if ( !ret )
+                {
+                    sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
+                    sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
+                    sysctl->u.psr_cat_op.u.l3_info.flags   = data[PSR_FLAG];
+                }
+            }
 
             if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
                 ret = -EFAULT;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 15/24] x86: refactor psr: implement get value flow for CDP.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (13 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 14/24] x86: refactor psr: implement get hw info " Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 16/24] x86: refactor psr: implement set value callback functions " Yi Sun
                   ` (8 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements L3 CDP get value callback function.

With this patch, 'psr-cat-show' can work for L3 CDP.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - remove type check in callback function.
---
 xen/arch/x86/psr.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index b856761..dc062ff 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -533,9 +533,25 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
     return true;
 }
 
+static bool l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
+                           enum cbm_type type, uint64_t *val)
+{
+    if ( cos > feat->info.l3_cdp_info.cos_max )
+        /* Use default value. */
+        cos = 0;
+
+    if ( type == PSR_CBM_TYPE_L3_DATA )
+        *val = get_cdp_data(feat, cos);
+    else
+        *val = get_cdp_code(feat, cos);
+
+    return true;
+}
+
 struct feat_ops l3_cdp_ops = {
     .get_cos_max = l3_cdp_get_cos_max,
     .get_feat_info = l3_cdp_get_feat_info,
+    .get_val = l3_cdp_get_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 16/24] x86: refactor psr: implement set value callback functions for CDP.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (14 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 15/24] x86: refactor psr: implement get value " Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
                   ` (7 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements L3 CDP set value related callback functions.

With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - remove type check in callback function.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - remove 'l3_cdp_get_cos_max_from_type'.
    - rename 'l3_cdp_exceeds_cos_max' to 'l3_cdp_fits_cos_max'.
---
 xen/arch/x86/psr.c | 118 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 118 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index dc062ff..596e5b1 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -548,10 +548,128 @@ static bool l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
     return true;
 }
 
+static unsigned int l3_cdp_get_cos_num(const struct feat_node *feat)
+{
+    return 2;
+}
+
+static int l3_cdp_get_old_val(uint64_t val[],
+                              const struct feat_node *feat,
+                              unsigned int old_cos)
+{
+    if ( old_cos > feat->info.l3_cdp_info.cos_max )
+        /* Use default value. */
+        old_cos = 0;
+
+    /* Data */
+    val[0] = get_cdp_data(feat, old_cos);
+    /* Code */
+    val[1] = get_cdp_code(feat, old_cos);
+
+    return 0;
+}
+
+static int l3_cdp_set_new_val(uint64_t val[],
+                              const struct feat_node *feat,
+                              enum cbm_type type,
+                              uint64_t m)
+{
+    if ( !psr_check_cbm(feat->info.l3_cdp_info.cbm_len, m) )
+        return -EINVAL;
+
+    if ( type == PSR_CBM_TYPE_L3_DATA )
+        val[0] = m;
+    else
+        val[1] = m;
+
+    return 0;
+}
+
+static int l3_cdp_compare_val(const uint64_t val[],
+                              const struct feat_node *feat,
+                              unsigned int cos, bool *found)
+{
+    uint64_t l3_def_cbm;
+
+    l3_def_cbm = (1ull << feat->info.l3_cdp_info.cbm_len) - 1;
+
+    /*
+     * Different features' cos_max are different. If cos id of the feature
+     * being set exceeds other feature's cos_max, the val of other feature
+     * must be default value. HW supports such case.
+     */
+    if ( cos > feat->info.l3_cdp_info.cos_max )
+    {
+        if ( val[0] != l3_def_cbm ||
+             val[1] != l3_def_cbm )
+        {
+            *found = false;
+            return -ENOENT;
+        }
+        *found = true;
+    }
+    else
+        *found = (val[0] == get_cdp_data(feat, cos) &&
+                  val[1] == get_cdp_code(feat, cos));
+
+    return 0;
+}
+
+static bool l3_cdp_fits_cos_max(const uint64_t val[],
+                                const struct feat_node *feat,
+                                unsigned int cos)
+{
+    uint64_t l3_def_cbm;
+
+    l3_def_cbm = (1ull << feat->info.l3_cdp_info.cbm_len) - 1;
+
+    if ( cos > feat->info.l3_cdp_info.cos_max &&
+         (val[0] != l3_def_cbm || val[1] != l3_def_cbm) )
+            /*
+             * Exceed cos_max and value to set is not default,
+             * return error.
+             */
+            return false;
+
+    return true;
+}
+
+static int l3_cdp_write_msr(unsigned int cos, const uint64_t val[],
+                            struct feat_node *feat)
+{
+    /*
+     * If input cos is more than the cos_max of the feature, we should
+     * not set the value.
+     */
+    if ( cos > feat->info.l3_cdp_info.cos_max )
+        return -EINVAL;
+
+    /* Data */
+    if ( get_cdp_data(feat, cos) != val[0] )
+    {
+        get_cdp_data(feat, cos) = val[0];
+        wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(cos), val[0]);
+    }
+    /* Code */
+    if ( get_cdp_code(feat, cos) != val[1] )
+    {
+        get_cdp_code(feat, cos) = val[1];
+        wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(cos), val[1]);
+    }
+
+    return 0;
+}
+
 struct feat_ops l3_cdp_ops = {
     .get_cos_max = l3_cdp_get_cos_max,
     .get_feat_info = l3_cdp_get_feat_info,
     .get_val = l3_cdp_get_val,
+    .get_cos_num = l3_cdp_get_cos_num,
+    .get_old_val = l3_cdp_get_old_val,
+    .set_new_val = l3_cdp_set_new_val,
+    .compare_val = l3_cdp_compare_val,
+    .fits_cos_max = l3_cdp_fits_cos_max,
+    .write_msr = l3_cdp_write_msr,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 17/24] x86: L2 CAT: implement CPU init and free flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (15 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 16/24] x86: refactor psr: implement set value callback functions " Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
                   ` (6 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements the CPU init and free flow for L2 CAT including
L2 CAT initialization callback function.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - remove 'feat_l2_cat' free in 'free_feature'.
    - Encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
    - Print socket info when 'opt_cpu_info' is true.
    - rename 'l2_cat_get_max_cos_max' to 'l2_cat_get_cos_max'.
    - rename 'dat[]' to 'data[]'
    - move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
---
 xen/arch/x86/psr.c        | 72 +++++++++++++++++++++++++++++++++++++++++++++++
 xen/include/asm-x86/psr.h |  1 +
 2 files changed, 73 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 596e5b1..5320ae6 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -94,6 +94,7 @@ struct feat_hw_info {
     union {
         struct psr_cat_hw_info l3_cat_info;
         struct psr_cat_hw_info l3_cdp_info;
+        struct psr_cat_hw_info l2_cat_info;
     };
 };
 
@@ -236,6 +237,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
  */
 static struct feat_node *feat_l3_cat;
 static struct feat_node *feat_l3_cdp;
+static struct feat_node *feat_l2_cat;
 
 /* Common functions. */
 static void free_feature(struct psr_socket_info *info)
@@ -672,6 +674,51 @@ struct feat_ops l3_cdp_ops = {
     .write_msr = l3_cdp_write_msr,
 };
 
+/* L2 CAT callback functions implementation. */
+static void l2_cat_init_feature(struct cpuid_leaf_regs regs,
+                                struct feat_node *feat,
+                                struct psr_socket_info *info)
+{
+    struct psr_cat_hw_info l2_cat;
+    unsigned int socket;
+
+    /* No valid values so do not enable the feature. */
+    if ( !regs.eax || !regs.edx )
+        return;
+
+    l2_cat.cbm_len = (regs.eax & CAT_CBM_LEN_MASK) + 1;
+    l2_cat.cos_max = min(opt_cos_max, regs.edx & CAT_COS_MAX_MASK);
+
+    /* cos=0 is reserved as default cbm(all ones). */
+    feat->cos_reg_val[0] = (1ull << l2_cat.cbm_len) - 1;
+
+    feat->feature = PSR_SOCKET_L2_CAT;
+    __set_bit(PSR_SOCKET_L2_CAT, &info->feat_mask);
+
+    feat->info.l2_cat_info = l2_cat;
+
+    info->nr_feat++;
+
+    /* Add this feature into list. */
+    list_add_tail(&feat->list, &info->feat_list);
+
+    socket = cpu_to_socket(smp_processor_id());
+    if ( opt_cpu_info )
+        printk(XENLOG_INFO
+           "L2 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u.\n",
+           socket, feat->info.l2_cat_info.cos_max,
+           feat->info.l2_cat_info.cbm_len);
+}
+
+static unsigned int l2_cat_get_cos_max(const struct feat_node *feat)
+{
+    return feat->info.l2_cat_info.cos_max;
+}
+
+struct feat_ops l2_cat_ops = {
+    .get_cos_max = l2_cat_get_cos_max,
+};
+
 static void __init parse_psr_bool(char *s, char *value, char *feature,
                                   unsigned int mask)
 {
@@ -1445,6 +1492,20 @@ static void cpu_init_work(void)
             l3_cat_init_feature(regs, feat, info);
         }
     }
+
+    cpuid_count(PSR_CPUID_LEVEL_CAT, 0,
+                &regs.eax, &regs.ebx, &regs.ecx, &regs.edx);
+    if ( regs.ebx & PSR_RESOURCE_TYPE_L2 )
+    {
+        /* Initialize L2 CAT according to CPUID. */
+        cpuid_count(PSR_CPUID_LEVEL_CAT, 2,
+                    &regs.eax, &regs.ebx, &regs.ecx, &regs.edx);
+
+        feat = feat_l2_cat;
+        feat_l2_cat = NULL;
+        feat->ops = l2_cat_ops;
+        l2_cat_init_feature(regs, feat, info);
+    }
 }
 
 static void cpu_fini_work(unsigned int cpu)
@@ -1503,6 +1564,17 @@ static int psr_cpu_prepare(unsigned int cpu)
         return -ENOMEM;
     }
 
+    if ( feat_l2_cat == NULL &&
+         (feat_l2_cat = xzalloc(struct feat_node)) == NULL )
+    {
+        xfree(feat_l3_cat);
+        feat_l3_cat = NULL;
+
+        xfree(feat_l3_cdp);
+        feat_l3_cdp = NULL;
+        return -ENOMEM;
+    }
+
     return 0;
 }
 
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 97214fe..d2c7a13 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -23,6 +23,7 @@
 
 /* Resource Type Enumeration */
 #define PSR_RESOURCE_TYPE_L3            0x2
+#define PSR_RESOURCE_TYPE_L2            0x4
 
 /* L3 Monitoring Features */
 #define PSR_CMT_L3_OCCUPANCY           0x1
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 18/24] x86: L2 CAT: implement get hw info flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (16 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 19/24] x86: L2 CAT: implement get value flow Yi Sun
                   ` (5 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements get HW info flow for L2 CAT including L2 CAT callback
function.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - rename 'dat[]' to 'data[]'
    - remove type check in callback function.
---
 xen/arch/x86/psr.c          | 16 ++++++++++++++++
 xen/arch/x86/sysctl.c       | 15 +++++++++++++++
 xen/include/asm-x86/psr.h   |  1 +
 xen/include/public/sysctl.h |  6 ++++++
 4 files changed, 38 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 5320ae6..b630c48 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -268,6 +268,9 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
     case PSR_CBM_TYPE_L3_CODE:
         feat_type = PSR_SOCKET_L3_CDP;
         break;
+    case PSR_CBM_TYPE_L2:
+        feat_type = PSR_SOCKET_L2_CAT;
+        break;
     default:
         feat_type = 0xFFFF;
         break;
@@ -715,8 +718,21 @@ static unsigned int l2_cat_get_cos_max(const struct feat_node *feat)
     return feat->info.l2_cat_info.cos_max;
 }
 
+static bool l2_cat_get_feat_info(const struct feat_node *feat,
+                                 uint32_t data[], uint32_t array_len)
+{
+    if ( !data || 2 > array_len )
+        return false;
+
+    data[CBM_LEN] = feat->info.l2_cat_info.cbm_len;
+    data[COS_MAX] = feat->info.l2_cat_info.cos_max;
+
+    return true;
+}
+
 struct feat_ops l2_cat_ops = {
     .get_cos_max = l2_cat_get_cos_max,
+    .get_feat_info = l2_cat_get_feat_info,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index a4c8cfe..ae3600a 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -207,6 +207,21 @@ long arch_do_sysctl(
                 ret = -EFAULT;
             break;
         }
+        case XEN_SYSCTL_PSR_CAT_get_l2_info:
+        {
+            uint32_t dat[2];
+            ret = psr_get_info(sysctl->u.psr_cat_op.target,
+                               PSR_CBM_TYPE_L2, dat, 2);
+            if ( ret )
+                break;
+
+            sysctl->u.psr_cat_op.u.l2_info.cbm_len = dat[CBM_LEN];
+            sysctl->u.psr_cat_op.u.l2_info.cos_max = dat[COS_MAX];
+
+            if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
+                ret = -EFAULT;
+            break;
+        }
         default:
             ret = -EOPNOTSUPP;
             break;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index d2c7a13..31aa332 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -56,6 +56,7 @@ enum cbm_type {
     PSR_CBM_TYPE_L3,
     PSR_CBM_TYPE_L3_CODE,
     PSR_CBM_TYPE_L3_DATA,
+    PSR_CBM_TYPE_L2,
 };
 
 extern struct psr_cmt *psr_cmt;
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 00f5e77..cbf5372 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
 DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
 
 #define XEN_SYSCTL_PSR_CAT_get_l3_info               0
+#define XEN_SYSCTL_PSR_CAT_get_l2_info               1
 struct xen_sysctl_psr_cat_op {
     uint32_t cmd;       /* IN: XEN_SYSCTL_PSR_CAT_* */
     uint32_t target;    /* IN */
@@ -754,6 +755,11 @@ struct xen_sysctl_psr_cat_op {
 #define XEN_SYSCTL_PSR_CAT_L3_CDP       (1u << 0)
             uint32_t flags;     /* OUT: CAT flags */
         } l3_info;
+
+        struct {
+            uint32_t cbm_len;   /* OUT: CBM length */
+            uint32_t cos_max;   /* OUT: Maximum COS */
+        } l2_info;
     } u;
 };
 typedef struct xen_sysctl_psr_cat_op xen_sysctl_psr_cat_op_t;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 19/24] x86: L2 CAT: implement get value flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (17 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 20/24] x86: L2 CAT: implement set " Yi Sun
                   ` (4 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements L2 CAT get value callback function and
interface in domctl.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - remove type check in callback function.
---
 xen/arch/x86/domctl.c       |  7 +++++++
 xen/arch/x86/psr.c          | 12 ++++++++++++
 xen/include/public/domctl.h |  1 +
 3 files changed, 20 insertions(+)

diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index db56500..af6153d 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1403,6 +1403,13 @@ long arch_do_domctl(
             copyback = 1;
             break;
 
+        case XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM:
+            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+                              &domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L2);
+            copyback = 1;
+            break;
+
         default:
             ret = -EOPNOTSUPP;
             break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index b630c48..1fad540 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -730,9 +730,21 @@ static bool l2_cat_get_feat_info(const struct feat_node *feat,
     return true;
 }
 
+static bool l2_cat_get_val(const struct feat_node *feat, unsigned int cos,
+                          enum cbm_type type, uint64_t *val)
+{
+    if ( cos > feat->info.l2_cat_info.cos_max )
+        cos = 0;
+
+    *val = feat->cos_reg_val[cos];
+
+    return true;
+}
+
 struct feat_ops l2_cat_ops = {
     .get_cos_max = l2_cat_get_cos_max,
     .get_feat_info = l2_cat_get_feat_info,
+    .get_val = l2_cat_get_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 85cbb7c..8c183ba 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
 #define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA    3
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE    4
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA    5
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM     7
     uint32_t cmd;       /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
     uint32_t target;    /* IN */
     uint64_t data;      /* IN/OUT */
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 20/24] x86: L2 CAT: implement set value flow.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (18 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 19/24] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 21/24] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
                   ` (3 subsequent siblings)
  23 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements L2 CAT set value related callback functions
and domctl interface.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - remove type check in callback function.
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
    - remove 'l2_cat_get_cos_max_from_type'.
    - rename 'l2_cat_exceeds_cos_max' to 'l2_cat_fits_cos_max'.
---
 xen/arch/x86/domctl.c           |  6 +++
 xen/arch/x86/psr.c              | 92 +++++++++++++++++++++++++++++++++++++++++
 xen/include/asm-x86/msr-index.h |  1 +
 xen/include/public/domctl.h     |  1 +
 4 files changed, 100 insertions(+)

diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index af6153d..2767c6a 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1382,6 +1382,12 @@ long arch_do_domctl(
                               PSR_CBM_TYPE_L3_DATA);
             break;
 
+        case XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM:
+            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+                              domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L2);
+            break;
+
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
             ret = psr_get_val(d, domctl->u.psr_cat_op.target,
                               &domctl->u.psr_cat_op.data,
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 1fad540..13d85e0 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -741,10 +741,102 @@ static bool l2_cat_get_val(const struct feat_node *feat, unsigned int cos,
     return true;
 }
 
+static unsigned int l2_cat_get_cos_num(const struct feat_node *feat)
+{
+    /* L2 CAT uses one COS. */
+    return 1;
+}
+
+static int l2_cat_get_old_val(uint64_t val[],
+                              const struct feat_node *feat,
+                              unsigned int old_cos)
+{
+    if ( old_cos > feat->info.l2_cat_info.cos_max )
+        /* Use default value. */
+        old_cos = 0;
+
+    val[0] = feat->cos_reg_val[old_cos];
+
+    return 0;
+}
+
+static int l2_cat_set_new_val(uint64_t val[],
+                              const struct feat_node *feat,
+                              enum cbm_type type,
+                              uint64_t m)
+{
+    if ( !psr_check_cbm(feat->info.l2_cat_info.cbm_len, m) )
+        return -EINVAL;
+
+    val[0] = m;
+
+    return 0;
+}
+
+static int l2_cat_compare_val(const uint64_t val[],
+                              const struct feat_node *feat,
+                              unsigned int cos, bool *found)
+{
+    uint64_t l2_def_cbm;
+
+    l2_def_cbm = (1ull << feat->info.l2_cat_info.cbm_len) - 1;
+
+    if ( cos > feat->info.l2_cat_info.cos_max )
+    {
+        if ( val[0] != l2_def_cbm )
+        {
+            *found = false;
+            return -ENOENT;
+        }
+        *found = true;
+    }
+    else
+        *found = (val[0] == feat->cos_reg_val[cos]);
+
+    return 0;
+}
+
+static bool l2_cat_fits_cos_max(const uint64_t val[],
+                                const struct feat_node *feat,
+                                unsigned int cos)
+{
+    uint64_t l2_def_cbm;
+
+    l2_def_cbm = (1ull << feat->info.l2_cat_info.cbm_len) - 1;
+
+    if ( cos > feat->info.l2_cat_info.cos_max &&
+         val[0] != l2_def_cbm )
+            /*
+             * Exceed cos_max and value to set is not default,
+             * return error.
+             */
+            return false;
+
+    return true;
+}
+
+static int l2_cat_write_msr(unsigned int cos, const uint64_t val[],
+                            struct feat_node *feat)
+{
+    if ( cos > feat->info.l2_cat_info.cos_max )
+        return -EINVAL;
+
+    feat->cos_reg_val[cos] = val[0];
+    wrmsrl(MSR_IA32_PSR_L2_MASK(cos), val[0]);
+
+    return 0;
+}
+
 struct feat_ops l2_cat_ops = {
     .get_cos_max = l2_cat_get_cos_max,
     .get_feat_info = l2_cat_get_feat_info,
     .get_val = l2_cat_get_val,
+    .get_cos_num = l2_cat_get_cos_num,
+    .get_old_val = l2_cat_get_old_val,
+    .set_new_val = l2_cat_set_new_val,
+    .compare_val = l2_cat_compare_val,
+    .fits_cos_max = l2_cat_fits_cos_max,
+    .write_msr = l2_cat_write_msr,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h
index 98dbff1..a41e63a 100644
--- a/xen/include/asm-x86/msr-index.h
+++ b/xen/include/asm-x86/msr-index.h
@@ -343,6 +343,7 @@
 #define MSR_IA32_PSR_L3_MASK(n)	(0x00000c90 + (n))
 #define MSR_IA32_PSR_L3_MASK_CODE(n)	(0x00000c90 + (n) * 2 + 1)
 #define MSR_IA32_PSR_L3_MASK_DATA(n)	(0x00000c90 + (n) * 2)
+#define MSR_IA32_PSR_L2_MASK(n)		(0x00000d10 + (n))
 
 /* Intel Model 6 */
 #define MSR_P6_PERFCTR(n)		(0x000000c1 + (n))
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 8c183ba..523a2cd 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
 #define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA    3
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE    4
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA    5
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM     6
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM     7
     uint32_t cmd;       /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
     uint32_t target;    /* IN */
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 21/24] tools: L2 CAT: support get HW info for L2 CAT.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (19 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 20/24] x86: L2 CAT: implement set " Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-27 15:18   ` Wei Liu
  2017-01-19  6:01 ` [PATCH RESEND v5 22/24] tools: L2 CAT: support show cbm " Yi Sun
                   ` (2 subsequent siblings)
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements xl/xc changes to support get HW info
for L2 CAT.

'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
info.

Example(on machine which only supports L2 CAT):
Cache Monitoring Technology (CMT):
Enabled         : 0
Cache Allocation Technology (CAT): L2
Socket ID       : 0
Maximum COS     : 3
CBM length      : 8
Default CBM     : 0xff

Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - modify commit message to remove error log.
    - replace unnecessary 'return' to 'break'.
    - restore 'libxl_psr_cat_get_l3_info' to keep interface backward compatible
      but change codes in it to call new function to get hw info.
    - add 'L2_CBM' into 'psr_cbm_type' because it is interface change which
      should be in same patch with new 'LIBXL_HAVE_' macro.
    - addjust logs sentence to make unnecessary error logs not show.
---
 tools/libxc/include/xenctrl.h |  6 ++---
 tools/libxc/xc_psr.c          | 40 +++++++++++++++++++++++----------
 tools/libxl/libxl.h           |  9 ++++++++
 tools/libxl/libxl_psr.c       | 19 +++++++++++-----
 tools/libxl/libxl_types.idl   |  1 +
 tools/libxl/xl_cmdimpl.c      | 52 +++++++++++++++++++++++++++++++++----------
 6 files changed, 95 insertions(+), 32 deletions(-)

diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 4ab0f57..7ea0c92 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2626,9 +2626,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
 int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
                                xc_psr_cat_type type, uint32_t target,
                                uint64_t *data);
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
-                           uint32_t *cos_max, uint32_t *cbm_len,
-                           bool *cdp_enabled);
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+                        uint32_t *cos_max, uint32_t *cbm_len,
+                        bool *cdp_enabled);
 
 int xc_get_cpu_levelling_caps(xc_interface *xch, uint32_t *caps);
 int xc_get_cpu_featureset(xc_interface *xch, uint32_t index,
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 43b3286..6c61aa5 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -317,24 +317,40 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
     return rc;
 }
 
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
-                           uint32_t *cos_max, uint32_t *cbm_len,
-                           bool *cdp_enabled)
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+                        uint32_t *cos_max, uint32_t *cbm_len, bool *cdp_enabled)
 {
-    int rc;
+    int rc = -1;
     DECLARE_SYSCTL;
 
     sysctl.cmd = XEN_SYSCTL_psr_cat_op;
-    sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
     sysctl.u.psr_cat_op.target = socket;
 
-    rc = xc_sysctl(xch, &sysctl);
-    if ( !rc )
-    {
-        *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
-        *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
-        *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
-                       XEN_SYSCTL_PSR_CAT_L3_CDP;
+    switch ( lvl ) {
+    case 2:
+        sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l2_info;
+        rc = xc_sysctl(xch, &sysctl);
+        if ( !rc )
+        {
+            *cos_max = sysctl.u.psr_cat_op.u.l2_info.cos_max;
+            *cbm_len = sysctl.u.psr_cat_op.u.l2_info.cbm_len;
+            *cdp_enabled = false;
+        }
+        break;
+    case 3:
+        sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
+        rc = xc_sysctl(xch, &sysctl);
+        if ( !rc )
+        {
+            *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
+            *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
+            *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
+                           XEN_SYSCTL_PSR_CAT_L3_CDP;
+        }
+        break;
+    default:
+        errno = EOPNOTSUPP;
+        break;
     }
 
     return rc;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 3924464..c75a928 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -904,6 +904,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, const libxl_mac *src);
  * If this is defined, the Code and Data Prioritization feature is supported.
  */
 #define LIBXL_HAVE_PSR_CDP 1
+
+/*
+ * LIBXL_HAVE_PSR_L2_CAT
+ *
+ * If this is defined, the L2 Cache Allocation Technology feature is supported.
+ */
+#define LIBXL_HAVE_PSR_L2_CAT 1
 #endif
 
 /*
@@ -2166,6 +2173,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
  * On success, the function returns an array of elements in 'info',
  * and the length in 'nr'.
  */
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+                           int *nr, unsigned int lvl);
 int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
                               int *nr);
 void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr);
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index ec5c79d..68c45da 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -352,8 +352,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
     return rc;
 }
 
-int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
-                              int *nr)
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+                           int *nr, unsigned int lvl)
 {
     GC_INIT(ctx);
     int rc;
@@ -380,9 +380,8 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
 
     libxl_for_each_set_bit(socketid, socketmap) {
         ptr[i].id = socketid;
-        if (xc_psr_cat_get_l3_info(ctx->xch, socketid, &ptr[i].cos_max,
-                                   &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
-            libxl__psr_cat_log_err_msg(gc, errno);
+        if (xc_psr_cat_get_info(ctx->xch, socketid, lvl, &ptr[i].cos_max,
+                                &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
             rc = ERROR_FAIL;
             free(ptr);
             goto out;
@@ -398,6 +397,16 @@ out:
     return rc;
 }
 
+int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+                              int *nr)
+{
+    int rc;
+
+    rc = libxl_psr_cat_get_info(ctx, info, nr, 3);
+
+    return rc;
+}
+
 void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr)
 {
     int i;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a612d1f..5a401b8 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -899,6 +899,7 @@ libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
     (1, "L3_CBM"),
     (2, "L3_CBM_CODE"),
     (3, "L3_CBM_DATA"),
+    (4, "L2_CBM"),
     ])
 
 libxl_psr_cat_info = Struct("psr_cat_info", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 7e8a8ae..42d6827 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9331,21 +9331,19 @@ int main_psr_cmt_show(int argc, char **argv)
 }
 #endif
 
-#ifdef LIBXL_HAVE_PSR_CAT
-static int psr_cat_hwinfo(void)
+#if defined(LIBXL_HAVE_PSR_CAT) || defined(LIBXL_HAVE_PSR_L2_CAT)
+static int psr_l3_cat_hwinfo(void)
 {
-    int rc;
-    int i, nr;
+    int rc, nr;
+    unsigned int i;
     uint32_t l3_cache_size;
     libxl_psr_cat_info *info;
 
-    printf("Cache Allocation Technology (CAT):\n");
-
-    rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
-    if (rc) {
-        fprintf(stderr, "Failed to get cat info\n");
+    rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+    if (rc)
         return rc;
-    }
+
+    printf("Cache Allocation Technology (CAT): L3\n");
 
     for (i = 0; i < nr; i++) {
         rc = libxl_psr_cmt_get_l3_cache_size(ctx, info[i].id, &l3_cache_size);
@@ -9454,7 +9452,7 @@ static int psr_cat_show(uint32_t domid)
     int rc;
     libxl_psr_cat_info *info;
 
-    rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
+    rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
     if (rc) {
         fprintf(stderr, "Failed to get cat info\n");
         return rc;
@@ -9471,6 +9469,32 @@ out:
     return rc;
 }
 
+static int psr_l2_cat_hwinfo(void)
+{
+    int rc;
+    unsigned int i;
+    int nr;
+    libxl_psr_cat_info *info;
+
+    rc = libxl_psr_cat_get_info(ctx, &info, &nr, 2);
+    if (rc)
+        return rc;
+
+    printf("Cache Allocation Technology (CAT): L2\n");
+
+    for (i = 0; i < nr; i++) {
+        /* There is no CMT on L2 cache so far. */
+        printf("%-16s: %u\n", "Socket ID", info[i].id);
+        printf("%-16s: %u\n", "Maximum COS", info[i].cos_max);
+        printf("%-16s: %u\n", "CBM length", info[i].cbm_len);
+        printf("%-16s: %#llx\n", "Default CBM",
+               (1ull << info[i].cbm_len) - 1);
+    }
+
+    libxl_psr_cat_info_list_free(info, nr);
+    return rc;
+}
+
 int main_psr_cat_cbm_set(int argc, char **argv)
 {
     uint32_t domid;
@@ -9588,7 +9612,11 @@ int main_psr_hwinfo(int argc, char **argv)
         ret = psr_cmt_hwinfo();
 
     if (!ret && (all || cat))
-        ret = psr_cat_hwinfo();
+        ret = psr_l3_cat_hwinfo();
+
+    /* L2 CAT is independent of CMT and L3 CAT */
+    if (all || cat)
+        ret = psr_l2_cat_hwinfo();
 
     return ret;
 }
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 22/24] tools: L2 CAT: support show cbm for L2 CAT.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (20 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 21/24] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-27 15:18   ` Wei Liu
  2017-01-19  6:01 ` [PATCH RESEND v5 23/24] tools: L2 CAT: support set " Yi Sun
  2017-01-19  6:01 ` [PATCH RESEND v5 24/24] docs: add L2 CAT description in docs Yi Sun
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements changes in xl/xc changes to support
showing CBM of L2 CAT.

The new level option is introduced to original CAT showing
command in order to show CBM for specified level CAT.
- 'xl psr-cat-show' is updated to show CBM of a domain
  according to input cache level.

Examples:
root@:~$ xl psr-cat-show -l2 1
Socket ID       : 0
Default CBM     : 0xff
   ID                     NAME             CBM
    1                 ubuntu14            0x7f

Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v5:
    - remove 'L2_CBM' in idl because it has been moved to patch 21.
---
 tools/libxc/include/xenctrl.h |  1 +
 tools/libxc/xc_psr.c          |  3 ++
 tools/libxl/xl_cmdimpl.c      | 81 ++++++++++++++++++++++++++++---------------
 tools/libxl/xl_cmdtable.c     |  3 +-
 4 files changed, 59 insertions(+), 29 deletions(-)

diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 7ea0c92..a009625 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2602,6 +2602,7 @@ enum xc_psr_cat_type {
     XC_PSR_CAT_L3_CBM      = 1,
     XC_PSR_CAT_L3_CBM_CODE = 2,
     XC_PSR_CAT_L3_CBM_DATA = 3,
+    XC_PSR_CAT_L2_CBM      = 4,
 };
 typedef enum xc_psr_cat_type xc_psr_cat_type;
 
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 6c61aa5..f0aed2d 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -299,6 +299,9 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
     case XC_PSR_CAT_L3_CBM_DATA:
         cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
         break;
+    case XC_PSR_CAT_L2_CBM:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM;
+        break;
     default:
         errno = EINVAL;
         return -1;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 42d6827..cdcee5f 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9379,7 +9379,7 @@ static void psr_cat_print_one_domain_cbm_type(uint32_t domid, uint32_t socketid,
 }
 
 static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
-                                         bool cdp_enabled)
+                                         bool cdp_enabled, unsigned int lvl)
 {
     char *domain_name;
 
@@ -9387,27 +9387,38 @@ static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
     printf("%5d%25s", domid, domain_name);
     free(domain_name);
 
-    if (!cdp_enabled) {
-        psr_cat_print_one_domain_cbm_type(domid, socketid,
-                                          LIBXL_PSR_CBM_TYPE_L3_CBM);
-    } else {
-        psr_cat_print_one_domain_cbm_type(domid, socketid,
-                                          LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+    switch (lvl) {
+    case 3:
+        if (!cdp_enabled) {
+            psr_cat_print_one_domain_cbm_type(domid, socketid,
+                                              LIBXL_PSR_CBM_TYPE_L3_CBM);
+        } else {
+            psr_cat_print_one_domain_cbm_type(domid, socketid,
+                                              LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+            psr_cat_print_one_domain_cbm_type(domid, socketid,
+                                              LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+        }
+        break;
+    case 2:
         psr_cat_print_one_domain_cbm_type(domid, socketid,
-                                          LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+                                          LIBXL_PSR_CBM_TYPE_L2_CBM);
+        break;
+    default:
+        printf("Input lvl %d is wrong!", lvl);
+        break;
     }
 
     printf("\n");
 }
 
 static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
-                                    bool cdp_enabled)
+                                    bool cdp_enabled, unsigned int lvl)
 {
     int i, nr_domains;
     libxl_dominfo *list;
 
     if (domid != INVALID_DOMID) {
-        psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
+        psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled, lvl);
         return 0;
     }
 
@@ -9417,49 +9428,55 @@ static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
     }
 
     for (i = 0; i < nr_domains; i++)
-        psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
+        psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled, lvl);
     libxl_dominfo_list_free(list, nr_domains);
 
     return 0;
 }
 
-static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
+static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info,
+                                unsigned int lvl)
 {
     int rc;
     uint32_t l3_cache_size;
 
-    rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
-    if (rc) {
-        fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
-                info->id);
-        return -1;
+    printf("%-16s: %u\n", "Socket ID", info->id);
+
+    /* So far, CMT only supports L3 cache. */
+    if (lvl == 3)
+    {
+        rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
+        if (rc) {
+            fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
+                    info->id);
+            return -1;
+        }
+        printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
     }
 
-    printf("%-16s: %u\n", "Socket ID", info->id);
-    printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
     printf("%-16s: %#llx\n", "Default CBM", (1ull << info->cbm_len) - 1);
     if (info->cdp_enabled)
         printf("%5s%25s%16s%16s\n", "ID", "NAME", "CBM (code)", "CBM (data)");
     else
         printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
 
-    return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled);
+    return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled, lvl);
 }
 
-static int psr_cat_show(uint32_t domid)
+static int psr_cat_show(uint32_t domid, unsigned int lvl)
 {
     int i, nr;
     int rc;
     libxl_psr_cat_info *info;
 
-    rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+    rc = libxl_psr_cat_get_info(ctx, &info, &nr, lvl);
     if (rc) {
-        fprintf(stderr, "Failed to get cat info\n");
+        fprintf(stderr, "Failed to get %s cat info\n", (lvl == 3)?"L3":"L2");
         return rc;
     }
 
     for (i = 0; i < nr; i++) {
-        rc = psr_cat_print_socket(domid, info + i);
+        rc = psr_cat_print_socket(domid, info + i, lvl);
         if (rc)
             goto out;
     }
@@ -9570,11 +9587,19 @@ int main_psr_cat_cbm_set(int argc, char **argv)
 
 int main_psr_cat_show(int argc, char **argv)
 {
-    int opt;
+    int opt = 0;
     uint32_t domid;
+    unsigned int lvl = 3;
 
-    SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cat-show", 0) {
-        /* No options */
+    static struct option opts[] = {
+        {"level", 1, 0, 'l'},
+        COMMON_LONG_OPTS
+    };
+
+    SWITCH_FOREACH_OPT(opt, "l:", opts, "psr-cat-show", 0) {
+    case 'l':
+        lvl = atoi(optarg);
+        break;
     }
 
     if (optind >= argc)
@@ -9586,7 +9611,7 @@ int main_psr_cat_show(int argc, char **argv)
         return 2;
     }
 
-    return psr_cat_show(domid);
+    return psr_cat_show(domid, lvl);
 }
 
 int main_psr_hwinfo(int argc, char **argv)
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index 588d5d9..c5fbad4 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -556,7 +556,8 @@ struct cmd_spec cmd_table[] = {
     { "psr-cat-show",
       &main_psr_cat_show, 0, 1,
       "Show Cache Allocation Technology information",
-      "<Domain>",
+      "[options] <Domain>",
+      "-l <level>        Specify the cache level to process, otherwise L3 cache is processed\n"
     },
 
 #endif
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 23/24] tools: L2 CAT: support set cbm for L2 CAT.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (21 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 22/24] tools: L2 CAT: support show cbm " Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-27 15:18   ` Wei Liu
  2017-01-19  6:01 ` [PATCH RESEND v5 24/24] docs: add L2 CAT description in docs Yi Sun
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch implements the xl/xc changes to support set CBM
for L2 CAT.

The new level option is introduced to original CAT setting
command in order to set CBM for specified level CAT.
- 'xl psr-cat-cbm-set' is updated to set cache capacity
  bitmasks(CBM) for a domain according to input cache level.

root@:~$ xl psr-cat-cbm-set -l2 1 0x7f

root@:~$ xl psr-cat-show -l2 1
Socket ID       : 0
Default CBM     : 0xff
   ID                     NAME             CBM
    1                 ubuntu14            0x7f

Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
 tools/libxc/xc_psr.c      |  3 +++
 tools/libxl/xl_cmdimpl.c  | 31 ++++++++++++++++++++-----------
 tools/libxl/xl_cmdtable.c |  1 +
 3 files changed, 24 insertions(+), 11 deletions(-)

diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index f0aed2d..67556bb 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -266,6 +266,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
     case XC_PSR_CAT_L3_CBM_DATA:
         cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
         break;
+    case XC_PSR_CAT_L2_CBM:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM;
+        break;
     default:
         errno = EINVAL;
         return -1;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index cdcee5f..a32438c 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -9523,19 +9523,21 @@ int main_psr_cat_cbm_set(int argc, char **argv)
     char *value;
     libxl_string_list socket_list;
     unsigned long start, end;
-    int i, j, len;
+    unsigned int i, j, len;
+    unsigned int lvl = 3;
 
     static struct option opts[] = {
         {"socket", 1, 0, 's'},
         {"data", 0, 0, 'd'},
         {"code", 0, 0, 'c'},
+        {"level", 1, 0, 'l'},
         COMMON_LONG_OPTS
     };
 
     libxl_socket_bitmap_alloc(ctx, &target_map, 0);
     libxl_bitmap_set_none(&target_map);
 
-    SWITCH_FOREACH_OPT(opt, "s:cd", opts, "psr-cat-cbm-set", 2) {
+    SWITCH_FOREACH_OPT(opt, "s:l:cd", opts, "psr-cat-cbm-set", 2) {
     case 's':
         trim(isspace, optarg, &value);
         split_string_into_string_list(value, ",", &socket_list);
@@ -9555,17 +9557,24 @@ int main_psr_cat_cbm_set(int argc, char **argv)
     case 'c':
         opt_code = 1;
         break;
+    case 'l':
+        lvl = atoi(optarg);
+        break;
     }
 
-    if (opt_data && opt_code) {
-        fprintf(stderr, "Cannot handle -c and -d at the same time\n");
-        return -1;
-    } else if (opt_data) {
-        type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
-    } else if (opt_code) {
-        type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
-    } else {
-        type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+    if (lvl == 2)
+        type = LIBXL_PSR_CBM_TYPE_L2_CBM;
+    else if (lvl == 3) {
+        if (opt_data && opt_code) {
+            fprintf(stderr, "Cannot handle -c and -d at the same time\n");
+            return ERROR_FAIL;
+        } else if (opt_data) {
+            type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
+        } else if (opt_code) {
+            type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+        } else {
+            type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+        }
     }
 
     if (libxl_bitmap_is_empty(&target_map))
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index c5fbad4..32c3ee5 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -550,6 +550,7 @@ struct cmd_spec cmd_table[] = {
       "Set cache capacity bitmasks(CBM) for a domain",
       "[options] <Domain> <CBM>",
       "-s <socket>       Specify the socket to process, otherwise all sockets are processed\n"
+      "-l <level>        Specify the cache level to process, otherwise L3 cache is processed\n"
       "-c                Set code CBM if CDP is supported\n"
       "-d                Set data CBM if CDP is supported\n"
     },
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* [PATCH RESEND v5 24/24] docs: add L2 CAT description in docs.
  2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (22 preceding siblings ...)
  2017-01-19  6:01 ` [PATCH RESEND v5 23/24] tools: L2 CAT: support set " Yi Sun
@ 2017-01-19  6:01 ` Yi Sun
  2017-01-27 15:18   ` Wei Liu
  23 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-19  6:01 UTC (permalink / raw)
  To: xen-devel
  Cc: wei.liu2, konrad.wilk, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng

This patch adds L2 CAT description in related documents.

Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
 docs/man/xl.pod.1.in      | 25 ++++++++++++++++++++++---
 docs/misc/xl-psr.markdown | 10 ++++++++--
 2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 8e2aa5b..2c41ea7 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -1701,6 +1701,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
 a domain is attached to the monitoring service, monitoring data can be shown
 for any of these monitoring types.
 
+There is no cache monitoring and memory bandwidth monitoring on L2 cache so
+far.
+
 =over 4
 
 =item B<psr-cmt-attach> [I<domain-id>]
@@ -1725,7 +1728,7 @@ monitor types are:
 
 Intel Broadwell and later server platforms offer capabilities to configure and
 make use of the Cache Allocation Technology (CAT) mechanisms, which enable more
-cache resources (i.e. L3 cache) to be made available for high priority
+cache resources (i.e. L3/L2 cache) to be made available for high priority
 applications. In the Xen implementation, CAT is used to control cache allocation
 on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
 (CBM) for the domain.
@@ -1735,7 +1738,7 @@ Intel Broadwell and later server platforms also offer Code/Data Prioritization
 applications. CDP is used on a per VM basis in the Xen implementation. To
 specify code or data CBM for the domain, CDP feature must be enabled and CBM
 type options need to be specified when setting CBM, and the type options (code
-and data) are mutually exclusive.
+and data) are mutually exclusive. There is no CDP support on L2 so far.
 
 =over 4
 
@@ -1752,6 +1755,11 @@ B<OPTIONS>
 
 Specify the socket to process, otherwise all sockets are processed.
 
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
 =item B<-c>, B<--code>
 
 Set code CBM when CDP is enabled.
@@ -1762,10 +1770,21 @@ Set data CBM when CDP is enabled.
 
 =back
 
-=item B<psr-cat-show> [I<domain-id>]
+=item B<psr-cat-show> [I<OPTIONS>] [I<domain-id>]
 
 Show CAT settings for a certain domain or all domains.
 
+B<OPTIONS>
+
+=over 4
+
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
+=back
+
 =back
 
 =head1 IGNORED FOR COMPATIBILITY WITH XM
diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
index c3c1e8e..bd2b6bd 100644
--- a/docs/misc/xl-psr.markdown
+++ b/docs/misc/xl-psr.markdown
@@ -70,7 +70,7 @@ total-mem-bandwidth instead of cache-occupancy). E.g. after a `xl psr-cmt-attach
 
 Cache Allocation Technology (CAT) is a new feature available on Intel
 Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
-partition cache allocation (i.e. L3 cache) based on application priority or
+partition cache allocation (i.e. L3/L2 cache) based on application priority or
 Class of Service (COS). Each COS is configured using capacity bitmasks (CBM)
 which represent cache capacity and indicate the degree of overlap and
 isolation between classes. System cache resource is divided into numbers of
@@ -119,13 +119,19 @@ A cbm is valid only when:
 In a multi-socket system, the same cbm will be set on each socket by default.
 Per socket cbm can be specified with the `--socket SOCKET` option.
 
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
+
 Setting the CBM may not be successful if insufficient COS is available. In
 such case unused COS(es) may be freed by setting CBM of all related domains to
 its default value(all-ones).
 
 Per domain CBM settings can be shown by:
 
-`xl psr-cat-show`
+`xl psr-cat-show [OPTIONS] <domid>`
+
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
 
 ## Code and Data Prioritization (CDP)
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document
  2017-01-19  6:01 ` [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document Yi Sun
@ 2017-01-20  9:39   ` Tian, Kevin
  2017-01-22  2:15     ` Yi Sun
  2017-01-30 18:10   ` Konrad Rzeszutek Wilk
  1 sibling, 1 reply; 55+ messages in thread
From: Tian, Kevin @ 2017-01-20  9:39 UTC (permalink / raw)
  To: Yi Sun, xen-devel
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, konrad.wilk,
	ian.jackson, mengxu, jbeulich, chao.p.peng

> From: Yi Sun
> Sent: Thursday, January 19, 2017 2:01 PM
> 
> This patch creates L2 CAT feature document in doc/features/.
> It describes details of L2 CAT.

A good write-up, but still some improvements required. :-)

> 
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
>  docs/features/intel_psr_l2_cat.pandoc | 347
> ++++++++++++++++++++++++++++++++++
>  1 file changed, 347 insertions(+)
>  create mode 100644 docs/features/intel_psr_l2_cat.pandoc
> 
> diff --git a/docs/features/intel_psr_l2_cat.pandoc
> b/docs/features/intel_psr_l2_cat.pandoc
> new file mode 100644
> index 0000000..77bd61f
> --- /dev/null
> +++ b/docs/features/intel_psr_l2_cat.pandoc
> @@ -0,0 +1,347 @@
> +% Intel L2 Cache Allocation Technology (L2 CAT) Feature
> +% Revision 1.0
> +
> +\clearpage
> +
> +# Basics
> +
> +---------------- ----------------------------------------------------
> +         Status: **Tech Preview**
> +
> +Architecture(s): Intel x86
> +
> +   Component(s): Hypervisor, toolstack
> +
> +       Hardware: Atom codename Goldmont and beyond CPUs
> +---------------- ----------------------------------------------------
> +
> +# Overview
> +
> +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
> +CPU's shared L2 cache based on application priority or Class of Service
> +(COS). Each CLOS is configured using capacity bitmasks (CBM) which
> +represent cache capacity and indicate the degree of overlap and
> +isolation between classes. Once L2 CAT is configured, the processor
> +allows access to portions of L2 cache according to the established
> +class of service.

I would suggest make this doc for all CAT features, otherwise some
content looks incomplete when you say adding new options or new
ranges with assumption that reader understands the existing stuff.
Yes I do notice you provide many background about L3 CAT/CDP. Let's
just make it complete.

Also as a permanent design, 'new' is not preferred since it will become
'existing' once it's checked in.

> +
> +## Terminology
> +
> +* CAT         Cache Allocation Technology
> +* CBM         Capacity BitMasks
> +* CDP         Code and Data Prioritization
> +* COS/CLOS    Class of Service
> +* MSRs        Machine Specific Registers
> +* PSR         Intel Platform Shared Resource
> +* VMM         Virtual Machine Monitor
> +
> +# User details
> +
> +* Feature Enabling:
> +
> +  Add "psr=cat" to boot line parameter to enable all supported level CAT
> +  features.
> +
> +* xl interfaces:
> +
> +  1. `psr-cat-show [OPTIONS] domain-id`:

Need an introduction of 'domain-id'. Is it a Xen domain ID, or
new hardware-defined domain ID (e.g. one socket per domain)?

> +
> +     Show domain L2 or L3 CAT CBM.

also please introduce how CBM actually works

> +
> +     New option `-l` is added.
> +     `-l2`: Show cbm for L2 cache.
> +     `-l3`: Show cbm for L3 cache.
> +
> +     If neither `-l2` nor `-l3` is given, show both of them. If any one
> +     is not supported, will print error info.
> +
> +  2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
> +
> +     Set domain L2 or L3 CBM.

to be consistent with 'show' - 'CBM' or 'CAT CBM'?

I understand those xl interfaces are there already. If there is chance
to change, I'd recommend removing 'cbm'. Just psr-cat-set to be
consistent with 'show'.

> +
> +     New option `-l` is added.
> +     `-l2`: Specify cbm for L2 cache.
> +     `-l3`: Specify cbm for L3 cache.
> +
> +     If neither `-l2` nor `-l3` is given, level 3 is the default option.
> +
> +  3. `psr-hwinfo [OPTIONS]`:
> +
> +     Show L2 & L3 CAT HW informations on every socket.

informations -> information

and which HW information? please elaborate.

and where is the interface of setting COS for VCPU?

> +
> +# Technical details
> +
> +L2 CAT is a member of Intel PSR features and part of CAT, it shares
> +some base PSR infrastructure in Xen.

Then add some background of PSR would be useful if nothing exists yet?

> +
> +## Hardware perspective
> +
> +L2 CAT defines a new range MSRs to assign different L2 cache access
> +patterns which are known as CBMs, each CBM is associated with a COS.
> +
> +```
> +
> +                        +----------------------------+----------------+
> +   IA32_PQR_ASSOC       | MSR (per socket)           |    Address     |
> + +----+---+-------+     +----------------------------+----------------+
> + |    |COS|       |     | IA32_L2_QOS_MASK_0         |     0xD10      |
> + +----+---+-------+     +----------------------------+----------------+
> +        └-------------> | ...                        |  ...           |
> +                        +----------------------------+----------------+
> +                        | IA32_L2_QOS_MASK_n         | 0xD10+n (n<64) |
> +                        +----------------------------+----------------+
> +```
> +
> +When context switch happens, the COS of VCPU is written to per-thread
> +MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
> +according to the corresponding CBM.
> +
> +## The relationship between L2 CAT and L3 CAT/CDP
> +
> +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
> +while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.
> +
> +L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
> +the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
> +patterns from L3 cache. 

"L2 CAT supports ..." or "L3 CAT supports..." for the last sentence? 

> Like L3 CAT/CDP requirement, the bits of CBM of
> +L2 CAT must be continuous too.
> +
> +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
> +associate register `IA32_PQR_ASSOC`, which means one COS associates to a
> +pair of L2 CBM and L3 CBM.
> +
> +Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
> +other PSR features in future). In some cases, a VM is permitted to have a
> +COS that is beyond one (or more) of PSR features but within the others.
> +For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
> +L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
> +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
> +limit) since COS 9 is beyond the max COS (8) of L2 CAT.

Does user space need to know such difference since L2 CAT may not 
be effective for this vCPU?
.."

> +
> +## Design Overview
> +
> +* Core COS/CBM association
> +
> +  When enforcing L2 CAT, all cores of domains have the same default
> +  COS (COS0) which associated to the fully open CBM (all ones bitmask)

please define 'open' when introducing CBM

> +  to access all L2 cache. The default COS is used only in hypervisor
> +  and is transparent to tool stack and user.
> +
> +  System administrator can change PSR allocation policy at runtime by
> +  tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds
> +  to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP
> +  is enabled, one COS corresponds to a 3-tuple, like [L2 CBM,
> +  L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled,
> +  things would be easier, one COS corresponds to one L2 CBM.
> +
> +* VCPU schedule
> +
> +  This part reuses L3 CAT COS infrastructure.

Please elaborate then.

> +
> +* Multi-sockets
> +
> +  Different sockets may have different L2 CAT capability (e.g. max COS)
> +  although it is consistent on the same socket. So the capability of
> +  per-socket L2 CAT is specified.

VCPU schedule design is important here. e.g. when migrating a VCPU 
from socket 1 to socket 2 (socket 1 has max COS as 16 while socket
2 as 8), will you prevent migration if VCPU has a COS (9)?

> +
> +## Implementation Description
> +
> +* Hypervisor interfaces:
> +
> +  1. Boot line parameter "psr=cat" now will enable L2 CAT and L3
> +     CAT if hardware supported.
> +
> +  2. SYSCTL:
> +          - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
> +
> +  3. DOMCTL:
> +          - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
> +          - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
> +
> +* xl interfaces:
> +
> +  1. psr-cat-show -l2 domain-id
> +          Show L2 cbm for a domain.
> +          => XEN_SYSCTL_PSR_CAT_get_l2_info /
> +             XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
> +
> +  2. psr-mba-set -l2 domain-id cbm
> +          Set L2 cbm for a domain.
> +          => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
> +
> +  3. psr-hwinfo
> +          Show PSR HW information, including L2 CAT
> +          => XEN_SYSCTL_PSR_CAT_get_l2_info

still no interface about setting COS.

> +
> +* Key data structure:
> +
> +   1. Feature HW info
> +
> +      ```
> +      struct psr_cat_hw_info {
> +          unsigned int cbm_len;
> +          unsigned int cos_max;
> +      };
> +      ```
> +
> +      - Member `cbm_len`
> +
> +        `cbm_len` is one of the hardware info of CAT.
> +
> +      - Member `cos_max`
> +
> +        `cos_max` is one of the hardware info of CAT.
> +
> +   2. Feature list node
> +
> +      ```
> +      struct feat_node {
> +          enum psr_feat_type feature;
> +          struct feat_ops ops;
> +          struct psr_cat_hw_info info;
> +          uint64_t cos_reg_val[MAX_COS_REG_NUM];
> +          struct list_head list;
> +      };
> +      ```
> +
> +      When a PSR enforcement feature is enabled, it will be added into a
> +      feature list. The head of the list is created in psr initialization.
> +
> +      - Member `feature`
> +
> +        `feature` is an integer number, to indicate which feature the list entry
> +        corresponds to.
> +
> +      - Member `ops`
> +
> +        `ops` maintains a callback function list of the feature. It will be introduced
> +        in details later.
> +
> +      - Member `info`
> +
> +        `info` maintains the feature HW information which can be got through
> +        psr_hwinfo command.
> +
> +      - Member `cos_reg_val`
> +
> +        `cos_reg_val` is an array to maintain the value set in all COS registers of
> +        the feature.
> +
> +   3. Per-socket PSR features information structure
> +
> +      ```
> +      struct psr_cat_socket_info {
> +          unsigned int feat_mask;
> +          unsigned int nr_feat;
> +          struct list_head feat_list;
> +          unsigned int cos_ref[MAX_COS_REG_NUM];
> +          spinlock_t ref_lock;
> +      };
> +      ```
> +
> +      We collect all PSR allocation features information of a socket in
> +      this `struct psr_cat_socket_info`.
> +
> +      - Member `feat_mask`
> +
> +        `feat_mask` is a bitmap, to indicate which feature is enabled on
> +        current socket. We define `feat_mask` bitmap as:
> +
> +        bit 0~1: L3 CAT status, [01] stands for L3 CAT only and [10]
> +                 stands for L3 CDP is enalbed.
> +
> +        bit 2: L2 CAT status.
> +
> +      - Member `cos_ref`
> +
> +        `cos_ref` is an array which maintains the reference of one COS.
> +        If the COS is used by one domain, the reference will increase one.
> +        If a domain releases the COS, the reference will decrease one. The
> +        array is indexed by COS.
> +
> +   4. Feature operation functions structure
> +
> +      ```
> +      struct feat_ops {
> +          void (*init_feature)(unsigned int eax, unsigned int ebx,
> +                               unsigned int ecx, unsigned int edx,
> +                               struct feat_node *feat,
> +                               struct psr_cat_socket_info *info);
> +          int (*get_feat_info)(const struct feat_node *feat, enum cbm_type type,
> +                               uint32_t dat[], uint32_t array_len);
> +          int (*get_val)(const struct feat_node *feat, unsigned int cos,
> +                         enum cbm_type type, uint64_t *val);
> +          unsigned int (*get_max_cos_max)(const struct feat_node *feat);
> +          unsigned int (*get_cos_num)(const struct feat_node *feat);
> +          int (*get_old_val)(uint64_t val[],
> +                             const struct feat_node *feat,
> +                             unsigned int old_cos);
> +          int (*set_new_val)(uint64_t val[],
> +                             const struct feat_node *feat,
> +                             unsigned int old_cos,
> +                             enum cbm_type type,
> +                             uint64_t m);
> +          int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
> +                             unsigned int cos, bool *found);
> +          unsigned int (*get_cos_max_from_type)(const struct feat_node *feat,
> +                                                enum cbm_type type);
> +          unsigned int (*exceeds_cos_max)(const uint64_t val[],
> +                                          const struct feat_node *feat,
> +                                          unsigned int cos);
> +          int (*write_msr)(unsigned int cos, const uint64_t val[],
> +                           struct feat_node *feat);
> +      };
> +      ```
> +
> +      We abstract above callback functions to encapsulate the feature specific
> +      behaviors into them. Then, it is easy to add a new feature. We just need:
> +          1) Implement such ops and callback functions for every feature.
> +          2) Register the ops into `struct feat_node`.
> +          3) Add the feature into feature list during CPU initialization.
> +
> +# Limitations
> +
> +L2 CAT can only work on HW which enables it(check by CPUID). So far, there
> +is no HW enables both L2 CAT and L3 CAT/CDP. But SW implementation has considered
> +such scenario to enable both L2 CAT and L3 CAT/CDP.
> +
> +# Testing
> +
> +L2 CAT uses same xl interfaces as L3 CAT/CDP. So, we can execute these
> +commands to verify L2 CAT and L3 CAT/CDP on different HWs support them.
> +
> +For example:
> +    root@:~$ xl psr-hwinfo --cat
> +    Cache Allocation Technology (CAT): L2
> +    Socket ID       : 0
> +    Maximum COS     : 3
> +    CBM length      : 8
> +    Default CBM     : 0xff
> +
> +    root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
> +
> +    root@:~$ xl psr-cat-show -l2 1
> +    Socket ID       : 0
> +    Default CBM     : 0xff
> +       ID                     NAME             CBM
> +        1                 ubuntu14            0x7f
> +
> +# Areas for improvement
> +
> +N/A
> +
> +# Known issues
> +
> +N/A
> +
> +# References
> +
> +"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES"
> [Intel® 64 and IA-32 Architectures Software Developer Manuals,
> vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-dev
> eloper-manuals.html)
> +
> +# History
> +
> +------------------------------------------------------------------------
> +Date       Revision Version  Notes
> +---------- -------- -------- -------------------------------------------
> +2016-08-12 1.0      Xen 4.9  Design document written
> +---------- -------- -------- -------------------------------------------
> --
> 1.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document
  2017-01-20  9:39   ` Tian, Kevin
@ 2017-01-22  2:15     ` Yi Sun
  2017-02-08  6:45       ` Tian, Kevin
  0 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-01-22  2:15 UTC (permalink / raw)
  To: Tian, Kevin
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On 17-01-20 09:39:41, Tian, Kevin wrote:
> > From: Yi Sun
> > Sent: Thursday, January 19, 2017 2:01 PM
> > 
> > This patch creates L2 CAT feature document in doc/features/.
> > It describes details of L2 CAT.
> 
> A good write-up, but still some improvements required. :-)
> 
Thanks a lot for review and the good suggestions!

> > 
> > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> > ---
> >  docs/features/intel_psr_l2_cat.pandoc | 347
> > ++++++++++++++++++++++++++++++++++
> >  1 file changed, 347 insertions(+)
> >  create mode 100644 docs/features/intel_psr_l2_cat.pandoc
> > 
> > diff --git a/docs/features/intel_psr_l2_cat.pandoc
> > b/docs/features/intel_psr_l2_cat.pandoc
> > new file mode 100644
> > index 0000000..77bd61f
> > --- /dev/null
> > +++ b/docs/features/intel_psr_l2_cat.pandoc
> > @@ -0,0 +1,347 @@
> > +% Intel L2 Cache Allocation Technology (L2 CAT) Feature
> > +% Revision 1.0
> > +
> > +\clearpage
> > +
> > +# Basics
> > +
> > +---------------- ----------------------------------------------------
> > +         Status: **Tech Preview**
> > +
> > +Architecture(s): Intel x86
> > +
> > +   Component(s): Hypervisor, toolstack
> > +
> > +       Hardware: Atom codename Goldmont and beyond CPUs
> > +---------------- ----------------------------------------------------
> > +
> > +# Overview
> > +
> > +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
> > +CPU's shared L2 cache based on application priority or Class of Service
> > +(COS). Each CLOS is configured using capacity bitmasks (CBM) which
> > +represent cache capacity and indicate the degree of overlap and
> > +isolation between classes. Once L2 CAT is configured, the processor
> > +allows access to portions of L2 cache according to the established
> > +class of service.
> 
> I would suggest make this doc for all CAT features, otherwise some
> content looks incomplete when you say adding new options or new
> ranges with assumption that reader understands the existing stuff.
> Yes I do notice you provide many background about L3 CAT/CDP. Let's
> just make it complete.
> 
> Also as a permanent design, 'new' is not preferred since it will become
> 'existing' once it's checked in.
> 
We have discussed this before. I proposed to add L3 CAT/CDP feature documents
later. But considering readers may not have background knowledge, I will try
to change this feature document to cover all CAT/CDP features.

> > +
> > +     New option `-l` is added.
> > +     `-l2`: Specify cbm for L2 cache.
> > +     `-l3`: Specify cbm for L3 cache.
> > +
> > +     If neither `-l2` nor `-l3` is given, level 3 is the default option.
> > +
> > +  3. `psr-hwinfo [OPTIONS]`:
> > +
> > +     Show L2 & L3 CAT HW informations on every socket.
> 
> informations -> information
> 
> and which HW information? please elaborate.
> 
> and where is the interface of setting COS for VCPU?
> 
What do you mean 'the interface of setting COS for VCPU'? User can set CBM
for one domain through 'psr-cat-cbm-set' now. Then, hypervisor will find
a matched COS ID or pick an available COS ID for the domain. User does not
need know which COS ID it is using.

> > +When context switch happens, the COS of VCPU is written to per-thread
> > +MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
> > +according to the corresponding CBM.
> > +
> > +## The relationship between L2 CAT and L3 CAT/CDP
> > +
> > +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
> > +while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.
> > +
> > +L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
> > +the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
> > +patterns from L3 cache. 
> 
> "L2 CAT supports ..." or "L3 CAT supports..." for the last sentence? 
> 
It is 'L2 CAT'. Sorry for confusion.

> > Like L3 CAT/CDP requirement, the bits of CBM of
> > +L2 CAT must be continuous too.
> > +
> > +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
> > +associate register `IA32_PQR_ASSOC`, which means one COS associates to a
> > +pair of L2 CBM and L3 CBM.
> > +
> > +Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
> > +other PSR features in future). In some cases, a VM is permitted to have a
> > +COS that is beyond one (or more) of PSR features but within the others.
> > +For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
> > +L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
> > +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
> > +limit) since COS 9 is beyond the max COS (8) of L2 CAT.
> 
> Does user space need to know such difference since L2 CAT may not 
> be effective for this vCPU?
> .."
> 
In fact, L2 CAT will use default value for such case. This is done by HW.
User space can know the max COS differences through 'psr-hwinfo'.

> > +
> > +* Multi-sockets
> > +
> > +  Different sockets may have different L2 CAT capability (e.g. max COS)
> > +  although it is consistent on the same socket. So the capability of
> > +  per-socket L2 CAT is specified.
> 
> VCPU schedule design is important here. e.g. when migrating a VCPU 
> from socket 1 to socket 2 (socket 1 has max COS as 16 while socket
> 2 as 8), will you prevent migration if VCPU has a COS (9)?
>
'psr-cat-cbm-set' can set CBM for one domain per socket. On each socket, we
maintain a COS array for all domains. One domain uses one COS at one time. One
COS saves the CBM of to work. So, when a VCPU of the domain is migrated from
socket 1 to socket 2, it follows configuration on socket 2. This mechanism has
been implemented since L3 CAT. L2 CAT follows it.

E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets
domain 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
is migrated from socket 1 to 2, the COS ID used will be 7, that means 0x3f
will be the CBM to work for this domain.

Thanks,
Sun Yi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 21/24] tools: L2 CAT: support get HW info for L2 CAT.
  2017-01-19  6:01 ` [PATCH RESEND v5 21/24] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
@ 2017-01-27 15:18   ` Wei Liu
  0 siblings, 0 replies; 55+ messages in thread
From: Wei Liu @ 2017-01-27 15:18 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Thu, Jan 19, 2017 at 02:01:23PM +0800, Yi Sun wrote:
> This patch implements xl/xc changes to support get HW info
> for L2 CAT.
> 
> 'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
> info.
> 
> Example(on machine which only supports L2 CAT):
> Cache Monitoring Technology (CMT):
> Enabled         : 0
> Cache Allocation Technology (CAT): L2
> Socket ID       : 0
> Maximum COS     : 3
> CBM length      : 8
> Default CBM     : 0xff
> 
> Signed-off-by: He Chen <he.chen@linux.intel.com>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>

Acked-by: Wei Liu <wei.liu2@citrix.com>

Only one nit below.
[...]
> -int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
> -                           uint32_t *cos_max, uint32_t *cbm_len,
> -                           bool *cdp_enabled)
> +int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
> +                        uint32_t *cos_max, uint32_t *cbm_len, bool *cdp_enabled)
>  {
> -    int rc;
> +    int rc = -1;
>      DECLARE_SYSCTL;
>  
>      sysctl.cmd = XEN_SYSCTL_psr_cat_op;
> -    sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
>      sysctl.u.psr_cat_op.target = socket;
>  
> -    rc = xc_sysctl(xch, &sysctl);
> -    if ( !rc )
> -    {
> -        *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
> -        *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
> -        *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
> -                       XEN_SYSCTL_PSR_CAT_L3_CDP;
> +    switch ( lvl ) {

Please put '{' on a new line.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 22/24] tools: L2 CAT: support show cbm for L2 CAT.
  2017-01-19  6:01 ` [PATCH RESEND v5 22/24] tools: L2 CAT: support show cbm " Yi Sun
@ 2017-01-27 15:18   ` Wei Liu
  0 siblings, 0 replies; 55+ messages in thread
From: Wei Liu @ 2017-01-27 15:18 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Thu, Jan 19, 2017 at 02:01:24PM +0800, Yi Sun wrote:
> This patch implements changes in xl/xc changes to support
> showing CBM of L2 CAT.
> 
> The new level option is introduced to original CAT showing
> command in order to show CBM for specified level CAT.
> - 'xl psr-cat-show' is updated to show CBM of a domain
>   according to input cache level.
> 
> Examples:
> root@:~$ xl psr-cat-show -l2 1
> Socket ID       : 0
> Default CBM     : 0xff
>    ID                     NAME             CBM
>     1                 ubuntu14            0x7f
> 
> Signed-off-by: He Chen <he.chen@linux.intel.com>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>

Acked-by: Wei Liu <wei.liu2@citrix.com>

One minor nit.

[...]
> -static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
> +static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info,
> +                                unsigned int lvl)
>  {
>      int rc;
>      uint32_t l3_cache_size;
>  
> -    rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
> -    if (rc) {
> -        fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
> -                info->id);
> -        return -1;
> +    printf("%-16s: %u\n", "Socket ID", info->id);
> +
> +    /* So far, CMT only supports L3 cache. */
> +    if (lvl == 3)
> +    {

Please put '{' on previous line.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 23/24] tools: L2 CAT: support set cbm for L2 CAT.
  2017-01-19  6:01 ` [PATCH RESEND v5 23/24] tools: L2 CAT: support set " Yi Sun
@ 2017-01-27 15:18   ` Wei Liu
  0 siblings, 0 replies; 55+ messages in thread
From: Wei Liu @ 2017-01-27 15:18 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Thu, Jan 19, 2017 at 02:01:25PM +0800, Yi Sun wrote:
> This patch implements the xl/xc changes to support set CBM
> for L2 CAT.
> 
> The new level option is introduced to original CAT setting
> command in order to set CBM for specified level CAT.
> - 'xl psr-cat-cbm-set' is updated to set cache capacity
>   bitmasks(CBM) for a domain according to input cache level.
> 
> root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
> 
> root@:~$ xl psr-cat-show -l2 1
> Socket ID       : 0
> Default CBM     : 0xff
>    ID                     NAME             CBM
>     1                 ubuntu14            0x7f
> 
> Signed-off-by: He Chen <he.chen@linux.intel.com>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>

Acked-by: Wei Liu <wei.liu2@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 24/24] docs: add L2 CAT description in docs.
  2017-01-19  6:01 ` [PATCH RESEND v5 24/24] docs: add L2 CAT description in docs Yi Sun
@ 2017-01-27 15:18   ` Wei Liu
  0 siblings, 0 replies; 55+ messages in thread
From: Wei Liu @ 2017-01-27 15:18 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Thu, Jan 19, 2017 at 02:01:26PM +0800, Yi Sun wrote:
> This patch adds L2 CAT description in related documents.
> 
> Signed-off-by: He Chen <he.chen@linux.intel.com>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>

Acked-by: Wei Liu <wei.liu2@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document
  2017-01-19  6:01 ` [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document Yi Sun
  2017-01-20  9:39   ` Tian, Kevin
@ 2017-01-30 18:10   ` Konrad Rzeszutek Wilk
  2017-01-30 20:39     ` Konrad Rzeszutek Wilk
  2017-02-04  7:06     ` Yi Sun
  1 sibling, 2 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-30 18:10 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Thu, Jan 19, 2017 at 02:01:03PM +0800, Yi Sun wrote:
> This patch creates L2 CAT feature document in doc/features/.
> It describes details of L2 CAT.

Perhaps also mention what is the title in the Intel SDM 
to look into as well?
Perhaps:
"See Intel Resource Director Technology Monitoring Features"
in the Intel SDM."

> 
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
>  docs/features/intel_psr_l2_cat.pandoc | 347 ++++++++++++++++++++++++++++++++++
>  1 file changed, 347 insertions(+)
>  create mode 100644 docs/features/intel_psr_l2_cat.pandoc
> 
> diff --git a/docs/features/intel_psr_l2_cat.pandoc b/docs/features/intel_psr_l2_cat.pandoc
> new file mode 100644
> index 0000000..77bd61f
> --- /dev/null
> +++ b/docs/features/intel_psr_l2_cat.pandoc
> @@ -0,0 +1,347 @@
> +% Intel L2 Cache Allocation Technology (L2 CAT) Feature
> +% Revision 1.0
> +
> +\clearpage
> +
> +# Basics
> +
> +---------------- ----------------------------------------------------
> +         Status: **Tech Preview**
> +
> +Architecture(s): Intel x86
> +
> +   Component(s): Hypervisor, toolstack
> +
> +       Hardware: Atom codename Goldmont and beyond CPUs
> +---------------- ----------------------------------------------------
> +
> +# Overview
> +
> +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
> +CPU's shared L2 cache based on application priority or Class of Service
> +(COS). Each CLOS is configured using capacity bitmasks (CBM) which
> +represent cache capacity and indicate the degree of overlap and

indicates
> +isolation between classes. Once L2 CAT is configured, the processor
> +allows access to portions of L2 cache according to the established
> +class of service.
> +
> +## Terminology
> +
> +* CAT         Cache Allocation Technology
> +* CBM         Capacity BitMasks
> +* CDP         Code and Data Prioritization
> +* COS/CLOS    Class of Service
> +* MSRs        Machine Specific Registers
> +* PSR         Intel Platform Shared Resource
> +* VMM         Virtual Machine Monitor
> +
> +# User details
> +
> +* Feature Enabling:
> +
> +  Add "psr=cat" to boot line parameter to enable all supported level CAT
> +  features.
> +
> +* xl interfaces:
> +
> +  1. `psr-cat-show [OPTIONS] domain-id`:
> +
> +     Show domain L2 or L3 CAT CBM.
> +
> +     New option `-l` is added.
> +     `-l2`: Show cbm for L2 cache.
> +     `-l3`: Show cbm for L3 cache.
> +
> +     If neither `-l2` nor `-l3` is given, show both of them. If any one
> +     is not supported, will print error info.
> +
> +  2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
> +
> +     Set domain L2 or L3 CBM.
> +
> +     New option `-l` is added.
> +     `-l2`: Specify cbm for L2 cache.
> +     `-l3`: Specify cbm for L3 cache.
> +
> +     If neither `-l2` nor `-l3` is given, level 3 is the default option.
> +
> +  3. `psr-hwinfo [OPTIONS]`:
> +
> +     Show L2 & L3 CAT HW informations on every socket.
> +
> +# Technical details
> +
> +L2 CAT is a member of Intel PSR features and part of CAT, it shares
> +some base PSR infrastructure in Xen.
> +
> +## Hardware perspective
> +
> +L2 CAT defines a new range MSRs to assign different L2 cache access
                             ^- 'of'

> +patterns which are known as CBMs, each CBM is associated with a COS.
> +
> +```
> +
> +                        +----------------------------+----------------+
> +   IA32_PQR_ASSOC       | MSR (per socket)           |    Address     |
> + +----+---+-------+     +----------------------------+----------------+
> + |    |COS|       |     | IA32_L2_QOS_MASK_0         |     0xD10      |
> + +----+---+-------+     +----------------------------+----------------+
> +        └-------------> | ...                        |  ...           |
> +                        +----------------------------+----------------+
> +                        | IA32_L2_QOS_MASK_n         | 0xD10+n (n<64) |
> +                        +----------------------------+----------------+
> +```
> +
> +When context switch happens, the COS of VCPU is written to per-thread
> +MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
> +according to the corresponding CBM.
> +
> +## The relationship between L2 CAT and L3 CAT/CDP
> +
> +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
> +while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.

s/all/both/

> +
> +L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by

s/by//
> +the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
> +patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of
> +L2 CAT must be continuous too.
> +
> +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
> +associate register `IA32_PQR_ASSOC`, which means one COS associates to a

s/associates to a/is associated with a/

> +pair of L2 CBM and L3 CBM.
> +
> +Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
> +other PSR features in future). In some cases, a VM is permitted to have a
> +COS that is beyond one (or more) of PSR features but within the others.
> +For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
> +L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
> +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
> +limit) since COS 9 is beyond the max COS (8) of L2 CAT.


Thank you for mentioning the above!

> +
> +## Design Overview
> +
> +* Core COS/CBM association
> +
> +  When enforcing L2 CAT, all cores of domains have the same default
> +  COS (COS0) which associated to the fully open CBM (all ones bitmask)
                     ^-is
> +  to access all L2 cache. The default COS is used only in hypervisor
> +  and is transparent to tool stack and user.
> +
> +  System administrator can change PSR allocation policy at runtime by
> +  tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds
> +  to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP
> +  is enabled, one COS corresponds to a 3-tuple, like [L2 CBM,
> +  L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled,
> +  things would be easier, one COS corresponds to one L2 CBM.
> +
> +* VCPU schedule
> +
> +  This part reuses L3 CAT COS infrastructure.
> +
> +* Multi-sockets
> +
> +  Different sockets may have different L2 CAT capability (e.g. max COS)
> +  although it is consistent on the same socket. So the capability of
> +  per-socket L2 CAT is specified.
> +
> +## Implementation Description
> +
> +* Hypervisor interfaces:
> +
> +  1. Boot line parameter "psr=cat" now will enable L2 CAT and L3
> +     CAT if hardware supported.
> +
> +  2. SYSCTL:
> +          - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
> +
> +  3. DOMCTL:
> +          - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
> +          - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
> +
> +* xl interfaces:
> +
> +  1. psr-cat-show -l2 domain-id
> +          Show L2 cbm for a domain.
> +          => XEN_SYSCTL_PSR_CAT_get_l2_info /
> +             XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
> +
> +  2. psr-mba-set -l2 domain-id cbm
> +          Set L2 cbm for a domain.
> +          => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
> +
> +  3. psr-hwinfo
> +          Show PSR HW information, including L2 CAT
> +          => XEN_SYSCTL_PSR_CAT_get_l2_info
> +
> +* Key data structure:
> +
> +   1. Feature HW info
> +
> +      ```
> +      struct psr_cat_hw_info {
> +          unsigned int cbm_len;
> +          unsigned int cos_max;
> +      };
> +      ```
> +
> +      - Member `cbm_len`
> +
> +        `cbm_len` is one of the hardware info of CAT.

Could you expand? Is it the max number of bits you can set? 

> +
> +      - Member `cos_max`
> +
> +        `cos_max` is one of the hardware info of CAT.
> +
> +   2. Feature list node
> +
> +      ```
> +      struct feat_node {
> +          enum psr_feat_type feature;
> +          struct feat_ops ops;
> +          struct psr_cat_hw_info info;
> +          uint64_t cos_reg_val[MAX_COS_REG_NUM];
> +          struct list_head list;
> +      };
> +      ```
> +
> +      When a PSR enforcement feature is enabled, it will be added into a
> +      feature list. The head of the list is created in psr initialization.
> +
> +      - Member `feature`
> +
> +        `feature` is an integer number, to indicate which feature the list entry
> +        corresponds to.
> +
> +      - Member `ops`
> +
> +        `ops` maintains a callback function list of the feature. It will be introduced
> +        in details later.

??? Like by each patch to this document?

> +
> +      - Member `info`
> +
> +        `info` maintains the feature HW information which can be got through

s/which an be got through/which are provided to/ 

Perhaps?
> +        psr_hwinfo command.
> +
> +      - Member `cos_reg_val`
> +
> +        `cos_reg_val` is an array to maintain the value set in all COS registers of
> +        the feature.
> +
> +   3. Per-socket PSR features information structure
> +
> +      ```
> +      struct psr_cat_socket_info {
> +          unsigned int feat_mask;
> +          unsigned int nr_feat;
> +          struct list_head feat_list;
> +          unsigned int cos_ref[MAX_COS_REG_NUM];
> +          spinlock_t ref_lock;
> +      };
> +      ```
> +
> +      We collect all PSR allocation features information of a socket in
> +      this `struct psr_cat_socket_info`.
> +
> +      - Member `feat_mask`
> +
> +        `feat_mask` is a bitmap, to indicate which feature is enabled on
> +        current socket. We define `feat_mask` bitmap as:
> +
> +        bit 0~1: L3 CAT status, [01] stands for L3 CAT only and [10]
> +                 stands for L3 CDP is enalbed.

enabled
> +
> +        bit 2: L2 CAT status.

And the 'nr_feat' is 3 ?

> +
> +      - Member `cos_ref`
> +
> +        `cos_ref` is an array which maintains the reference of one COS.

Is it safe to assume that this maps to 'cos_reg_val' ? If so you may
want to mention that.

> +        If the COS is used by one domain, the reference will increase one.

s/one/by one/

> +        If a domain releases the COS, the reference will decrease one. The

s/decrease one/decrease by one/
> +        array is indexed by COS.
> +
> +   4. Feature operation functions structure
> +
> +      ```
> +      struct feat_ops {
> +          void (*init_feature)(unsigned int eax, unsigned int ebx,
> +                               unsigned int ecx, unsigned int edx,
> +                               struct feat_node *feat,
> +                               struct psr_cat_socket_info *info);
> +          int (*get_feat_info)(const struct feat_node *feat, enum cbm_type type,
> +                               uint32_t dat[], uint32_t array_len);
> +          int (*get_val)(const struct feat_node *feat, unsigned int cos,
> +                         enum cbm_type type, uint64_t *val);
> +          unsigned int (*get_max_cos_max)(const struct feat_node *feat);
> +          unsigned int (*get_cos_num)(const struct feat_node *feat);
> +          int (*get_old_val)(uint64_t val[],
> +                             const struct feat_node *feat,
> +                             unsigned int old_cos);
> +          int (*set_new_val)(uint64_t val[],
> +                             const struct feat_node *feat,
> +                             unsigned int old_cos,
> +                             enum cbm_type type,
> +                             uint64_t m);
> +          int (*compare_val)(const uint64_t val[], const struct feat_node *feat,
> +                             unsigned int cos, bool *found);
> +          unsigned int (*get_cos_max_from_type)(const struct feat_node *feat,
> +                                                enum cbm_type type);
> +          unsigned int (*exceeds_cos_max)(const uint64_t val[],
> +                                          const struct feat_node *feat,
> +                                          unsigned int cos);
> +          int (*write_msr)(unsigned int cos, const uint64_t val[],
> +                           struct feat_node *feat);
> +      };
> +      ```
> +
> +      We abstract above callback functions to encapsulate the feature specific
> +      behaviors into them. Then, it is easy to add a new feature. We just need:
> +          1) Implement such ops and callback functions for every feature.
> +          2) Register the ops into `struct feat_node`.
> +          3) Add the feature into feature list during CPU initialization.
> +
> +# Limitations
> +
> +L2 CAT can only work on HW which enables it(check by CPUID). So far, there
> +is no HW enables both L2 CAT and L3 CAT/CDP. But SW implementation has considered

s/no HW/no HW which/
> +such scenario to enable both L2 CAT and L3 CAT/CDP.
> +
> +# Testing
> +
> +L2 CAT uses same xl interfaces as L3 CAT/CDP. So, we can execute these
> +commands to verify L2 CAT and L3 CAT/CDP on different HWs support them.
> +
> +For example:
> +    root@:~$ xl psr-hwinfo --cat
> +    Cache Allocation Technology (CAT): L2
> +    Socket ID       : 0
> +    Maximum COS     : 3
> +    CBM length      : 8
> +    Default CBM     : 0xff
> +
> +    root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
> +
> +    root@:~$ xl psr-cat-show -l2 1
> +    Socket ID       : 0
> +    Default CBM     : 0xff
> +       ID                     NAME             CBM
> +        1                 ubuntu14            0x7f
> +
> +# Areas for improvement
> +
> +N/A
> +
> +# Known issues
> +
> +N/A
> +
> +# References
> +
> +"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
> +
> +# History
> +
> +------------------------------------------------------------------------
> +Date       Revision Version  Notes
> +---------- -------- -------- -------------------------------------------
> +2016-08-12 1.0      Xen 4.9  Design document written
> +---------- -------- -------- -------------------------------------------
> -- 
> 1.9.1
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document
  2017-01-30 18:10   ` Konrad Rzeszutek Wilk
@ 2017-01-30 20:39     ` Konrad Rzeszutek Wilk
  2017-02-04  7:24       ` Yi Sun
  2017-02-04  7:06     ` Yi Sun
  1 sibling, 1 reply; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-30 20:39 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

> > +# Testing
> > +
> > +L2 CAT uses same xl interfaces as L3 CAT/CDP. So, we can execute these
> > +commands to verify L2 CAT and L3 CAT/CDP on different HWs support them.
> > +
> > +For example:
> > +    root@:~$ xl psr-hwinfo --cat
> > +    Cache Allocation Technology (CAT): L2
> > +    Socket ID       : 0
> > +    Maximum COS     : 3
> > +    CBM length      : 8
> > +    Default CBM     : 0xff
> > +
> > +    root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
> > +
> > +    root@:~$ xl psr-cat-show -l2 1
> > +    Socket ID       : 0
> > +    Default CBM     : 0xff
> > +       ID                     NAME             CBM
> > +        1                 ubuntu14            0x7f
> > +

I tried finding the Intel SDM (December 2016) what the format
of CBM is as the value '0x7f' does not really mean much to me.

Right above Figure 17.28 it says:

"Figure 17-27 also shows three examples of sets of Cache Capacity Bitmasks. For simplicity these are represented
as 8-bit vectors, though this may vary _depending on the implementation and how the mask is mapped to the avail-
able cache capacity._"


So in other words - not documented. 

Then later is says:

" Rather, this is a convenient manner to represent capacity,
overlap and isolation of cache space. For example, executing a POPCNT instruction (population count of set bits) on
the capacity bitmask can provide the fraction of cache space that a class of service can allocate into."

OK, so _can_ (but not _MUST_), so again implementation specific - and
can provide a fraction of cache space.

Which would imply that the values could be:

0x0F - half of L2
0x03 - quarter of L2

Is there some other documentation that explains this in more details?

If it is like I mentioned would it make sense to have 'xl' be capable
of dealing with string values? such as "three-quarters", "half", etc
and then set it? Or percentage and map that to the correct value?

Like:

xl psr-cat-cbm-set -l2 ubuntu14 50%

would be quite obvious instead of say 0x0F?

Thanks.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 02/24] x86: refactor psr: remove L3 CAT/CDP codes.
  2017-01-19  6:01 ` [PATCH RESEND v5 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
@ 2017-01-30 22:05   ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-30 22:05 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Thu, Jan 19, 2017 at 02:01:04PM +0800, Yi Sun wrote:
> The current cache allocation codes in psr.c do not consider
> future features addition and are not friendly to extend.
> 
> To make psr.c be more flexible to add new features and fulfill
> the program principle, open for extension but closed for
> modification, we have to refactor the psr.c:
> 1. Analyze cache allocation features and abstract general data
>    structures.
> 2. Analyze the init and all other functions flow, abstract all
>    steps that different features may have different implementations.
>    Make these steps be callback functions and register feature
>    specific fuctions. Then, the main processes will not be changed
>    when introducing a new feature.
> 
> Because the quantity of refactor codes is big and the logics are
> changed a lot, it will cause reviewers confused if just change
> old codes. Reviewers have to understand both old codes and new
> implementations. After review iterations from V1 to V3, Jan has
> proposed to remove all old cache allocation codes firstly, then
> implement new codes step by step. This will help to make codes
> be more easily reviewable.
> 
> There is no construction without destruction. So, this patch
> removes all current L3 CAT/CDP codes in psr.c. The following
> patches will introduce the new mechanism.
> 
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> Acked-by: Jan Beulich <jbeulich@suse.com>

Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures.
  2017-01-19  6:01 ` [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures Yi Sun
@ 2017-01-30 22:20   ` Konrad Rzeszutek Wilk
  2017-01-31 10:10     ` Jan Beulich
  2017-02-05  1:53     ` Yi Sun
  0 siblings, 2 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-30 22:20 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

.snip..
> CDP is a special feature which uses two entries of the array
> for one COS ID. So, the number of CDP COS registers is the half of L3
> CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
> it is enabled. CDP uses the COS registers array as below.
> 
>                          +-----------+-----------+-----------+-----------+-----------+
> CDP cos_reg_val[] index: |     0     |     1     |     2     |     3     |    ...    |
>                          +-----------+-----------+-----------+-----------+-----------+
>                   value: | cos0 code | cos0 data | cos1 code | cos1 data |    ...    |
>                          +-----------+-----------+-----------+-----------+-----------+
> 
> For more details, please refer spec and codes.

Thanks for the description. Only had one comment about it:

'spec and codes'? Do you mean to specification. But what codes?
This one?
No need to mention that, that is kind of implicit.
> 
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> v5:
>     - explain CDP more in commit message.
>     - remove exact SDM chapter number but only keep title.
>     - remove init_feature from callback function ops structure.
> ---
>  xen/arch/x86/psr.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 104 insertions(+)
> 
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 96a8589..f7ff3fc 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -17,12 +17,116 @@
>  #include <xen/cpu.h>
>  #include <xen/err.h>
>  #include <xen/sched.h>
> +#include <xen/list.h>
>  #include <asm/psr.h>
>  
> +/*
> + * Terminology:
> + * - CAT         Cache Allocation Technology
> + * - CBM         Capacity BitMasks
> + * - CDP         Code and Data Prioritization
> + * - COS/CLOS    Class of Service. Also mean COS registers.
> + * - COS_MAX     Max number of COS for the feature (minus 1)
> + * - MSRs        Machine Specific Registers
> + * - PSR         Intel Platform Shared Resource
> + */
> +
>  #define PSR_CMT        (1<<0)
>  #define PSR_CAT        (1<<1)
>  #define PSR_CDP        (1<<2)
>  
> +/*
> + * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
> + * the MSRs range from 0C90H through 0D0FH (inclusive), enables support for

s/enables/enable/
> + * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
> + *
> + * The MSRs range from 0D10H through 0D4FH (inclusive), enables support for

s/enables/enable/
> + * up to 64 L2 CAT COS. The COS_ID=[0,63].
> + *
> + * So, the maximum COS register count of one feature is 128.
> + */
> +#define MAX_COS_REG_CNT  128
> +
> +/*
> + * PSR features are managed per socket. Below structure defines the members
> + * used to manage these features.
> + * feat_mask - Mask used to record features enabled on socket. There may be
> + *             some features enabled at same time.
> + * nr_feat   - Record how many features enabled.
> + * feat_list - A list used to manage all features enabled.
> + * cos_ref   - A reference count array to record how many domains are using the
> + *             COS_ID.
> + *             Every entry of cos_ref corresponds to one COS ID.
> + * ref_lock  - A lock to protect cos_ref.
> + */
> +struct psr_socket_info {
> +    /*
> +     * bit 0:   L3 CAT
> +     * bit 1:   L3 CDP
> +     * bit 2:   L2 CAT

Why not an enum? I am going to assume it is due to you programming
this in the MSRs. If so, I would recommend you have #define instead
of a comment.

#define L3_CAT (1U<<0)
#define L3_CDP (1U<<1)

. and so on..

> +     */
> +    unsigned int feat_mask;
> +    unsigned int nr_feat;
> +    struct list_head feat_list;
> +    unsigned int cos_ref[MAX_COS_REG_CNT];
> +    spinlock_t ref_lock;
> +};
> +
> +enum psr_feat_type {
> +    PSR_SOCKET_L3_CAT = 0,
> +    PSR_SOCKET_L3_CDP,
> +    PSR_SOCKET_L2_CAT,

Is there particular mapping between these and the feat_mask bit mask
values?

It kind of begs to be combined (unless you really need the 'feat_mask'
to be of specific order - in which case make sure you have BUILD_BUG_ON
to make sure nobody moves the #define values around - or put a comment
saying that you need the specific order of bits).

> +};
> +
> +/* CAT/CDP HW info data structure. */
> +struct psr_cat_hw_info {
> +    unsigned int cbm_len;
> +    unsigned int cos_max;
> +};
> +
> +/* Encapsulate feature specific HW info here. */
> +struct feat_hw_info {
> +    union {
> +        struct psr_cat_hw_info l3_cat_info;
> +    };
> +};
> +
> +struct feat_node;
> +
> +/*
> + * This structure defines feature operation callback functions. Every feature
> + * enabled MUST implement such callback functions and register them to ops.
> + *
> + * Feature specific behaviors will be encapsulated into these callback
> + * functions. Then, the main flows will not be changed when introducing a new
> + * feature.
> + */
> +struct feat_ops {
> +    /* get_cos_max is used to get feature's cos_max. */
> +    unsigned int (*get_cos_max)(const struct feat_node *feat);
> +};
> +
> +/*
> + * This structure represents one feature.
> + * feature     - Which feature it is.
> + * feat_ops    - Feature operation callback functions.
> + * info        - Feature HW info.
> + * cos_reg_val - Array to store the values of COS registers. One entry stores
> + *               the value of one COS register.
> + *               For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
> + *               For CDP, two entries correspond to one COS_ID. E.g.
> + *               COS_ID=0 corresponds to cos_reg_val[0] (Data) and
> + *               cos_reg_val[1] (Code).
> + * list        - Feature list.
> + */
> +struct feat_node {
> +    enum psr_feat_type feature;
> +    struct feat_ops ops;
> +    struct feat_hw_info info;
> +    uint64_t cos_reg_val[MAX_COS_REG_CNT];
> +    struct list_head list;
> +};
> +
>  struct psr_assoc {
>      uint64_t val;
>      uint64_t cos_mask;
> -- 
> 1.9.1
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 04/24] x86: refactor psr: implement CPU init and free flow.
  2017-01-19  6:01 ` [PATCH RESEND v5 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
@ 2017-01-31  2:44   ` Konrad Rzeszutek Wilk
  2017-01-31 10:14     ` Jan Beulich
  2017-01-31 10:53     ` Andrew Cooper
  0 siblings, 2 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-31  2:44 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Thu, Jan 19, 2017 at 02:01:06PM +0800, Yi Sun wrote:
> This patch implements the CPU init and free flow including L3 CAT
> initialization and feature list free.
> 
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> v5:
>     - modify commit message beacuse of code changes.
>     - add 'struct cpuid_leaf_regs' to save cpu registers value to reduce
>       parameters of init_feature function.
>     - modify comments to make them accurate.
>     - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
>     - use 'list_for_each_entry_safe' when free features.
>     - do not delete 'feat_l3_cat' to make it can be reused when cpu online.
>     - use 'current_cpu_data'.
>     - clear 'X86_FEATURE_PQE' if cpuid_level is not right.
>     - Print socket info when 'opt_cpu_info' is true.
>     - remove 'cpu_prepare_work' function and move contents of it into
>       'psr_cpu_prepare'.
> ---
>  xen/arch/x86/psr.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 174 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index f7ff3fc..e9dc07a 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -35,6 +35,9 @@
>  #define PSR_CAT        (1<<1)
>  #define PSR_CDP        (1<<2)
>  
> +#define CAT_CBM_LEN_MASK 0x1f
> +#define CAT_COS_MAX_MASK 0xffff
> +
>  /*
>   * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
>   * the MSRs range from 0C90H through 0D0FH (inclusive), enables support for
> @@ -127,6 +130,13 @@ struct feat_node {
>      struct list_head list;
>  };
>  
> +struct cpuid_leaf_regs {
> +    unsigned int eax;
> +    unsigned int ebx;
> +    unsigned int ecx;
> +    unsigned int edx;
> +};

Could you use 'struct cpuid_leaf' in x86_emulate.h ?

It would only require "#include <asm/x86_emulate.h>" I believe.

> +
>  struct psr_assoc {
>      uint64_t val;
>      uint64_t cos_mask;
> @@ -134,11 +144,76 @@ struct psr_assoc {
>  
>  struct psr_cmt *__read_mostly psr_cmt;
>  
> +static struct psr_socket_info *__read_mostly socket_info;
> +
>  static unsigned int opt_psr;
>  static unsigned int __initdata opt_rmid_max = 255;
> +static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
>  static uint64_t rmid_mask;
>  static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
>  
> +/*
> + * Declare global feature list entry for every feature to facilitate the
> + * feature list creation. It will be allocated in psr_cpu_prepare() and
> + * inserted into feature list in cpu_init_work().

You may also want to say it is protected by the cpu_add_remove_lock
spinlock.

> + */
> +static struct feat_node *feat_l3_cat;
> +
> +/* Common functions. */
> +static void free_feature(struct psr_socket_info *info)
> +{
> +    struct feat_node *feat, *next;
> +
> +    if ( !info )
> +        return;
> +
> +    list_for_each_entry_safe(feat, next, &info->feat_list, list)
> +    {
> +        clear_bit(feat->feature, &info->feat_mask);

Would it make sense to use __clear_bit?

As in you have already a lock (cpu_add_remove_lock) so nobody
can change the feat_mask. Hence thinking there is no need for the
lock operation?

> +        list_del(&feat->list);
> +        xfree(feat);
> +    }
> +}
> +
> +/* L3 CAT functions implementation. */
> +static void l3_cat_init_feature(struct cpuid_leaf_regs regs,
> +                                struct feat_node *feat,
> +                                struct psr_socket_info *info)
> +{
> +    struct psr_cat_hw_info l3_cat;
> +    unsigned int socket;
> +
> +    /* No valid value so do not enable feature. */
> +    if ( !regs.eax || !regs.edx )
> +        return;
> +
> +    l3_cat.cbm_len = (regs.eax & CAT_CBM_LEN_MASK) + 1;
> +    l3_cat.cos_max = min(opt_cos_max, regs.edx & CAT_COS_MAX_MASK);
> +
> +    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> +    feat->cos_reg_val[0] = (1ull << l3_cat.cbm_len) - 1;
> +
> +    feat->feature = PSR_SOCKET_L3_CAT;
> +    __set_bit(PSR_SOCKET_L3_CAT, &info->feat_mask);

Aha! So you do reuse the enum! In which case you really
want to make sure you mention that in 'struct psr_socket_info'
the 'feat_mask' is actually the bit values of enum psr_feat_type.

Also do you want to add an ASSERT before you call the __set_bit:

ASSERT ( !test_bit(PSR_SOCKET_L3_CAT, &info->feat_mask) );

?
> +
> +    feat->info.l3_cat_info = l3_cat;
> +
> +    info->nr_feat++;
> +
> +    /* Add this feature into list. */
> +    list_add_tail(&feat->list, &info->feat_list);
> +
> +    socket = cpu_to_socket(smp_processor_id());
> +    if ( opt_cpu_info )
> +        printk(XENLOG_INFO
> +           "L3 CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",

Odd spacing. I would think that the "L3 CAT: .. "
would be on the same column as XENLOG_INFO ?

Or you could flip this logic (if you are worried about the silly
80 characters requirement) around and do:

if ( !opt_cpu_info )
	return

And then do the printk on its own line?
> +           socket, feat->info.l3_cat_info.cos_max,
> +           feat->info.l3_cat_info.cbm_len);
> +}
> +
> +static const struct feat_ops l3_cat_ops = {
> +};
> +
>  static void __init parse_psr_bool(char *s, char *value, char *feature,
>                                    unsigned int mask)
>  {
> @@ -178,6 +253,9 @@ static void __init parse_psr_param(char *s)
>          if ( val_str && !strcmp(s, "rmid_max") )
>              opt_rmid_max = simple_strtoul(val_str, NULL, 0);
>  
> +        if ( val_str && !strcmp(s, "cos_max") )
> +            opt_cos_max = simple_strtoul(val_str, NULL, 0);
> +
>          s = ss + 1;
>      } while ( ss );
>  }
> @@ -333,18 +411,108 @@ void psr_domain_free(struct domain *d)
>      psr_free_rmid(d);
>  }
>  
> +static void cpu_init_work(void)
> +{
> +    struct psr_socket_info *info;
> +    unsigned int socket;
> +    unsigned int cpu = smp_processor_id();
> +    struct feat_node *feat;
> +    struct cpuid_leaf_regs regs;

Could you use the x86_emulate.h one?

Would it also make sense to make sure that you initialize
to default values:

	struct cpuid_leaf regs = { .a = 0, .b = 0, .c = 0, .d = 0 };
> +
> +    if ( !cpu_has(&current_cpu_data, X86_FEATURE_PQE) )
> +        return;
> +    else if ( current_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
> +    {
> +        clear_bit(X86_FEATURE_PQE, current_cpu_data.x86_capability);
> +        return;
> +    }
> +
> +    socket = cpu_to_socket(cpu);
> +    info = socket_info + socket;
> +    if ( info->feat_mask )
> +        return;
> +
> +    INIT_LIST_HEAD(&info->feat_list);
> +    spin_lock_init(&info->ref_lock);
> +
> +    cpuid_count(PSR_CPUID_LEVEL_CAT, 0,
> +                &regs.eax, &regs.ebx, &regs.ecx, &regs.edx);
> +    if ( regs.ebx & PSR_RESOURCE_TYPE_L3 )
> +    {
> +        cpuid_count(PSR_CPUID_LEVEL_CAT, 1,
> +                    &regs.eax, &regs.ebx, &regs.ecx, &regs.edx);
> +
> +        feat = feat_l3_cat;
> +        feat_l3_cat = NULL;

.. so how does this work when this is initialized on a third
socket? Won't feat_l3_cat at that point be NULL leading to this
one below becoming a NULL pointer?

Ah wait. psr_cpu_prepare will allocate it.

Hm, could you add a comment about this?

/* psr_cpu_prepare will allocate it on subsequent CPU onlining. */

> +        feat->ops = l3_cat_ops;
> +
> +        l3_cat_init_feature(regs, feat, info);
> +    }
> +}
> +
> +static void cpu_fini_work(unsigned int cpu)
> +{
> +    unsigned int socket = cpu_to_socket(cpu);
> +
> +    if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
> +    {
> +        free_feature(socket_info + socket);
> +    }
> +}
> +
> +static void __init init_psr(void)
> +{
> +    if ( opt_cos_max < 1 )
> +    {
> +        printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
> +        return;
> +    }
> +
> +    socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
> +
> +    if ( !socket_info )
> +    {
> +        printk(XENLOG_INFO "Fail to alloc socket_info!\n");

Failed
> +        return;
> +    }
> +}
> +
> +static void __init psr_free(void)
> +{
> +    unsigned int i;
> +
> +    for ( i = 0; i < nr_sockets; i++ )
> +        free_feature(&socket_info[i]);
> +
> +    xfree(socket_info);
> +    socket_info = NULL;
> +}
> +
>  static int psr_cpu_prepare(unsigned int cpu)
>  {
> +    if ( !socket_info )
> +        return 0;
> +
> +    /* Malloc memory for the global feature head here. */
> +    if ( feat_l3_cat == NULL &&
> +         (feat_l3_cat = xzalloc(struct feat_node)) == NULL )

/me scratches his head.

Lets say we have a two socket machine. Each socket has
two CPUs (so total of 4 CPUs).

On CPU0  (in psr_presmp_init) it allocates feat_l3_cat. Then
later in (in psr_presmp_init) you call psr_cpu_init which
sets feat_l3_cat to NULL (and 'feat' is feat_l3_cat).

When CPU1 starts up then, psr_cpu_prepare is called
which means we allocate a new feat_l3_cat. 'cpu_init_work'
short-circuits (as info->feat_mask has a value) and does not
do anything.

When CPU2 (socket two), starts up then feat_l3_cat (as 'feat')
is used (in psr_cpu_init and again sets feat_l3_cat to NULL).

The last CPU3 (socket #2) starts up, and since feat_l3_cat is
NULL - it is allocated. But in 'cpu_init_work' it short-circuits
so we don't use third allocated 'feat_l3_cat' .

And we have an 'feat_l3_cat' that is not used for anything..

In other words - a memory leak. Granted a very small one.


> +        return -ENOMEM;
> +
>      return 0;
>  }
>  
>  static void psr_cpu_init(void)
>  {
> +    if ( socket_info )
> +        cpu_init_work();
> +
>      psr_assoc_init();
>  }
>  
>  static void psr_cpu_fini(unsigned int cpu)
>  {
> +    if ( socket_info )
> +        cpu_fini_work(cpu);
>      return;
>  }
>  
> @@ -386,10 +554,14 @@ static int __init psr_presmp_init(void)
>      if ( (opt_psr & PSR_CMT) && opt_rmid_max )
>          init_psr_cmt(opt_rmid_max);
>  
> -    psr_cpu_prepare(0);
> +    if ( opt_psr & PSR_CAT )
> +        init_psr();
> +
> +    if ( psr_cpu_prepare(0) )
> +        psr_free();
>  
>      psr_cpu_init();
> -    if ( psr_cmt_enabled() )
> +    if ( psr_cmt_enabled() || socket_info )
>          register_cpu_notifier(&cpu_nfb);
>  
>      return 0;
> -- 
> 1.9.1
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures.
  2017-01-30 22:20   ` Konrad Rzeszutek Wilk
@ 2017-01-31 10:10     ` Jan Beulich
  2017-01-31 14:12       ` Konrad Rzeszutek Wilk
  2017-02-05  1:53     ` Yi Sun
  1 sibling, 1 reply; 55+ messages in thread
From: Jan Beulich @ 2017-01-31 10:10 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	Yi Sun, mengxu, xen-devel, chao.p.peng

>>> On 30.01.17 at 23:20, <konrad.wilk@oracle.com> wrote:
>> --- a/xen/arch/x86/psr.c
>> +++ b/xen/arch/x86/psr.c
>> @@ -17,12 +17,116 @@
>>  #include <xen/cpu.h>
>>  #include <xen/err.h>
>>  #include <xen/sched.h>
>> +#include <xen/list.h>
>>  #include <asm/psr.h>
>>  
>> +/*
>> + * Terminology:
>> + * - CAT         Cache Allocation Technology
>> + * - CBM         Capacity BitMasks
>> + * - CDP         Code and Data Prioritization
>> + * - COS/CLOS    Class of Service. Also mean COS registers.
>> + * - COS_MAX     Max number of COS for the feature (minus 1)
>> + * - MSRs        Machine Specific Registers
>> + * - PSR         Intel Platform Shared Resource
>> + */
>> +
>>  #define PSR_CMT        (1<<0)
>>  #define PSR_CAT        (1<<1)
>>  #define PSR_CDP        (1<<2)
>>  
>> +/*
>> + * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
>> + * the MSRs range from 0C90H through 0D0FH (inclusive), enables support for
> 
> s/enables/enable/
>> + * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
>> + *
>> + * The MSRs range from 0D10H through 0D4FH (inclusive), enables support for
> 
> s/enables/enable/

For both of them - why? Both talk about a (single) range.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 04/24] x86: refactor psr: implement CPU init and free flow.
  2017-01-31  2:44   ` Konrad Rzeszutek Wilk
@ 2017-01-31 10:14     ` Jan Beulich
  2017-01-31 14:13       ` Konrad Rzeszutek Wilk
  2017-01-31 10:53     ` Andrew Cooper
  1 sibling, 1 reply; 55+ messages in thread
From: Jan Beulich @ 2017-01-31 10:14 UTC (permalink / raw)
  To: Yi Sun, Konrad Rzeszutek Wilk
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, xen-devel, chao.p.peng

>>> On 31.01.17 at 03:44, <konrad.wilk@oracle.com> wrote:
> On Thu, Jan 19, 2017 at 02:01:06PM +0800, Yi Sun wrote:
>> @@ -127,6 +130,13 @@ struct feat_node {
>>      struct list_head list;
>>  };
>>  
>> +struct cpuid_leaf_regs {
>> +    unsigned int eax;
>> +    unsigned int ebx;
>> +    unsigned int ecx;
>> +    unsigned int edx;
>> +};
> 
> Could you use 'struct cpuid_leaf' in x86_emulate.h ?
> 
> It would only require "#include <asm/x86_emulate.h>" I believe.

Indeed - I recall specifically having asked for that.

>>  static int psr_cpu_prepare(unsigned int cpu)
>>  {
>> +    if ( !socket_info )
>> +        return 0;
>> +
>> +    /* Malloc memory for the global feature head here. */
>> +    if ( feat_l3_cat == NULL &&
>> +         (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
> 
> /me scratches his head.
> 
> Lets say we have a two socket machine. Each socket has
> two CPUs (so total of 4 CPUs).
> 
> On CPU0  (in psr_presmp_init) it allocates feat_l3_cat. Then
> later in (in psr_presmp_init) you call psr_cpu_init which
> sets feat_l3_cat to NULL (and 'feat' is feat_l3_cat).
> 
> When CPU1 starts up then, psr_cpu_prepare is called
> which means we allocate a new feat_l3_cat. 'cpu_init_work'
> short-circuits (as info->feat_mask has a value) and does not
> do anything.
> 
> When CPU2 (socket two), starts up then feat_l3_cat (as 'feat')
> is used (in psr_cpu_init and again sets feat_l3_cat to NULL).
> 
> The last CPU3 (socket #2) starts up, and since feat_l3_cat is
> NULL - it is allocated. But in 'cpu_init_work' it short-circuits
> so we don't use third allocated 'feat_l3_cat' .
> 
> And we have an 'feat_l3_cat' that is not used for anything..
> 
> In other words - a memory leak. Granted a very small one.

I did recommend to do it that way, to simplify things a little.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 04/24] x86: refactor psr: implement CPU init and free flow.
  2017-01-31  2:44   ` Konrad Rzeszutek Wilk
  2017-01-31 10:14     ` Jan Beulich
@ 2017-01-31 10:53     ` Andrew Cooper
  1 sibling, 0 replies; 55+ messages in thread
From: Andrew Cooper @ 2017-01-31 10:53 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk, Yi Sun
  Cc: wei.liu2, he.chen, dario.faggioli, ian.jackson, mengxu, jbeulich,
	chao.p.peng, xen-devel

On 31/01/17 02:44, Konrad Rzeszutek Wilk wrote:
> On Thu, Jan 19, 2017 at 02:01:06PM +0800, Yi Sun wrote:
>> This patch implements the CPU init and free flow including L3 CAT
>> initialization and feature list free.
>>
>> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
>> ---
>> v5:
>>     - modify commit message beacuse of code changes.
>>     - add 'struct cpuid_leaf_regs' to save cpu registers value to reduce
>>       parameters of init_feature function.
>>     - modify comments to make them accurate.
>>     - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
>>     - use 'list_for_each_entry_safe' when free features.
>>     - do not delete 'feat_l3_cat' to make it can be reused when cpu online.
>>     - use 'current_cpu_data'.
>>     - clear 'X86_FEATURE_PQE' if cpuid_level is not right.
>>     - Print socket info when 'opt_cpu_info' is true.
>>     - remove 'cpu_prepare_work' function and move contents of it into
>>       'psr_cpu_prepare'.
>> ---
>>  xen/arch/x86/psr.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
>>  1 file changed, 174 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
>> index f7ff3fc..e9dc07a 100644
>> --- a/xen/arch/x86/psr.c
>> +++ b/xen/arch/x86/psr.c
>> @@ -35,6 +35,9 @@
>>  #define PSR_CAT        (1<<1)
>>  #define PSR_CDP        (1<<2)
>>  
>> +#define CAT_CBM_LEN_MASK 0x1f
>> +#define CAT_COS_MAX_MASK 0xffff
>> +
>>  /*
>>   * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
>>   * the MSRs range from 0C90H through 0D0FH (inclusive), enables support for
>> @@ -127,6 +130,13 @@ struct feat_node {
>>      struct list_head list;
>>  };
>>  
>> +struct cpuid_leaf_regs {
>> +    unsigned int eax;
>> +    unsigned int ebx;
>> +    unsigned int ecx;
>> +    unsigned int edx;
>> +};
> Could you use 'struct cpuid_leaf' in x86_emulate.h ?
>
> It would only require "#include <asm/x86_emulate.h>" I believe.

I expect my cpuid_leaf work is newer than this series.

If there are going to be external users, I would also recommend moving
the cpuid_{,count}_leaf() helpers to static inlines in cpuid.h.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures.
  2017-01-31 10:10     ` Jan Beulich
@ 2017-01-31 14:12       ` Konrad Rzeszutek Wilk
  2017-01-31 15:07         ` Jan Beulich
  0 siblings, 1 reply; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-31 14:12 UTC (permalink / raw)
  To: Jan Beulich
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	Yi Sun, mengxu, xen-devel, chao.p.peng

On Tue, Jan 31, 2017 at 03:10:14AM -0700, Jan Beulich wrote:
> >>> On 30.01.17 at 23:20, <konrad.wilk@oracle.com> wrote:
> >> --- a/xen/arch/x86/psr.c
> >> +++ b/xen/arch/x86/psr.c
> >> @@ -17,12 +17,116 @@
> >>  #include <xen/cpu.h>
> >>  #include <xen/err.h>
> >>  #include <xen/sched.h>
> >> +#include <xen/list.h>
> >>  #include <asm/psr.h>
> >>  
> >> +/*
> >> + * Terminology:
> >> + * - CAT         Cache Allocation Technology
> >> + * - CBM         Capacity BitMasks
> >> + * - CDP         Code and Data Prioritization
> >> + * - COS/CLOS    Class of Service. Also mean COS registers.
> >> + * - COS_MAX     Max number of COS for the feature (minus 1)
> >> + * - MSRs        Machine Specific Registers
> >> + * - PSR         Intel Platform Shared Resource
> >> + */
> >> +
> >>  #define PSR_CMT        (1<<0)
> >>  #define PSR_CAT        (1<<1)
> >>  #define PSR_CDP        (1<<2)
> >>  
> >> +/*
> >> + * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
> >> + * the MSRs range from 0C90H through 0D0FH (inclusive), enables support for
> > 
> > s/enables/enable/
> >> + * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
> >> + *
> >> + * The MSRs range from 0D10H through 0D4FH (inclusive), enables support for
> > 
> > s/enables/enable/
> 
> For both of them - why? Both talk about a (single) range.

The 's' on the verb makes it singular. The MSRs is plural.

You don't do plural verb and plural subject - so the 's' has to be either
on the MSRs (and 'enable'), or you move the s from MSRs ('MSR' and 'enables').

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 04/24] x86: refactor psr: implement CPU init and free flow.
  2017-01-31 10:14     ` Jan Beulich
@ 2017-01-31 14:13       ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-31 14:13 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Yi Sun, xen-devel, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, chao.p.peng, wei.liu2

On Tue, Jan 31, 2017 at 03:14:25AM -0700, Jan Beulich wrote:
> >>> On 31.01.17 at 03:44, <konrad.wilk@oracle.com> wrote:
> > On Thu, Jan 19, 2017 at 02:01:06PM +0800, Yi Sun wrote:
> >> @@ -127,6 +130,13 @@ struct feat_node {
> >>      struct list_head list;
> >>  };
> >>  
> >> +struct cpuid_leaf_regs {
> >> +    unsigned int eax;
> >> +    unsigned int ebx;
> >> +    unsigned int ecx;
> >> +    unsigned int edx;
> >> +};
> > 
> > Could you use 'struct cpuid_leaf' in x86_emulate.h ?
> > 
> > It would only require "#include <asm/x86_emulate.h>" I believe.
> 
> Indeed - I recall specifically having asked for that.
> 
> >>  static int psr_cpu_prepare(unsigned int cpu)
> >>  {
> >> +    if ( !socket_info )
> >> +        return 0;
> >> +
> >> +    /* Malloc memory for the global feature head here. */
> >> +    if ( feat_l3_cat == NULL &&
> >> +         (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
> > 
> > /me scratches his head.
> > 
> > Lets say we have a two socket machine. Each socket has
> > two CPUs (so total of 4 CPUs).
> > 
> > On CPU0  (in psr_presmp_init) it allocates feat_l3_cat. Then
> > later in (in psr_presmp_init) you call psr_cpu_init which
> > sets feat_l3_cat to NULL (and 'feat' is feat_l3_cat).
> > 
> > When CPU1 starts up then, psr_cpu_prepare is called
> > which means we allocate a new feat_l3_cat. 'cpu_init_work'
> > short-circuits (as info->feat_mask has a value) and does not
> > do anything.
> > 
> > When CPU2 (socket two), starts up then feat_l3_cat (as 'feat')
> > is used (in psr_cpu_init and again sets feat_l3_cat to NULL).
> > 
> > The last CPU3 (socket #2) starts up, and since feat_l3_cat is
> > NULL - it is allocated. But in 'cpu_init_work' it short-circuits
> > so we don't use third allocated 'feat_l3_cat' .
> > 
> > And we have an 'feat_l3_cat' that is not used for anything..
> > 
> > In other words - a memory leak. Granted a very small one.
> 
> I did recommend to do it that way, to simplify things a little.

Aha! In which case perhaps a little comment saying that it is
OK to have this one memory leak to make the code simpler.


> 
> Jan
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures.
  2017-01-31 14:12       ` Konrad Rzeszutek Wilk
@ 2017-01-31 15:07         ` Jan Beulich
  2017-01-31 17:32           ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 55+ messages in thread
From: Jan Beulich @ 2017-01-31 15:07 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	Yi Sun, mengxu, xen-devel, chao.p.peng

>>> On 31.01.17 at 15:12, <konrad.wilk@oracle.com> wrote:
> On Tue, Jan 31, 2017 at 03:10:14AM -0700, Jan Beulich wrote:
>> >>> On 30.01.17 at 23:20, <konrad.wilk@oracle.com> wrote:
>> >> --- a/xen/arch/x86/psr.c
>> >> +++ b/xen/arch/x86/psr.c
>> >> @@ -17,12 +17,116 @@
>> >>  #include <xen/cpu.h>
>> >>  #include <xen/err.h>
>> >>  #include <xen/sched.h>
>> >> +#include <xen/list.h>
>> >>  #include <asm/psr.h>
>> >>  
>> >> +/*
>> >> + * Terminology:
>> >> + * - CAT         Cache Allocation Technology
>> >> + * - CBM         Capacity BitMasks
>> >> + * - CDP         Code and Data Prioritization
>> >> + * - COS/CLOS    Class of Service. Also mean COS registers.
>> >> + * - COS_MAX     Max number of COS for the feature (minus 1)
>> >> + * - MSRs        Machine Specific Registers
>> >> + * - PSR         Intel Platform Shared Resource
>> >> + */
>> >> +
>> >>  #define PSR_CMT        (1<<0)
>> >>  #define PSR_CAT        (1<<1)
>> >>  #define PSR_CDP        (1<<2)
>> >>  
>> >> +/*
>> >> + * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
>> >> + * the MSRs range from 0C90H through 0D0FH (inclusive), enables support for
>> > 
>> > s/enables/enable/
>> >> + * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
>> >> + *
>> >> + * The MSRs range from 0D10H through 0D4FH (inclusive), enables support for
>> > 
>> > s/enables/enable/
>> 
>> For both of them - why? Both talk about a (single) range.
> 
> The 's' on the verb makes it singular. The MSRs is plural.
> 
> You don't do plural verb and plural subject - so the 's' has to be either
> on the MSRs (and 'enable'), or you move the s from MSRs ('MSR' and 
> 'enables').

But that's my point, just that the subject is "range" here. Otoh it
could also be "The MSRs ranging from ... enable support".

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures.
  2017-01-31 15:07         ` Jan Beulich
@ 2017-01-31 17:32           ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-31 17:32 UTC (permalink / raw)
  To: Jan Beulich
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	Yi Sun, mengxu, xen-devel, chao.p.peng

On Tue, Jan 31, 2017 at 08:07:17AM -0700, Jan Beulich wrote:
> >>> On 31.01.17 at 15:12, <konrad.wilk@oracle.com> wrote:
> > On Tue, Jan 31, 2017 at 03:10:14AM -0700, Jan Beulich wrote:
> >> >>> On 30.01.17 at 23:20, <konrad.wilk@oracle.com> wrote:
> >> >> --- a/xen/arch/x86/psr.c
> >> >> +++ b/xen/arch/x86/psr.c
> >> >> @@ -17,12 +17,116 @@
> >> >>  #include <xen/cpu.h>
> >> >>  #include <xen/err.h>
> >> >>  #include <xen/sched.h>
> >> >> +#include <xen/list.h>
> >> >>  #include <asm/psr.h>
> >> >>  
> >> >> +/*
> >> >> + * Terminology:
> >> >> + * - CAT         Cache Allocation Technology
> >> >> + * - CBM         Capacity BitMasks
> >> >> + * - CDP         Code and Data Prioritization
> >> >> + * - COS/CLOS    Class of Service. Also mean COS registers.
> >> >> + * - COS_MAX     Max number of COS for the feature (minus 1)
> >> >> + * - MSRs        Machine Specific Registers
> >> >> + * - PSR         Intel Platform Shared Resource
> >> >> + */
> >> >> +
> >> >>  #define PSR_CMT        (1<<0)
> >> >>  #define PSR_CAT        (1<<1)
> >> >>  #define PSR_CDP        (1<<2)
> >> >>  
> >> >> +/*
> >> >> + * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
> >> >> + * the MSRs range from 0C90H through 0D0FH (inclusive), enables support for
> >> > 
> >> > s/enables/enable/
> >> >> + * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
> >> >> + *
> >> >> + * The MSRs range from 0D10H through 0D4FH (inclusive), enables support for
> >> > 
> >> > s/enables/enable/
> >> 
> >> For both of them - why? Both talk about a (single) range.
> > 
> > The 's' on the verb makes it singular. The MSRs is plural.
> > 
> > You don't do plural verb and plural subject - so the 's' has to be either
> > on the MSRs (and 'enable'), or you move the s from MSRs ('MSR' and 
> > 'enables').
> 
> But that's my point, just that the subject is "range" here. Otoh it
> could also be "The MSRs ranging from ... enable support".

Right. That would be good too.
> 
> Jan
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 05/24] x86: refactor psr: implement Domain init/free and schedule flows.
  2017-01-19  6:01 ` [PATCH RESEND v5 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
@ 2017-01-31 19:52   ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-31 19:52 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Thu, Jan 19, 2017 at 02:01:07PM +0800, Yi Sun wrote:
> This patch implements the Domain init/free and schedule flows.
> 
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> v5:
>     - define macro 'PSR_ASSOC_REG_POS' to replace integer 32.
>     - rename 'l3_cat_get_max_cos_max' to 'l3_cat_get_cos_max'.
>     - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
> ---
>  xen/arch/x86/psr.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 61 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index e9dc07a..7f06235 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -50,6 +50,8 @@
>   */
>  #define MAX_COS_REG_CNT  128
>  
> +#define PSR_ASSOC_REG_POS 32

Perhaps PSR_ASSOC_REG_SHIFT ?

Otherwise: Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 06/24] x86: refactor psr: implement get hw info flow.
  2017-01-19  6:01 ` [PATCH RESEND v5 06/24] x86: refactor psr: implement get hw info flow Yi Sun
@ 2017-01-31 20:17   ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-31 20:17 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

.snip..
> +static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> +{
> +    enum psr_feat_type feat_type;
> +
> +    /* Judge if feature is enabled. */
> +    switch ( type ) {

The { should be on its own line.

> +    case PSR_CBM_TYPE_L3:
> +        feat_type = PSR_SOCKET_L3_CAT;
> +        break;
> +    default:
> +        feat_type = 0xFFFF;

Perhaps have an extra field in psr_feat_type titled 'PSR_SOCKET_UNKNOWN = 0xFFFF' 
and use that ?

> +        break;
> +    }
> +
> +    return feat_type;
> +}
> +
>  /* L3 CAT functions implementation. */
>  static void l3_cat_init_feature(struct cpuid_leaf_regs regs,
>                                  struct feat_node *feat,
> @@ -218,8 +238,22 @@ static unsigned int l3_cat_get_cos_max(const struct feat_node *feat)
>      return feat->info.l3_cat_info.cos_max;
>  }
>  
> +static bool l3_cat_get_feat_info(const struct feat_node *feat,
> +                                 uint32_t data[], unsigned int array_len)
> +{
> +    if ( !data || 3 > array_len )
> +        return false;
> +
> +    data[CBM_LEN] = feat->info.l3_cat_info.cbm_len;
> +    data[COS_MAX] = feat->info.l3_cat_info.cos_max;
> +    data[PSR_FLAG] = 0;
> +
> +    return true;
> +}
> +
>  static const struct feat_ops l3_cat_ops = {
>      .get_cos_max = l3_cat_get_cos_max,
> +    .get_feat_info = l3_cat_get_feat_info,
>  };
>  
>  static void __init parse_psr_bool(char *s, char *value, char *feature,
> @@ -425,10 +459,43 @@ void psr_ctxt_switch_to(struct domain *d)
>      }
>  }
>  
> -int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
> -                        uint32_t *cos_max, uint32_t *flags)
> +static struct psr_socket_info *get_socket_info(unsigned int socket)
>  {
> -    return 0;
> +    if ( !socket_info )
> +        return ERR_PTR(-ENODEV);
> +
> +    if ( socket >= nr_sockets )
> +        return ERR_PTR(-ENOTSOCK);

<chuckles>

I think ENXIO is more appropiate. Or ERANGE

> +
> +    if ( !socket_info[socket].feat_mask )
> +        return ERR_PTR(-ENOENT);
> +
> +    return socket_info + socket;
> +}
> +
> +int psr_get_info(unsigned int socket, enum cbm_type type,
> +                 uint32_t data[], unsigned int array_len)
> +{
> +    const struct psr_socket_info *info = get_socket_info(socket);
> +    const struct feat_node *feat;
> +    enum psr_feat_type feat_type;
> +
> +    if ( IS_ERR(info) )
> +        return PTR_ERR(info);
> +
> +    feat_type = psr_cbm_type_to_feat_type(type);
> +    list_for_each_entry(feat, &info->feat_list, list)
> +    {
> +        if ( feat->feature != feat_type )
> +            continue;
> +
> +        if ( feat->ops.get_feat_info(feat, data, array_len) )
> +            return 0;
> +        else
> +            return -EINVAL;
> +    }
> +
> +    return -ENOENT;
>  }
>  
>  int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
> index 14e7dc7..d90db78 100644
> --- a/xen/arch/x86/sysctl.c
> +++ b/xen/arch/x86/sysctl.c
> @@ -176,15 +176,19 @@ long arch_do_sysctl(
>          switch ( sysctl->u.psr_cat_op.cmd )
>          {
>          case XEN_SYSCTL_PSR_CAT_get_l3_info:
> -            ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
> -                                      &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
> -                                      &sysctl->u.psr_cat_op.u.l3_info.cos_max,
> -                                      &sysctl->u.psr_cat_op.u.l3_info.flags);
> +        {
> +            uint32_t data[3];
> +            ret = psr_get_info(sysctl->u.psr_cat_op.target,
> +                               PSR_CBM_TYPE_L3, data, 3);
> +
> +            sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[CBM_LEN];
> +            sysctl->u.psr_cat_op.u.l3_info.cos_max = data[COS_MAX];
> +            sysctl->u.psr_cat_op.u.l3_info.flags   = data[PSR_FLAG];
>  
>              if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
>                  ret = -EFAULT;
>              break;
> -
> +        }
>          default:
>              ret = -EOPNOTSUPP;
>              break;
> diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
> index 57f47e9..e3b18bc 100644
> --- a/xen/include/asm-x86/psr.h
> +++ b/xen/include/asm-x86/psr.h
> @@ -33,6 +33,11 @@
>  /* L3 CDP Enable bit*/
>  #define PSR_L3_QOS_CDP_ENABLE_BIT       0x0
>  
> +/* Used by psr_get_info() */
> +#define CBM_LEN  0
> +#define COS_MAX  1
> +#define PSR_FLAG 2

Odd. Looking at that file I see those #define but all of them are on
an odd spacing. But that was with previous commits that are in.

Perhaps put them all on the same column?

> +
>  struct psr_cmt_l3 {
>      unsigned int features;
>      unsigned int upscaling_factor;
> @@ -63,8 +68,8 @@ int psr_alloc_rmid(struct domain *d);
>  void psr_free_rmid(struct domain *d);
>  void psr_ctxt_switch_to(struct domain *d);
>  
> -int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
> -                        uint32_t *cos_max, uint32_t *flags);
> +int psr_get_info(unsigned int socket, enum cbm_type type,
> +                 uint32_t data[], unsigned int array_len);
>  int psr_get_l3_cbm(struct domain *d, unsigned int socket,
>                     uint64_t *cbm, enum cbm_type type);
>  int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> -- 
> 1.9.1
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 07/24] x86: refactor psr: implement get value flow.
  2017-01-19  6:01 ` [PATCH RESEND v5 07/24] x86: refactor psr: implement get value flow Yi Sun
@ 2017-01-31 20:29   ` Konrad Rzeszutek Wilk
  2017-02-07  2:47     ` Yi Sun
  0 siblings, 1 reply; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-31 20:29 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Thu, Jan 19, 2017 at 02:01:09PM +0800, Yi Sun wrote:
> This patch implements get value flow including L3 CAT callback
> function.
> 
> It also changes domctl interface to make it more general.
> 
> With this patch, 'psr-cat-show' can work for L3 CAT.

Could you add:

"but not for L3 code/data which is implemented in patch
titled XYZ" ?

> 
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> ---
> v5:
>     - rename 'dat[]' to 'data[]'
>     - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
>     - check if feature type match in caller of feature callback function.
> ---
>  xen/arch/x86/domctl.c     | 18 +++++++++---------
>  xen/arch/x86/psr.c        | 41 ++++++++++++++++++++++++++++++++++++++---
>  xen/include/asm-x86/psr.h |  4 ++--
>  3 files changed, 49 insertions(+), 14 deletions(-)
> 
> diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
> index ab141b1..11d2127 100644
> --- a/xen/arch/x86/domctl.c
> +++ b/xen/arch/x86/domctl.c
> @@ -1383,23 +1383,23 @@ long arch_do_domctl(
>              break;
>  
>          case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
> -            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
> -                                 &domctl->u.psr_cat_op.data,
> -                                 PSR_CBM_TYPE_L3);
> +            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
> +                              &domctl->u.psr_cat_op.data,
> +                              PSR_CBM_TYPE_L3);
>              copyback = 1;
>              break;
>  
>          case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
> -            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
> -                                 &domctl->u.psr_cat_op.data,
> -                                 PSR_CBM_TYPE_L3_CODE);
> +            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
> +                              &domctl->u.psr_cat_op.data,
> +                              PSR_CBM_TYPE_L3_CODE);
>              copyback = 1;
>              break;
>  
>          case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
> -            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
> -                                 &domctl->u.psr_cat_op.data,
> -                                 PSR_CBM_TYPE_L3_DATA);
> +            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
> +                              &domctl->u.psr_cat_op.data,
> +                              PSR_CBM_TYPE_L3_DATA);
>              copyback = 1;
>              break;
>  
> diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
> index 319bfcc..3cbb60c 100644
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -112,6 +112,9 @@ struct feat_ops {
>      /* get_feat_info is used to get feature HW info. */
>      bool (*get_feat_info)(const struct feat_node *feat,
>                            uint32_t data[], unsigned int array_len);
> +    /* get_val is used to get feature COS register value. */
> +    bool (*get_val)(const struct feat_node *feat, unsigned int cos,
> +                    enum cbm_type type, uint64_t *val);
>  };
>  
>  /*
> @@ -251,9 +254,22 @@ static bool l3_cat_get_feat_info(const struct feat_node *feat,
>      return true;
>  }
>  
> +static bool l3_cat_get_val(const struct feat_node *feat, unsigned int cos,
> +                           enum cbm_type type, uint64_t *val)
> +{
> +    if ( cos > feat->info.l3_cat_info.cos_max )
> +        /* Use default value. */
> +        cos = 0;
> +
> +    *val =  feat->cos_reg_val[cos];

Extra space there. No need for it.
> +
> +    return true;
> +}
> +
>  static const struct feat_ops l3_cat_ops = {
>      .get_cos_max = l3_cat_get_cos_max,
>      .get_feat_info = l3_cat_get_feat_info,
> +    .get_val = l3_cat_get_val,
>  };
>  
>  static void __init parse_psr_bool(char *s, char *value, char *feature,
> @@ -498,10 +514,29 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
>      return -ENOENT;
>  }
>  
> -int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> -                   uint64_t *cbm, enum cbm_type type)
> +int psr_get_val(struct domain *d, unsigned int socket,
> +                uint64_t *val, enum cbm_type type)
>  {
> -    return 0;
> +    const struct psr_socket_info *info = get_socket_info(socket);
> +    unsigned int cos = d->arch.psr_cos_ids[socket];
> +    const struct feat_node *feat;
> +    enum psr_feat_type feat_type;
> +
> +    if ( IS_ERR(info) )
> +        return PTR_ERR(info);
> +
> +    feat_type = psr_cbm_type_to_feat_type(type);
> +    list_for_each_entry(feat, &info->feat_list, list)
> +    {
> +        if ( feat->feature != feat_type )
> +            continue;
> +
> +        if ( feat->ops.get_val(feat, cos, type, val) )
> +            /* Found */

No need. The 'psr_get_info' does not have this.

> +            return 0;
> +    }
> +
> +    return -ENOENT;

This function looks quite similar to 'psr_get_info'.

Perhaps it may make sense to have an common one that has an
extra argument (whether to call get_val or get_feat_info)?

And then psr_get_val and psr_get_info can call in this common
code with this extra argument attached?

>  }
>  
>  int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
> index e3b18bc..d50e359 100644
> --- a/xen/include/asm-x86/psr.h
> +++ b/xen/include/asm-x86/psr.h
> @@ -70,8 +70,8 @@ void psr_ctxt_switch_to(struct domain *d);
>  
>  int psr_get_info(unsigned int socket, enum cbm_type type,
>                   uint32_t data[], unsigned int array_len);
> -int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> -                   uint64_t *cbm, enum cbm_type type);
> +int psr_get_val(struct domain *d, unsigned int socket,
> +                uint64_t *val, enum cbm_type type);
>  int psr_set_l3_cbm(struct domain *d, unsigned int socket,
>                     uint64_t cbm, enum cbm_type type);
>  
> -- 
> 1.9.1
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 09/24] x86: refactor psr: set value: assemble features value array.
  2017-01-19  6:01 ` [PATCH RESEND v5 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
@ 2017-01-31 20:57   ` Konrad Rzeszutek Wilk
  2017-02-07  2:51     ` Yi Sun
  0 siblings, 1 reply; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-01-31 20:57 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

>  static int assemble_val_array(uint64_t *val,
> @@ -550,7 +641,25 @@ static int assemble_val_array(uint64_t *val,
>                                const struct psr_socket_info *info,
>                                unsigned int old_cos)
>  {
> -    return -EINVAL;
> +    const struct feat_node *feat;
> +    int ret;
> +    uint64_t *val_tmp = val;
> +
> +    if ( !val )
> +        return -EINVAL;
> +
> +    /* Get all features current values according to old_cos. */
> +    list_for_each_entry(feat, &info->feat_list, list)
> +    {
> +        /* value getting order is same as feature list */
> +        ret = feat->ops.get_old_val(val_tmp, feat, old_cos);

Shouldn't we check ret for negative values?
> +
> +        val_tmp += feat->ops.get_cos_num(feat);
> +        if ( val_tmp - val > array_len)
> +            return -EINVAL;


Perhaps: ENOSPC ?

Also this function does do any assembling. It just verifies.
Perhaps other patches add extra work here? In which case you may
want to mention that in the commit description.

> +    }
> +
> +    return 0;
>  }
>  
>  static int set_new_val_to_array(uint64_t *val,
> @@ -560,7 +669,37 @@ static int set_new_val_to_array(uint64_t *val,
>                                  enum cbm_type type,
>                                  uint64_t m)
>  {
> -    return -EINVAL;
> +    const struct feat_node *feat;
> +    int ret;
> +    uint64_t *val_tmp = val;
> +
> +    /* Set new value into array according to feature's position in array. */
> +    list_for_each_entry(feat, &info->feat_list, list)
> +    {
> +        if ( feat->feature != feat_type )
> +        {
> +            val_tmp += feat->ops.get_cos_num(feat);
> +            if ( val_tmp - val > array_len)
> +                return -EINVAL;

-ENOSPC?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document
  2017-01-30 18:10   ` Konrad Rzeszutek Wilk
  2017-01-30 20:39     ` Konrad Rzeszutek Wilk
@ 2017-02-04  7:06     ` Yi Sun
  1 sibling, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-02-04  7:06 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

Thanks a lot for reviewing the patches! Sorry for late to reply. I am on
vacation for Chinese New Year now. :)

On 17-01-30 13:10:33, Konrad Rzeszutek Wilk wrote:
> On Thu, Jan 19, 2017 at 02:01:03PM +0800, Yi Sun wrote:
> > +   2. Feature list node
> > +
> > +      ```
> > +      struct feat_node {
> > +          enum psr_feat_type feature;
> > +          struct feat_ops ops;
> > +          struct psr_cat_hw_info info;
> > +          uint64_t cos_reg_val[MAX_COS_REG_NUM];
> > +          struct list_head list;
> > +      };
> > +      ```
> > +
> > +      When a PSR enforcement feature is enabled, it will be added into a
> > +      feature list. The head of the list is created in psr initialization.
> > +
> > +      - Member `feature`
> > +
> > +        `feature` is an integer number, to indicate which feature the list entry
> > +        corresponds to.
> > +
> > +      - Member `ops`
> > +
> > +        `ops` maintains a callback function list of the feature. It will be introduced
> > +        in details later.
> 
> ??? Like by each patch to this document?
> 
Do you mean 'Like by "which" patch to this document'? Shall I input the patch title which

BRs,
Sun Yi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document
  2017-01-30 20:39     ` Konrad Rzeszutek Wilk
@ 2017-02-04  7:24       ` Yi Sun
  0 siblings, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-02-04  7:24 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On 17-01-30 15:39:28, Konrad Rzeszutek Wilk wrote:
> > > +# Testing
> > > +
> > > +L2 CAT uses same xl interfaces as L3 CAT/CDP. So, we can execute these
> > > +commands to verify L2 CAT and L3 CAT/CDP on different HWs support them.
> > > +
> > > +For example:
> > > +    root@:~$ xl psr-hwinfo --cat
> > > +    Cache Allocation Technology (CAT): L2
> > > +    Socket ID       : 0
> > > +    Maximum COS     : 3
> > > +    CBM length      : 8
> > > +    Default CBM     : 0xff
> > > +
> > > +    root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
> > > +
> > > +    root@:~$ xl psr-cat-show -l2 1
> > > +    Socket ID       : 0
> > > +    Default CBM     : 0xff
> > > +       ID                     NAME             CBM
> > > +        1                 ubuntu14            0x7f
> > > +
> 
> I tried finding the Intel SDM (December 2016) what the format
> of CBM is as the value '0x7f' does not really mean much to me.
> 
> Right above Figure 17.28 it says:
> 
> "Figure 17-27 also shows three examples of sets of Cache Capacity Bitmasks. For simplicity these are represented
> as 8-bit vectors, though this may vary _depending on the implementation and how the mask is mapped to the avail-
> able cache capacity._"
> 
> 
> So in other words - not documented. 
> 
> Then later is says:
> 
> " Rather, this is a convenient manner to represent capacity,
> overlap and isolation of cache space. For example, executing a POPCNT instruction (population count of set bits) on
> the capacity bitmask can provide the fraction of cache space that a class of service can allocate into."
>
Right after this sentence, the spec mentions "In addition to the fraction, the exact location of the
bits also shows whether the class of service overlaps with other classes of service or is entirely
isolated in terms of cache space used." 

For example:
For domain 1, assign 0xF to it.
For domain 2, assign 0x30 to it.
That means, the cache spaces that domain 1 and domain 2 are using are isolated.

For domain 3, assgin 0x3 to it.
That means, domain 3 cache space overlaps domain 1.

So, we may not just input how much space to be assigned to one domain. :)

> OK, so _can_ (but not _MUST_), so again implementation specific - and
> can provide a fraction of cache space.
> 
> Which would imply that the values could be:
> 
> 0x0F - half of L2
> 0x03 - quarter of L2
> 
> Is there some other documentation that explains this in more details?
> 
> If it is like I mentioned would it make sense to have 'xl' be capable
> of dealing with string values? such as "three-quarters", "half", etc
> and then set it? Or percentage and map that to the correct value?
> 
> Like:
> 
> xl psr-cat-cbm-set -l2 ubuntu14 50%
> 
> would be quite obvious instead of say 0x0F?
> 
> Thanks.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures.
  2017-01-30 22:20   ` Konrad Rzeszutek Wilk
  2017-01-31 10:10     ` Jan Beulich
@ 2017-02-05  1:53     ` Yi Sun
  1 sibling, 0 replies; 55+ messages in thread
From: Yi Sun @ 2017-02-05  1:53 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On 17-01-30 17:20:39, Konrad Rzeszutek Wilk wrote:
> > +struct psr_socket_info {
> > +    /*
> > +     * bit 0:   L3 CAT
> > +     * bit 1:   L3 CDP
> > +     * bit 2:   L2 CAT
> 
> Why not an enum? I am going to assume it is due to you programming
> this in the MSRs. If so, I would recommend you have #define instead
> of a comment.
> 
> #define L3_CAT (1U<<0)
> #define L3_CDP (1U<<1)
> 
> . and so on..
> 
> > +     */
> > +    unsigned int feat_mask;
> > +    unsigned int nr_feat;
> > +    struct list_head feat_list;
> > +    unsigned int cos_ref[MAX_COS_REG_CNT];
> > +    spinlock_t ref_lock;
> > +};
> > +
> > +enum psr_feat_type {
> > +    PSR_SOCKET_L3_CAT = 0,
> > +    PSR_SOCKET_L3_CDP,
> > +    PSR_SOCKET_L2_CAT,
> 
> Is there particular mapping between these and the feat_mask bit mask
> values?
> 
> It kind of begs to be combined (unless you really need the 'feat_mask'
> to be of specific order - in which case make sure you have BUILD_BUG_ON
> to make sure nobody moves the #define values around - or put a comment
> saying that you need the specific order of bits).
> 
Like what you mentioned in patch 5 review comments, the feat_mask bit mask
reuses 'enum psr_feat_type'. I will add comments to describe this. Thanks!

BRs,
Sun Yi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 07/24] x86: refactor psr: implement get value flow.
  2017-01-31 20:29   ` Konrad Rzeszutek Wilk
@ 2017-02-07  2:47     ` Yi Sun
  2017-02-07 13:56       ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-02-07  2:47 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

Hi,

Thanks for reviewing! I agree with your comments except below one. Could you
please check my response?

On 17-01-31 15:29:34, Konrad Rzeszutek Wilk wrote:
> On Thu, Jan 19, 2017 at 02:01:09PM +0800, Yi Sun wrote:
> > +int psr_get_val(struct domain *d, unsigned int socket,
> > +                uint64_t *val, enum cbm_type type)
> >  {
> > -    return 0;
> > +    const struct psr_socket_info *info = get_socket_info(socket);
> > +    unsigned int cos = d->arch.psr_cos_ids[socket];
> > +    const struct feat_node *feat;
> > +    enum psr_feat_type feat_type;
> > +
> > +    if ( IS_ERR(info) )
> > +        return PTR_ERR(info);
> > +
> > +    feat_type = psr_cbm_type_to_feat_type(type);
> > +    list_for_each_entry(feat, &info->feat_list, list)
> > +    {
> > +        if ( feat->feature != feat_type )
> > +            continue;
> > +
> > +        if ( feat->ops.get_val(feat, cos, type, val) )
> > +            /* Found */
> 
> No need. The 'psr_get_info' does not have this.
> 
> > +            return 0;
> > +    }
> > +
> > +    return -ENOENT;
> 
> This function looks quite similar to 'psr_get_info'.
> 
> Perhaps it may make sense to have an common one that has an
> extra argument (whether to call get_val or get_feat_info)?
> 
> And then psr_get_val and psr_get_info can call in this common
> code with this extra argument attached?
>
Yes, the both functions are almost same. But I feel not good to combine them to
one function. I think it makes the interface not explicit. As there are only 3
interfaces exposed by psr.c, I want to keep current implementation. Is that
acceptable to you?

Thanks,
Sun Yi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 09/24] x86: refactor psr: set value: assemble features value array.
  2017-01-31 20:57   ` Konrad Rzeszutek Wilk
@ 2017-02-07  2:51     ` Yi Sun
  2017-02-07 13:57       ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 55+ messages in thread
From: Yi Sun @ 2017-02-07  2:51 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On 17-01-31 15:57:26, Konrad Rzeszutek Wilk wrote:
> >  static int assemble_val_array(uint64_t *val,
> > @@ -550,7 +641,25 @@ static int assemble_val_array(uint64_t *val,
> >                                const struct psr_socket_info *info,
> >                                unsigned int old_cos)
> >  {
> > -    return -EINVAL;
> > +    const struct feat_node *feat;
> > +    int ret;
> > +    uint64_t *val_tmp = val;
> > +
> > +    if ( !val )
> > +        return -EINVAL;
> > +
> > +    /* Get all features current values according to old_cos. */
> > +    list_for_each_entry(feat, &info->feat_list, list)
> > +    {
> > +        /* value getting order is same as feature list */
> > +        ret = feat->ops.get_old_val(val_tmp, feat, old_cos);
> 
> Shouldn't we check ret for negative values?
> > +
> > +        val_tmp += feat->ops.get_cos_num(feat);
> > +        if ( val_tmp - val > array_len)
> > +            return -EINVAL;
> 
> 
> Perhaps: ENOSPC ?
> 
> Also this function does do any assembling. It just verifies.
> Perhaps other patches add extra work here? In which case you may
> want to mention that in the commit description.
> 
Array "assembling" is done here. But maybe the "assembling" is not accurate.
In this function, I want to get all features old values and put them into
the array according to every feature's position in feature list.

Do you have more accurate word than "assemble"? Thanks!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 07/24] x86: refactor psr: implement get value flow.
  2017-02-07  2:47     ` Yi Sun
@ 2017-02-07 13:56       ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-07 13:56 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Tue, Feb 07, 2017 at 10:47:01AM +0800, Yi Sun wrote:
> Hi,
> 
> Thanks for reviewing! I agree with your comments except below one. Could you
> please check my response?
> 
> On 17-01-31 15:29:34, Konrad Rzeszutek Wilk wrote:
> > On Thu, Jan 19, 2017 at 02:01:09PM +0800, Yi Sun wrote:
> > > +int psr_get_val(struct domain *d, unsigned int socket,
> > > +                uint64_t *val, enum cbm_type type)
> > >  {
> > > -    return 0;
> > > +    const struct psr_socket_info *info = get_socket_info(socket);
> > > +    unsigned int cos = d->arch.psr_cos_ids[socket];
> > > +    const struct feat_node *feat;
> > > +    enum psr_feat_type feat_type;
> > > +
> > > +    if ( IS_ERR(info) )
> > > +        return PTR_ERR(info);
> > > +
> > > +    feat_type = psr_cbm_type_to_feat_type(type);
> > > +    list_for_each_entry(feat, &info->feat_list, list)
> > > +    {
> > > +        if ( feat->feature != feat_type )
> > > +            continue;
> > > +
> > > +        if ( feat->ops.get_val(feat, cos, type, val) )
> > > +            /* Found */
> > 
> > No need. The 'psr_get_info' does not have this.
> > 
> > > +            return 0;
> > > +    }
> > > +
> > > +    return -ENOENT;
> > 
> > This function looks quite similar to 'psr_get_info'.
> > 
> > Perhaps it may make sense to have an common one that has an
> > extra argument (whether to call get_val or get_feat_info)?
> > 
> > And then psr_get_val and psr_get_info can call in this common
> > code with this extra argument attached?
> >
> Yes, the both functions are almost same. But I feel not good to combine them to
> one function. I think it makes the interface not explicit. As there are only 3
> interfaces exposed by psr.c, I want to keep current implementation. Is that
> acceptable to you?

Keep the interface as is.

You would have _three_ functions:

psr_get_val
psr_get_info

and the one both of them would call which would be called:

__psr_get

which would do the heavy lifting.
> 
> Thanks,
> Sun Yi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 09/24] x86: refactor psr: set value: assemble features value array.
  2017-02-07  2:51     ` Yi Sun
@ 2017-02-07 13:57       ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 55+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-07 13:57 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

On Tue, Feb 07, 2017 at 10:51:27AM +0800, Yi Sun wrote:
> On 17-01-31 15:57:26, Konrad Rzeszutek Wilk wrote:
> > >  static int assemble_val_array(uint64_t *val,
> > > @@ -550,7 +641,25 @@ static int assemble_val_array(uint64_t *val,
> > >                                const struct psr_socket_info *info,
> > >                                unsigned int old_cos)
> > >  {
> > > -    return -EINVAL;
> > > +    const struct feat_node *feat;
> > > +    int ret;
> > > +    uint64_t *val_tmp = val;
> > > +
> > > +    if ( !val )
> > > +        return -EINVAL;
> > > +
> > > +    /* Get all features current values according to old_cos. */
> > > +    list_for_each_entry(feat, &info->feat_list, list)
> > > +    {
> > > +        /* value getting order is same as feature list */
> > > +        ret = feat->ops.get_old_val(val_tmp, feat, old_cos);
> > 
> > Shouldn't we check ret for negative values?
> > > +
> > > +        val_tmp += feat->ops.get_cos_num(feat);
> > > +        if ( val_tmp - val > array_len)
> > > +            return -EINVAL;
> > 
> > 
> > Perhaps: ENOSPC ?
> > 
> > Also this function does do any assembling. It just verifies.
> > Perhaps other patches add extra work here? In which case you may
> > want to mention that in the commit description.
> > 
> Array "assembling" is done here. But maybe the "assembling" is not accurate.
> In this function, I want to get all features old values and put them into
> the array according to every feature's position in feature list.

combine?
> 
> Do you have more accurate word than "assemble"? Thanks!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

* Re: [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document
  2017-01-22  2:15     ` Yi Sun
@ 2017-02-08  6:45       ` Tian, Kevin
  0 siblings, 0 replies; 55+ messages in thread
From: Tian, Kevin @ 2017-02-08  6:45 UTC (permalink / raw)
  To: Yi Sun
  Cc: wei.liu2, he.chen, andrew.cooper3, dario.faggioli, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel

> From: Yi Sun [mailto:yi.y.sun@linux.intel.com]
> Sent: Sunday, January 22, 2017 10:15 AM
> 
> > > +
> > > +     New option `-l` is added.
> > > +     `-l2`: Specify cbm for L2 cache.
> > > +     `-l3`: Specify cbm for L3 cache.
> > > +
> > > +     If neither `-l2` nor `-l3` is given, level 3 is the default option.
> > > +
> > > +  3. `psr-hwinfo [OPTIONS]`:
> > > +
> > > +     Show L2 & L3 CAT HW informations on every socket.
> >
> > informations -> information
> >
> > and which HW information? please elaborate.
> >
> > and where is the interface of setting COS for VCPU?
> >
> What do you mean 'the interface of setting COS for VCPU'? User can set CBM
> for one domain through 'psr-cat-cbm-set' now. Then, hypervisor will find
> a matched COS ID or pick an available COS ID for the domain. User does not
> need know which COS ID it is using.

possibly you want to explain above logic in this doc. I didn't see any
discussion of COS ID in the text.

> 
> > > +
> > > +* Multi-sockets
> > > +
> > > +  Different sockets may have different L2 CAT capability (e.g. max COS)
> > > +  although it is consistent on the same socket. So the capability of
> > > +  per-socket L2 CAT is specified.
> >
> > VCPU schedule design is important here. e.g. when migrating a VCPU
> > from socket 1 to socket 2 (socket 1 has max COS as 16 while socket
> > 2 as 8), will you prevent migration if VCPU has a COS (9)?
> >
> 'psr-cat-cbm-set' can set CBM for one domain per socket. On each socket, we
> maintain a COS array for all domains. One domain uses one COS at one time. One
> COS saves the CBM of to work. So, when a VCPU of the domain is migrated from
> socket 1 to socket 2, it follows configuration on socket 2. This mechanism has
> been implemented since L3 CAT. L2 CAT follows it.
> 
> E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets
> domain 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
> is migrated from socket 1 to 2, the COS ID used will be 7, that means 0x3f
> will be the CBM to work for this domain.

OK, so user will pick valid CBM on each socket for a specific domain. Also
please include such info in your next version.

Thanks
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 55+ messages in thread

end of thread, other threads:[~2017-02-08  6:46 UTC | newest]

Thread overview: 55+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-19  6:01 [PATCH RESEND v5 00/24] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 01/24] docs: create L2 Cache Allocation Technology (CAT) feature document Yi Sun
2017-01-20  9:39   ` Tian, Kevin
2017-01-22  2:15     ` Yi Sun
2017-02-08  6:45       ` Tian, Kevin
2017-01-30 18:10   ` Konrad Rzeszutek Wilk
2017-01-30 20:39     ` Konrad Rzeszutek Wilk
2017-02-04  7:24       ` Yi Sun
2017-02-04  7:06     ` Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 02/24] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
2017-01-30 22:05   ` Konrad Rzeszutek Wilk
2017-01-19  6:01 ` [PATCH RESEND v5 03/24] x86: refactor psr: implement main data structures Yi Sun
2017-01-30 22:20   ` Konrad Rzeszutek Wilk
2017-01-31 10:10     ` Jan Beulich
2017-01-31 14:12       ` Konrad Rzeszutek Wilk
2017-01-31 15:07         ` Jan Beulich
2017-01-31 17:32           ` Konrad Rzeszutek Wilk
2017-02-05  1:53     ` Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 04/24] x86: refactor psr: implement CPU init and free flow Yi Sun
2017-01-31  2:44   ` Konrad Rzeszutek Wilk
2017-01-31 10:14     ` Jan Beulich
2017-01-31 14:13       ` Konrad Rzeszutek Wilk
2017-01-31 10:53     ` Andrew Cooper
2017-01-19  6:01 ` [PATCH RESEND v5 05/24] x86: refactor psr: implement Domain init/free and schedule flows Yi Sun
2017-01-31 19:52   ` Konrad Rzeszutek Wilk
2017-01-19  6:01 ` [PATCH RESEND v5 06/24] x86: refactor psr: implement get hw info flow Yi Sun
2017-01-31 20:17   ` Konrad Rzeszutek Wilk
2017-01-19  6:01 ` [PATCH RESEND v5 07/24] x86: refactor psr: implement get value flow Yi Sun
2017-01-31 20:29   ` Konrad Rzeszutek Wilk
2017-02-07  2:47     ` Yi Sun
2017-02-07 13:56       ` Konrad Rzeszutek Wilk
2017-01-19  6:01 ` [PATCH RESEND v5 08/24] x86: refactor psr: set value: implement framework Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 09/24] x86: refactor psr: set value: assemble features value array Yi Sun
2017-01-31 20:57   ` Konrad Rzeszutek Wilk
2017-02-07  2:51     ` Yi Sun
2017-02-07 13:57       ` Konrad Rzeszutek Wilk
2017-01-19  6:01 ` [PATCH RESEND v5 10/24] x86: refactor psr: set value: implement cos finding flow Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 11/24] x86: refactor psr: set value: implement cos id picking flow Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 12/24] x86: refactor psr: set value: implement write msr flow Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 13/24] x86: refactor psr: implement CPU init and free flow for CDP Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 14/24] x86: refactor psr: implement get hw info " Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 15/24] x86: refactor psr: implement get value " Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 16/24] x86: refactor psr: implement set value callback functions " Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 17/24] x86: L2 CAT: implement CPU init and free flow Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 18/24] x86: L2 CAT: implement get hw info flow Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 19/24] x86: L2 CAT: implement get value flow Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 20/24] x86: L2 CAT: implement set " Yi Sun
2017-01-19  6:01 ` [PATCH RESEND v5 21/24] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
2017-01-27 15:18   ` Wei Liu
2017-01-19  6:01 ` [PATCH RESEND v5 22/24] tools: L2 CAT: support show cbm " Yi Sun
2017-01-27 15:18   ` Wei Liu
2017-01-19  6:01 ` [PATCH RESEND v5 23/24] tools: L2 CAT: support set " Yi Sun
2017-01-27 15:18   ` Wei Liu
2017-01-19  6:01 ` [PATCH RESEND v5 24/24] docs: add L2 CAT description in docs Yi Sun
2017-01-27 15:18   ` Wei Liu

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.