All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c
@ 2017-03-16 11:07 Yi Sun
  2017-03-16 11:07 ` [PATCH v9 01/25] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
                   ` (25 more replies)
  0 siblings, 26 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

Hi all,

We plan to bring a new PSR (Platform Shared Resource) feature called
Intel L2 Cache Allocation Technology (L2 CAT) to Xen. It has been enabled
in Linux Kernel.

Besides the L2 CAT implementaion, we refactor the psr.c to make it more
flexible and easily to extend to add new features. We abstract the general
operations of all features and encapsulate them into a structure. Then,
the development of new feature is simple to mainly implement these callback
functions.

The patch set can be found at:
https://github.com/yisun-git/xen.git l2_cat_v9

---
Acked and Reviewed list before V9:

a - Acked-by
r - Reviewed-by

  r  patch 1  - docs: create Cache Allocation Technology (CAT) and Code and
                Data Prioritization (CDP) feature document
  ar patch 2  - x86: refactor psr: remove L3 CAT/CDP codes.
  r  patch 3  - x86: refactor psr: implement main data structures.
  r  patch 6  - x86: refactor psr: L3 CAT: implement Domain init/free and
                schedule flows.
  r  patch 7  - x86: refactor psr: L3 CAT: implement get hw info flow.
  r  patch 8  - x86: refactor psr: L3 CAT: implement get value flow.
  a  patch 22 - tools: L2 CAT: support get HW info for L2 CAT.
  a  patch 23 - tools: L2 CAT: support show cbm for L2 CAT.
  a  patch 24 - tools: L2 CAT: support set cbm for L2 CAT.
  a  patch 25 - docs: add L2 CAT description in docs.

---
V9 change list:

Patch 1:
    - add 'CMT' to the list of acronyms.
      (suggested by Wei Liu)
    - change feature list to feature array.
    - modify data structure descriptions according to latest codes.
    - modify revision.
Patch 3:
    - replace feature list to a feature pointer array.
      (suggested by Roger Pau)
    - add 'PSR_SOCKET_MAX_FEAT' in 'enum psr_feat_type' to know features
      account.
      (suggested by Roger Pau)
    - move 'feat_ops' declaration into 'feat_node' structure.
      (suggested by Roger Pau)
    - directly use uninon for feature HW info and move its declaration into
      'feat_node' structure.
      (suggested by Roger Pau)
    - remove 'enum psr_feat_type feature' declared in 'feat_ops' because it is
      not useful after using feature pointer array.
      (suggested by Roger Pau)
    - rename 'l3_cat_info' to 'cat_info' to be used by all CAT/CDP features.
    - remove 'nr_feat' which is only for a record.
      (suggested by Jan Beulich)
    - add 'cos_num' to record how many COS registers are used by a feature in
      one time access.
      (suggested by Jan Beulich)
    - replace 'uint64_t' to 'uint32_t' for cbm value because SDM specifies the
      max 32 bits for it.
      (suggested by Jan Beulich)
Patch 4:
    - create this patch alone to move 'cpuid_count_leaf'.
      (suggested by Wei Liu)
Patch 5:
    - add commit message to explain the flows.
    - handle cpu offline and online again case to read MSRs registers values
      back and save them into cos array to make user can get real data.
    - create a new patch about moving 'cpuid_count_leaf'.
      (suggested by Wei Liu)
    - modify comment to explain why not free some resource in 'free_feature'.
      (suggested by Wei Liu)
    - implement 'psr_alloc_feat_enabled' to check if allocation feature is
      enabled in cmdline and some initialization work done.
      (suggested by Wei Liu)
    - implement 'cat_default_val' to set default value for CAT features.
      (suggested by Wei Liu)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - implement a common 'cat_init_feature' to replace L3 CAT/L2 CAT specific
      init functions.
      (suggested by Roger Pau)
    - modify comments for global feature node.
      (suggested by Jan Beulich)
    - remove unnecessary comments.
      (suggested by Jan Beulich)
    - remove unnecessary 'else'.
      (suggested by Jan Beulich)
    - remove 'nr_feat'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - check global flag with boot cpu operations.
      (suggested by Jan Beulich)
    - remove 'cpu_init_work' and move codes into 'psr_cpu_init'.
      (suggested by Jan Beulich)
    - remove 'cpu_fini_work' and move codes into 'psr_cpu_fini'.
      (suggested by Jan Beulich)
    - assign value for 'cos_num'.
      (suggested by Jan Beulich)
    - change about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 6:
    - rename 'l3_cat_get_cos_max' to 'cat_get_cos_max' to cover all CAT/CDP
      features.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - implement 'psr_alloc_cos' to match 'psr_free_cos'.
      (suggested by Wei Liu)
    - use 'psr_alloc_feat_enabled'.
      (suggested by Wei Liu)
    - fix coding style issue.
      (suggested by Wei Liu)
    - remove 'inline'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - remove 'psr_cos_ids' check in 'psr_free_cos'.
      (suggested by Jan Beulich)
Patch 7:
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - define 'PSR_INFO_SIZE'.
      (suggested by Roger Pau)
    - fix coding style issue.
      (suggested by Roger Pau and Jan Beulich)
    - use 'ARRAY_SIZE'.
      (suggested by Roger Pau)
    - rename 'l3_cat_get_feat_info' to 'cat_get_feat_info' to make it a common
      function for both L3/L2 CAT.
      (suggested by Roger Pau)
    - move constant to the right of comparison.
      (suggested by Wei Liu)
    - remove wrong comment.
      (suggested by Jan Beulich)
    - rename macros used by psr_get_info to make them meaningful.
      (suggested by Jan Beulich)
    - remove assignment for 'PSR_SOCKET_UNKNOWN'.
      (suggested by Jan Beulich)
    - retain blank line after 'case XEN_SYSCTL_PSR_CAT_get_l3_info'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - move common data check into common function.
      (suggested by Jan Beulich)
Patch 8:
    - add commit message to explain there is an user space interface.
    - rename 'l3_cat_get_val' to 'cat_get_val' to cover all L3/L2 CAT features.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - change parameter of 'psr_get'. Use 'psr_cos_ids' directly to replace
      domain. Also declare it to 'const'.
      (suggested by Jan Beulich)
    - change code flow to remove 'psr_get' but add 'psr_get_feat' to make codes
      more reasonable.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - move cos check into common function because this check is required by all
      features.
      (suggested by Jan Beulich)
    - fix coding style issue.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'..
      (suggested by Jan Beulich)
Patch 9:
    - use goto style error handling in 'psr_set_val'.
      (suggested by Wei Liu)
    - use ASSERT for checking old_cos.
      (suggested by Wei Liu and Jan Beulich)
    - fix coding style issue.
      (suggested by Wei Liu)
    - rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
      (suggested by Wei Liu)
    - use 'spin_is_locked' to check ref_lock.
      (suggested by Roger Pau)
    - add an input parameter 'array_len' for 'write_psr_msr'.
    - check 'socket_info' and 'psr_cos_ids' in this patch.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - fix commit message words.
      (suggested by Jan Beulich)
    - change 'assemble_val_array' to 'gather_val_array'.
      (suggested by Jan Beulich)
    - change 'set_new_val_to_array' to 'insert_new_val_to_array'.
      (suggested by Jan Beulich)
    - change parameter 'm' of 'insert_new_val_to_array' to 'new_val'.
      (suggested by Jan Beulich)
    - change 'write_psr_msr' to 'write_psr_msrs'.
      (suggested by Jan Beulich)
    - correct comments.
      (suggested by Jan Beulich)
    - remove unnecessary comments.
      (suggested by Jan Beulich)
    - adjust conditions after 'find_cos' to save a level of indentation.
      (suggested by Jan Beulich)
    - add 'ASSERT(!old_cos || ref[old_cos])'.
      (suggested by Jan Beulich)
    - move ASSERT() check into locked region.
      (suggested by Jan Beulich)
    - replace parameter '*val' to 'val[]' in some functions.
      (suggested by Jan Beulich)
    - change 'write_psr_msr' parameters to prepare to only set one new value
      for one feature.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
    - add explanation about context switch.
      (suggested by Jan Beulich)
Patch 10:
    - add comments about boundary checking.
      (suggested by Wei Liu)
    - rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
      (suggested by Wei Liu)
    - rename 'l3_cat_get_cos_num' to 'cat_get_cos_num' to cover all L3/L2 CAT
      features.
      (suggested by Roger Pau)
    - rename 'l3_cat_get_old_val' to 'cat_get_old_val' to cover all L3/L2 CAT
      features and reuse cat_get_val in it.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - replace 'm' to 'new_val'.
      (suggested by Jan Beulich)
    - move cos check outside callback function.
      (suggested by Jan Beulich)
    - remove 'get_cos_num' callback function.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 11:
    - modify comments of 'compare_val' to be same as current implementation.
      (suggested by Wei Liu)
    - fix indentation issue.
      (suggested by Wei Liu)
    - rename 'l3_cat_compare_val' to 'cat_compare_val' to cover all L3/L2 CAT
      features.
      (suggested by Roger Pau)
    - remove parameter 'found' from 'cat_compare_val' and modify the return
      values to let caller know if the id is found or not.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - replace 'get_cos_num' to 'feat->cos_num'.
      (suggested by Jan Beulich)
    - directly use 'cos_reg_val[0]' as default value.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 12:
    - modify return value of 'pick_avail_cos' to make it more accurate.
    - rename 'l3_cat_fits_cos_max' to 'cat_fits_cos_max' to cover L3/L2 CAT
      features.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - fix comment.
      (suggested by Wei Liu)
    - directly use 'cos_reg_val[0]' as default value.
      (suggested by Jan Beulich)
    - replace 'get_cos_num' to 'feat->cos_num'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 13:
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - add 'array_len' in 'struct cos_write_info' and check if val array
      exceeds it.
    - modify 'write_psr_msr' flow only to set one value a time. No need to
      set whole feature array values.
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 14:
    - modify commit message to describe flow clearer.
    - handle cpu offline and online again case to read MSRs registers values
      back and save them into cos array to make user can get real data.
    - modify error handling process in 'psr_cpu_prepare' to reduce redundant
      codes.
    - modify 'get_cdp_data' and 'get_cdp_code' to make them standard.
      (suggested by Roger Pau and Jan Beulich)
    - encapsulate CDP operations into 'cat_init_feature' to reduce redundant
      codes.
      (suggested by Roger Pau)
    - reuse 'cat_get_cos_max' for CDP.
      (suggested by Roger Pau)
    - handle 'PSR_CDP' in psr_presmp_init to make init work can be done when
      there is only 'psr=cdp' in cmdline.
    - remove unnecessary comment.
      (suggested by Jan Beulich)
    - move CDP related codes in 'cpu_init_work' into 'psr_cpu_init'.
      (suggested by Jan Beulich)
    - add codes to handle CDP's 'cos_num'.
      (suggested by Jan Beulich)
    - fix coding style issue.
      (suggested by Jan Beulich)
    - do not free resources when allocation fails in 'psr_cpu_prepare'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 15:
    - modify commit message to explain flow more clearly.
    - reuse 'cat_get_feat_info' for CDP to reduce redundant codes.
      (suggested by Roger Pau)
    - fix coding style issues.
      (suggested by Wei Liu and Roger Pau)
    - rename macros used by psr_get_info to make them meaningful.
      (suggested by Jan Beulich)
Patch 16:
    - modify the type of 'l3_cdp_get_val' to 'void'.
    - cos checking has been done in common function so remove related codes
      in CDP callback function.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 17:
    - add comment to explain why CDP uses 2 COSs.
      (suggested by Wei Liu)
    - use 'cat_default_val'.
      (suggested by Wei Liu)
    - remove 'l3_cdp_get_cos_num' because we can directly get cos_num from
      feat_node now.
      (suggested by Jan Beulich)
    - remove cos checking because it has been moved to common function.
      (suggested by Jan Beulich)
    - l3_cdp_set_new_val parameter 'm' is changed to 'new_val'.
      (suggested by Jan Beulich)
    - directly use get_cdp_data(feat, 0) and get_cdp_code(feat, 0) to get
      default value.
      (suggested by Jan Beulich)
    - modify 'l3_cdp_write_msr' flow to write value into register according
      to input type.
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 18:
    - modify error handling process in 'psr_cpu_prepare' to reduce redundant
      codes.
    - reuse 'cat_init_feature' and 'cat_get_cos_max' for L2 CAT to reduce
      redundant codes.
      (suggested by Roger Pau)
    - remove unnecessary comment.
      (suggested by Jan Beulich)
    - move L2 CAT related codes from 'cpu_init_work' into 'psr_cpu_init'.
      (suggested by Jan Beulich)
    - do not free resource when allocation fails in 'psr_cpu_prepare'.
      (suggested by Jan Beulich)
Patch 19:
    - reuse 'cat_get_feat_info' for L2 CAT to reduce redundant codes.
      (suggested by Roger Pau)
    - modify sysctl implementation of L2 CAT to input data[3] to use
      'cat_get_feat_info'.
      (suggested by Roger Pau)
    - modify macros names to newly defined ones.
      (suggested by Jan Beulich)
    - remove 'l2_info' to reuse 'l3_info'.
      (suggested by Jan Beulich)
    - modify macro name according to previous patch change.
      (suggested by Jan Beulich)
Patch 20:
    - reuse 'cat_get_val' for L2 CAT to reduce redundant codes
      (suggested by Roger Pau)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 21:
    - reuse some CAT common functions for L2 CAT to reduce redundant codes.
      (suggested by Roger Pau)
    - remove parameter 'found' from 'cat_compare_val' and modify the return
      values to let caller know if the id is found or not. These things are
      done in patch "x86: refactor psr: set value: implement cos finding flow."
      (suggested by Roger Pau and Dario Faggioli)
    - remove 'get_cos_num' related codes.
      (suggested by Jan Beulich)
    - modify 'l2_cat_write_msr' according to previous patch change.
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
Patch 22:
    - add some cases to handle return error no.
    - move xl_cmdimpl.c codes into xl/xl_psr.c.
    - change 'l3_info' to 'cat_info' to cover both L3 and L2 CAT.
Patch 23:
    - move xl_cmdimpl.c changes into xl/xl_psr.c.
    - move xl_cmdtable.c changes into xl/xl_cmdtable.c.
Patch 24:
    - handle the case to set both CODE and DATA for CDP at same time.
      For such case, user does not input '-c' or '-d' to set CDP cbm.
    - move xl_cmdimpl.c changes into xl/xl_psr.c.
    - move xl_cmdtable.c changes into xl/xl_cmdtable.c.

Yi Sun (25):
  docs: create Cache Allocation Technology (CAT) and Code and Data
    Prioritization (CDP) feature document
  x86: refactor psr: remove L3 CAT/CDP codes.
  x86: refactor psr: implement main data structures.
  x86: move cpuid_count_leaf from cpuid.c to processor.h.
  x86: refactor psr: L3 CAT: implement CPU init and free flow.
  x86: refactor psr: L3 CAT: implement Domain init/free and schedule
    flows.
  x86: refactor psr: L3 CAT: implement get hw info flow.
  x86: refactor psr: L3 CAT: implement get value flow.
  x86: refactor psr: L3 CAT: set value: implement framework.
  x86: refactor psr: L3 CAT: set value: assemble features value array.
  x86: refactor psr: L3 CAT: set value: implement cos finding flow.
  x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  x86: refactor psr: L3 CAT: set value: implement write msr flow.
  x86: refactor psr: CDP: implement CPU init and free flow.
  x86: refactor psr: CDP: implement get hw info flow.
  x86: refactor psr: CDP: implement get value flow.
  x86: refactor psr: CDP: implement set value callback functions.
  x86: L2 CAT: implement CPU init and free flow.
  x86: L2 CAT: implement get hw info flow.
  x86: L2 CAT: implement get value flow.
  x86: L2 CAT: implement set value flow.
  tools: L2 CAT: support get HW info for L2 CAT.
  tools: L2 CAT: support show cbm for L2 CAT.
  tools: L2 CAT: support set cbm for L2 CAT.
  docs: add L2 CAT description in docs.

 docs/features/intel_psr_cat_cdp.pandoc |  469 ++++++++++
 docs/man/xl.pod.1.in                   |   25 +-
 docs/misc/xl-psr.markdown              |   18 +-
 tools/libxc/include/xenctrl.h          |    7 +-
 tools/libxc/xc_psr.c                   |   45 +-
 tools/libxl/libxl.h                    |    9 +
 tools/libxl/libxl_psr.c                |   70 +-
 tools/libxl/libxl_types.idl            |    1 +
 tools/xl/xl_cmdtable.c                 |    6 +-
 tools/xl/xl_psr.c                      |  168 ++--
 xen/arch/x86/cpuid.c                   |    6 -
 xen/arch/x86/domctl.c                  |   53 +-
 xen/arch/x86/psr.c                     | 1497 +++++++++++++++++++++++++-------
 xen/arch/x86/sysctl.c                  |   58 +-
 xen/include/asm-x86/msr-index.h        |    1 +
 xen/include/asm-x86/processor.h        |    7 +
 xen/include/asm-x86/psr.h              |   27 +-
 xen/include/public/domctl.h            |    2 +
 xen/include/public/sysctl.h            |    3 +-
 19 files changed, 2025 insertions(+), 447 deletions(-)
 create mode 100644 docs/features/intel_psr_cat_cdp.pandoc

-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* [PATCH v9 01/25] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
@ 2017-03-16 11:07 ` Yi Sun
  2017-03-16 11:07 ` [PATCH v9 02/25] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
                   ` (24 subsequent siblings)
  25 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch creates CAT and CDP feature document in doc/features/. It describes
key points to implement L3 CAT/CDP and L2 CAT which is described in details in
Intel SDM "INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES".

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v9:
    - add 'CMT' to the list of acronyms.
      (suggested by Wei Liu)
    - change feature list to feature array.
    - modify data structure descriptions according to latest codes.
    - modify revision.
v8:
    - change revision info.
      (suggested by Konrad Rzeszutek Wilk)
    - add content int 'Areas for improvement'.
      (suggested by Konrad Rzeszutek Wilk)
v7:
    - correct typo.
      (suggested by Konrad Rzeszutek Wilk)
    - replace application/VM to domain.
      (suggested by Konrad Rzeszutek Wilk)
    - amend description of `feat_mask` to make it clearer.
      (suggested by Konrad Rzeszutek Wilk)
    - update revision.
      (suggested by Konrad Rzeszutek Wilk)
    - other minor fixes.
      (suggested by Konrad Rzeszutek Wilk)
v6:
    - write a new feature document to cover L3 CAT/CDP and L2 CAT.
      (suggested by Kevin Tian)
    - adjust 'Terminology' position in document.
      (suggested by Dario Faggioli)
    - fix wordings.
      (suggested by Dario Faggioli, Kevin Tian and Konrad Rzeszutek Wilk)
    - add SDM chapter title in commit message.
      (suggested by Konrad Rzeszutek Wilk)
    - add more explanations.
      (suggested by Kevin Tian)
v4:
    - change file name to be more descriptive, 'intel_psr_l2_cat.pandoc'.
      (suggested by Dario Faggioli)
    - remove 'Ext' and 'New' prefixes.
      (suggested by Dario Faggioli)
    - remove change log in Revison part.
      (suggested by Dario Faggioli)
    - adjust Xen release number to 4.9 to show this feature targets 4.9.
      (suggested by Dario Faggioli)
    - provide 'Terminology' and more sections.
      (suggested by Dario Faggioli)
    - fix wordings.
      (suggested by Konrad Rzeszutek Wilk)
    - remove chapter number.
      (suggested by Konrad Rzeszutek Wilk)
v3:
    - make design document be a patch.
      (suggested by Konrad Rzeszutek Wilk)
v2:
    - provide chapter for the L2 CAT.
      (suggested by Meng Xu)
---
 docs/features/intel_psr_cat_cdp.pandoc | 469 +++++++++++++++++++++++++++++++++
 1 file changed, 469 insertions(+)
 create mode 100644 docs/features/intel_psr_cat_cdp.pandoc

diff --git a/docs/features/intel_psr_cat_cdp.pandoc b/docs/features/intel_psr_cat_cdp.pandoc
new file mode 100644
index 0000000..022fbdc
--- /dev/null
+++ b/docs/features/intel_psr_cat_cdp.pandoc
@@ -0,0 +1,469 @@
+% Intel Cache Allocation Technology and Code and Data Prioritization Features
+% Revision 1.9
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+         Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+   Component(s): Hypervisor, toolstack
+
+       Hardware: L3 CAT: Haswell and beyond CPUs
+                 CDP   : Broadwell and beyond CPUs
+                 L2 CAT: Atom codename Goldmont and beyond CPUs
+---------------- ----------------------------------------------------
+
+# Terminology
+
+* CAT         Cache Allocation Technology
+* CBM         Capacity BitMasks
+* CDP         Code and Data Prioritization
+* CMT         Cache Monitoring Technology
+* COS/CLOS    Class of Service
+* MSRs        Machine Specific Registers
+* PSR         Intel Platform Shared Resource
+
+# Overview
+
+Intel provides a set of allocation capabilities including Cache Allocatation
+Technology (CAT) and Code and Data Prioritization (CDP).
+
+CAT allows an OS or hypervisor to control allocation of a CPU's shared cache
+based on application/domain priority or Class of Service (COS). Each COS is
+configured using capacity bitmasks (CBMs) which represent cache capacity and
+indicate the degree of overlap and isolation between classes. Once CAT is co-
+nfigured, the processor allows access to portions of cache according to the
+established COS. Intel Xeon processor E5 v4 family (and some others) introduce
+capabilities to configure and make use of the CAT mechanism on the L3 cache.
+Intel Goldmont processor provides support for control over the L2 cache.
+
+Code and Data Prioritization (CDP) Technology is an extension of CAT. CDP
+enables isolation and separate prioritization of code and data fetches to
+the L3 cache in a SW configurable manner, which can enable workload priorit-
+ization and tuning of cache capacity to the characteristics of the workload.
+CDP extends CAT by providing separate code and data masks per Class of Service
+(COS). When SW configures to enable CDP, L3 CAT is disabled.
+
+# User details
+
+* Feature Enabling:
+
+  Add "psr=cat" to boot line parameter to enable all supported level CAT featu-
+  res. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW.
+
+* xl interfaces:
+
+  1. `psr-cat-show [OPTIONS] domain-id`:
+
+     Show L2 CAT or L3 CAT/CDP CBM of the domain designated by Xen domain-id.
+
+     Option `-l`:
+     `-l2`: Show cbm for L2 cache.
+     `-l3`: Show cbm for L3 cache.
+
+     If `-lX` is specified and LX is not supported, print error.
+     If no `-l` is specified, level 3 is the default option.
+
+  2. `psr-cat-set [OPTIONS] domain-id cbm`:
+
+     Set L2 CAT or L3 CAT/CDP CBM to the domain designated by Xen domain-id.
+
+     Option `-s`: Specify the socket to process, otherwise all sockets are
+     processed.
+
+     Option `-l`:
+     `-l2`: Specify cbm for L2 cache.
+     `-l3`: Specify cbm for L3 cache.
+
+     If `-lX` is specified and LX is not supported, print error.
+     If no `-l` is specified, level 3 is the default option.
+
+     Option `-c` or `-d`:
+     `-c`: Set L3 CDP code cbm.
+     `-d`: Set L3 CDP data cbm.
+
+  3. `psr-hwinfo [OPTIONS]`:
+
+     Show CMT & L2 CAT & L3 CAT/CDP HW information on every socket.
+
+     Option `-m, --cmt`: Show Cache Monitoring Technology (CMT) hardware info.
+
+     Option `-a, --cat`: Show CAT/CDP hardware info.
+
+# Technical details
+
+L3 CAT/CDP and L2 CAT are all members of Intel PSR features, they share the base
+PSR infrastructure in Xen.
+
+## Hardware perspective
+
+  CAT/CDP defines a range of MSRs to assign different cache access patterns
+  which are known as CBMs, each CBM is associated with a COS.
+
+  ```
+  E.g. L2 CAT:
+                          +----------------------------+----------------+
+     IA32_PQR_ASSOC       | MSR (per socket)           |    Address     |
+   +----+---+-------+     +----------------------------+----------------+
+   |    |COS|       |     | IA32_L2_QOS_MASK_0         |     0xD10      |
+   +----+---+-------+     +----------------------------+----------------+
+          └-------------> | ...                        |  ...           |
+                          +----------------------------+----------------+
+                          | IA32_L2_QOS_MASK_n         | 0xD10+n (n<64) |
+                          +----------------------------+----------------+
+  ```
+
+  L3 CAT/CDP uses a range of MSRs from 0xC90 ~ 0xC90+n (n<128).
+
+  L2 CAT uses a range of MSRs from 0xD10 ~ 0xD10+n (n<64), following the L3
+  CAT/CDP MSRs, setting different L2 cache accessing patterns from L3 cache is
+  supported.
+
+  Every MSR stores a CBM value. A capacity bitmask (CBM) provides a hint to the
+  hardware indicating the cache space a domain should be limited to as well as
+  providing an indication of overlap and isolation in the CAT-capable cache from
+  other domains contending for the cache.
+
+  Sample cache capacity bitmasks for a bitlength of 8 are shown below. Please
+  note that all (and only) contiguous '1' combinations are allowed (e.g. FFFFH,
+  0FF0H, 003CH, etc.).
+
+  ```
+       +----+----+----+----+----+----+----+----+
+       | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+       +----+----+----+----+----+----+----+----+
+  COS0 | A  | A  | A  | A  | A  | A  | A  | A  | Default Bitmask
+       +----+----+----+----+----+----+----+----+
+  COS1 | A  | A  | A  | A  | A  | A  | A  | A  |
+       +----+----+----+----+----+----+----+----+
+  COS2 | A  | A  | A  | A  | A  | A  | A  | A  |
+       +----+----+----+----+----+----+----+----+
+
+       +----+----+----+----+----+----+----+----+
+       | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+       +----+----+----+----+----+----+----+----+
+  COS0 | A  | A  | A  | A  | A  | A  | A  | A  | Overlapped Bitmask
+       +----+----+----+----+----+----+----+----+
+  COS1 |    |    |    |    | A  | A  | A  | A  |
+       +----+----+----+----+----+----+----+----+
+  COS2 |    |    |    |    |    |    | A  | A  |
+       +----+----+----+----+----+----+----+----+
+
+       +----+----+----+----+----+----+----+----+
+       | M7 | M6 | M5 | M4 | M3 | M2 | M1 | M0 |
+       +----+----+----+----+----+----+----+----+
+  COS0 | A  | A  | A  | A  |    |    |    |    | Isolated Bitmask
+       +----+----+----+----+----+----+----+----+
+  COS1 |    |    |    |    | A  | A  |    |    |
+       +----+----+----+----+----+----+----+----+
+  COS2 |    |    |    |    |    |    | A  | A  |
+       +----+----+----+----+----+----+----+----+
+  ```
+
+  We can get the CBM length through CPUID. The default value of CBM is calcul-
+  ated by `(1ull << cbm_len) - 1`. That is a fully open bitmask, all ones bitm-
+  ask. The COS[0] always stores the default value without change.
+
+  There is a `IA32_PQR_ASSOC` register which stores the COS ID of the VCPU. HW
+  enforces cache allocation according to the corresponding CBM.
+
+## The relationship between L3 CAT/CDP and L2 CAT
+
+  HW may support all features. By default, CDP is disabled on the processor.
+  If the L3 CAT MSRs are used without enabling CDP, the processor operates in
+  a traditional CAT-only mode. When CDP is enabled:
+  * the CAT mask MSRs are re-mapped into interleaved pairs of mask MSRs for
+    data or code fetches.
+  * the range of COS for CAT is re-indexed, with the lower-half of the COS
+    range available for CDP.
+
+  L2 CAT is independent of L3 CAT/CDP, which means L2 CAT can be enabled while
+  L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are both enabled.
+
+  As a requirement, the bits of CBM of CAT/CDP must be continuous.
+
+  N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same associate
+  register `IA32_PQR_ASSOC`, which means one COS is associated with a pair of
+  L2 CAT CBM and L3 CAT/CDP CBM.
+
+  Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or other
+  PSR features in future). In some cases, a domain is permitted to have a COS
+  that is beyond one (or more) of PSR features but within the others. For
+  instance, let's assume the max COS of L2 CAT is 8 but the max COS of L3
+  CAT is 16, when a domain is assigned 9 as COS, the L3 CAT CBM associated to
+  COS 9 would be enforced, but for L2 CAT, the HW works as default value is
+  set since COS 9 is beyond the max COS (8) of L2 CAT.
+
+## Design Overview
+
+* Core COS/CBM association
+
+  When enforcing CAT/CDP, all cores of domains have the same default COS (COS0)
+  which is associated with the fully open CBM (all ones bitmask) to access all
+  cache. The default COS is used only in hypervisor and is transparent to tool
+  stack and user.
+
+  System administrator can change PSR allocation policy at runtime by tool stack.
+  Since L2 CAT shares COS with L3 CAT/CDP, a COS corresponds to a 2-tuple, like
+  [L2 CBM, L3 CBM] with only-CAT enabled, when CDP is enabled, one COS correspo-
+  nds to a 3-tuple, like [L2 CBM, L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT
+  nor L3 CDP is enabled, things would be easier, one COS corresponds to one L2
+  CBM.
+
+* VCPU schedule
+
+  When context switch happens, the COS of VCPU is written to per-thread MSR
+  `IA32_PQR_ASSOC`, and then hardware enforces cache allocation according to
+  the corresponding CBM.
+
+* Multi-sockets
+
+  Different sockets may have different CAT/CDP capability (e.g. max COS) alth-
+  ough it is consistent on the same socket. So the capability of per-socket CAT/
+  CDP is specified.
+
+  'psr-cat-set' can set CBM for one domain per socket. On each socket, we main-
+  tain a COS array for all domains. One domain uses one COS at one time. One COS
+  stores the CBM of the domain to work. So, when a VCPU of the domain is migrat-
+  ed from socket 1 to socket 2, it follows configuration on socket 2.
+
+  E.g. user sets domain 1 CBM on socket 1 to 0x7f which uses COS 9 but sets do-
+  main 1 CBM on socket 2 to 0x3f which uses COS 7. When VCPU of this domain
+  is migrated from socket 1 to 2, the COS ID used is 7, that means 0x3f is the
+  CBM to work for this domain 1 now.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+  1. Boot line parameter "psr=cat" enables L2 CAT and L3 CAT if hardware suppo-
+     rted. "psr=cdp" enables CDP if hardware supported.
+
+  2. SYSCTL:
+          - XEN_SYSCTL_PSR_CAT_get_l3_info: Get L3 CAT/CDP information.
+          - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
+
+  3. DOMCTL:
+          - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM: Get L3 CBM for a domain.
+          - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM: Set L3 CBM for a domain.
+          - XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE: Get CDP Code CBM for a domain.
+          - XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE: Set CDP Code CBM for a domain.
+          - XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA: Get CDP Data CBM for a domain.
+          - XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA: Set CDP Data CBM for a domain.
+          - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
+          - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
+
+* xl interfaces:
+
+  1. psr-cat-show -lX domain-id
+          Show LX cbm for a domain.
+          => XEN_SYSCTL_PSR_CAT_get_l3_info    /
+             XEN_SYSCTL_PSR_CAT_get_l2_info    /
+             XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM  /
+             XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE /
+             XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA /
+             XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
+
+  2. psr-cat-set -lX domain-id cbm
+          Set LX cbm for a domain.
+          => XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM  /
+             XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE /
+             XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA /
+             XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
+
+  3. psr-hwinfo
+          Show PSR HW information, including L3 CAT/CDP/L2 CAT
+          => XEN_SYSCTL_PSR_CAT_get_l3_info /
+             XEN_SYSCTL_PSR_CAT_get_l2_info
+
+* Key data structure:
+
+   1. Feature HW info
+
+      ```
+      struct psr_cat_hw_info {
+          unsigned int cbm_len;
+          unsigned int cos_max;
+      };
+      ```
+
+      - Member `cbm_len`
+
+        `cbm_len` is one of the hardware info of CAT. It means the max number
+        of bits to set.
+
+      - Member `cos_max`
+
+        `cos_max` is one of the hardware info of CAT. It means the max number
+        of COS registers.
+
+   2. Feature node
+
+      ```
+      struct feat_node {
+          struct feat_ops {
+              unsigned int (*get_cos_max)(const struct feat_node *feat);
+              bool (*get_feat_info)(const struct feat_node *feat,
+                                    uint32_t data[], unsigned int array_len);
+              void (*get_val)(const struct feat_node *feat, unsigned int cos,
+                              enum cbm_type type, uint32_t *val);
+              void (*get_old_val)(uint32_t val[],
+                                  const struct feat_node *feat,
+                                  unsigned int old_cos);
+              int (*set_new_val)(uint32_t val[],
+                                 const struct feat_node *feat,
+                                 enum cbm_type type,
+                                 uint32_t new_val);
+              int (*compare_val)(const uint32_t val[], const struct feat_node *feat,
+                                 unsigned int cos);
+              bool (*fits_cos_max)(const uint32_t val[],
+                                   const struct feat_node *feat,
+                                   unsigned int cos);
+              void (*write_msr)(unsigned int cos, uint32_t val,
+                                enum cbm_type type, struct feat_node *feat);
+          } ops;
+
+          union {
+              struct psr_cat_hw_info cat_info;
+          } info;
+
+          uint32_t cos_reg_val[MAX_COS_REG_CNT];
+          unsigned int cos_num;
+      };
+      ```
+
+      When a PSR enforcement feature is enabled, it will be added into a
+      feature array.
+
+      - Member `ops`
+
+        `ops` maintains a callback function list of the feature.
+
+        We abstract above callback functions to encapsulate the feature specific
+        behaviors into them. Then, it is easy to add a new feature:
+          1) Implement such ops and callback functions for every feature.
+          2) Register the ops into `struct feat_node`.
+          3) Add the feature into feature array during CPU initialization.
+
+      - Member `info`
+
+        `info` maintains the feature HW information which is provided to
+        psr_hwinfo command.
+
+      - Member `cos_reg_val`
+
+        `cos_reg_val` is an array to maintain the value set in all COS registers
+        of the feature. The array is indexed by COS ID.
+
+      - Member `cos_num`
+
+        `cos_num` is the number of COS registers the feature uses, e.g. L3/L2
+        CAT uses 1 register but CDP uses 2 registers.
+
+   3. Per-socket PSR features information structure
+
+      ```
+      struct psr_socket_info {
+          unsigned int feat_mask;
+          struct feat_node *features[PSR_SOCKET_MAX_FEAT];
+          unsigned int cos_ref[MAX_COS_REG_NUM];
+          spinlock_t ref_lock;
+      };
+      ```
+
+      We collect all PSR allocation features information of a socket in this
+      `struct psr_socket_info`.
+
+      - Member `feat_mask`
+
+        `feat_mask` is a bitmap, to indicate which feature is enabled on current
+        socket. See values defined in `enum psr_feat_type`. E.g.
+
+        bit 0: L3 CAT status.
+        bit 1: L3 CDP status.
+        bit 2: L2 CAT status.
+
+      - Member `features`
+
+        `features` is a pointer array to save all enabled features poniters
+        according to feature position defined in `enum psr_feat_type`.
+
+      - Member `cos_ref`
+
+        `cos_ref` is an array which maintains the reference of one COS. It maps
+        to cos_reg_val[MAX_COS_REG_NUM] in `struct feat_node`. If one COS is
+        used by one domain, the corresponding reference will increase by one. If
+        a domain releases the COS, the reference will decrease by one. The array
+        is indexed by COS ID.
+
+# Limitations
+
+CAT/CDP can only work on HW which enables it(check by CPUID). So far, there is
+no HW which enables both L2 CAT and L3 CAT/CDP. But SW implementation has cons-
+idered such scenario to enable both L2 CAT and L3 CAT/CDP.
+
+# Testing
+
+We can execute above xl commands to verify L2 CAT and L3 CAT/CDP on different
+HWs support them.
+
+For example:
+    root@:~$ xl psr-hwinfo --cat
+    Cache Allocation Technology (CAT): L2
+    Socket ID       : 0
+    Maximum COS     : 3
+    CBM length      : 8
+    Default CBM     : 0xff
+
+    root@:~$ xl psr-cat-cbm-set -l2 1 0x7f
+
+    root@:~$ xl psr-cat-show -l2 1
+    Socket ID       : 0
+    Default CBM     : 0xff
+       ID                     NAME             CBM
+        1                 ubuntu14            0x7f
+
+# Areas for improvement
+
+A hexadecimal number is used to set/show CBM for a domain now. Although this
+is convenient to cover overlap/isolated bitmask requirement, it is not user-
+friendly.
+
+To improve this, the libxl interfaces can be wrapped in libvirt to provide more
+usr-friendly interfaces to user, e.g. a percentage number of the cache to set
+and show.
+
+# Known issues
+
+N/A
+
+# References
+
+"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
+2016-08-12 1.0      Xen 4.9  Design document written
+2017-02-13 1.7      Xen 4.9  Changes:
+                             1. Modify the design document to cover L3
+                                CAT/CDP and L2 CAT;
+                             2. Fix typos;
+                             3. Amend description of `feat_mask` to make
+                                it clearer;
+                             4. Other minor changes.
+2017-02-15 1.8      Xen 4.9  Changes:
+                             1. Add content in 'Areas for improvement';
+                             2. Adjust revision number.
+2017-03-16 1.9      Xen 4.9  Changes:
+                             1. Add 'CMT' in 'Terminology';
+                             2. Change 'feature list' to 'feature array'.
+                             3. Modify data structure descriptions.
+                             4. Adjust revision number.
+---------- -------- -------- -------------------------------------------
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 02/25] x86: refactor psr: remove L3 CAT/CDP codes.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
  2017-03-16 11:07 ` [PATCH v9 01/25] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
@ 2017-03-16 11:07 ` Yi Sun
  2017-03-16 11:07 ` [PATCH v9 03/25] x86: refactor psr: implement main data structures Yi Sun
                   ` (23 subsequent siblings)
  25 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

The current cache allocation codes in psr.c do not consider
future features addition and are not friendly to extend.

To make psr.c be more flexible to add new features and fulfill
the program principle, open for extension but closed for
modification, we have to refactor the psr.c:
1. Analyze cache allocation features and abstract general data
   structures.
2. Analyze the init and all other functions flow, abstract all
   steps that different features may have different implementations.
   Make these steps be callback functions and register feature
   specific fuctions. Then, the main processes will not be changed
   when introducing a new feature.

Because the quantity of refactor codes is big and the logics are
changed a lot, it will cause reviewers confused if just change
old codes. Reviewers have to understand both old codes and new
implementations. After review iterations from V1 to V3, Jan has
proposed to remove all old cache allocation codes firstly, then
implement new codes step by step. This will help to make codes
be more easily reviewable.

There is no construction without destruction. So, this patch
removes all current L3 CAT/CDP codes in psr.c. The following
patches will introduce the new mechanism.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v4:
    - create this patch to make codes easily understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 470 +----------------------------------------------------
 1 file changed, 5 insertions(+), 465 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0b5073c..96a8589 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -23,24 +23,6 @@
 #define PSR_CAT        (1<<1)
 #define PSR_CDP        (1<<2)
 
-struct psr_cat_cbm {
-    union {
-        uint64_t cbm;
-        struct {
-            uint64_t code;
-            uint64_t data;
-        };
-    };
-    unsigned int ref;
-};
-
-struct psr_cat_socket_info {
-    unsigned int cbm_len;
-    unsigned int cos_max;
-    struct psr_cat_cbm *cos_to_cbm;
-    spinlock_t cbm_lock;
-};
-
 struct psr_assoc {
     uint64_t val;
     uint64_t cos_mask;
@@ -48,26 +30,11 @@ struct psr_assoc {
 
 struct psr_cmt *__read_mostly psr_cmt;
 
-static unsigned long *__read_mostly cat_socket_enable;
-static struct psr_cat_socket_info *__read_mostly cat_socket_info;
-static unsigned long *__read_mostly cdp_socket_enable;
-
 static unsigned int opt_psr;
 static unsigned int __initdata opt_rmid_max = 255;
-static unsigned int __read_mostly opt_cos_max = 255;
 static uint64_t rmid_mask;
 static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
 
-static struct psr_cat_cbm *temp_cos_to_cbm;
-
-static unsigned int get_socket_cpu(unsigned int socket)
-{
-    if ( likely(socket < nr_sockets) )
-        return cpumask_any(socket_cpumask[socket]);
-
-    return nr_cpu_ids;
-}
-
 static void __init parse_psr_bool(char *s, char *value, char *feature,
                                   unsigned int mask)
 {
@@ -107,9 +74,6 @@ static void __init parse_psr_param(char *s)
         if ( val_str && !strcmp(s, "rmid_max") )
             opt_rmid_max = simple_strtoul(val_str, NULL, 0);
 
-        if ( val_str && !strcmp(s, "cos_max") )
-            opt_cos_max = simple_strtoul(val_str, NULL, 0);
-
         s = ss + 1;
     } while ( ss );
 }
@@ -213,16 +177,7 @@ static inline void psr_assoc_init(void)
 {
     struct psr_assoc *psra = &this_cpu(psr_assoc);
 
-    if ( cat_socket_info )
-    {
-        unsigned int socket = cpu_to_socket(smp_processor_id());
-
-        if ( test_bit(socket, cat_socket_enable) )
-            psra->cos_mask = ((1ull << get_count_order(
-                             cat_socket_info[socket].cos_max)) - 1) << 32;
-    }
-
-    if ( psr_cmt_enabled() || psra->cos_mask )
+    if ( psr_cmt_enabled() )
         rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
 }
 
@@ -231,12 +186,6 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
     *reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
 }
 
-static inline void psr_assoc_cos(uint64_t *reg, unsigned int cos,
-                                 uint64_t cos_mask)
-{
-    *reg = (*reg & ~cos_mask) | (((uint64_t)cos << 32) & cos_mask);
-}
-
 void psr_ctxt_switch_to(struct domain *d)
 {
     struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -245,459 +194,54 @@ void psr_ctxt_switch_to(struct domain *d)
     if ( psr_cmt_enabled() )
         psr_assoc_rmid(&reg, d->arch.psr_rmid);
 
-    if ( psra->cos_mask )
-        psr_assoc_cos(&reg, d->arch.psr_cos_ids ?
-                      d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
-                      0, psra->cos_mask);
-
     if ( reg != psra->val )
     {
         wrmsrl(MSR_IA32_PSR_ASSOC, reg);
         psra->val = reg;
     }
 }
-static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket)
-{
-    if ( !cat_socket_info )
-        return ERR_PTR(-ENODEV);
-
-    if ( socket >= nr_sockets )
-        return ERR_PTR(-ENOTSOCK);
-
-    if ( !test_bit(socket, cat_socket_enable) )
-        return ERR_PTR(-ENOENT);
-
-    return cat_socket_info + socket;
-}
-
-static inline bool_t cdp_is_enabled(unsigned int socket)
-{
-    return cdp_socket_enable && test_bit(socket, cdp_socket_enable);
-}
 
 int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
                         uint32_t *cos_max, uint32_t *flags)
 {
-    struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
-    if ( IS_ERR(info) )
-        return PTR_ERR(info);
-
-    *cbm_len = info->cbm_len;
-    *cos_max = info->cos_max;
-
-    *flags = 0;
-    if ( cdp_is_enabled(socket) )
-        *flags |= XEN_SYSCTL_PSR_CAT_L3_CDP;
-
     return 0;
 }
 
 int psr_get_l3_cbm(struct domain *d, unsigned int socket,
                    uint64_t *cbm, enum cbm_type type)
 {
-    struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-    bool_t cdp_enabled = cdp_is_enabled(socket);
-
-    if ( IS_ERR(info) )
-        return PTR_ERR(info);
-
-    switch ( type )
-    {
-    case PSR_CBM_TYPE_L3:
-        if ( cdp_enabled )
-            return -EXDEV;
-        *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
-        break;
-
-    case PSR_CBM_TYPE_L3_CODE:
-        if ( !cdp_enabled )
-            *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
-        else
-            *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].code;
-        break;
-
-    case PSR_CBM_TYPE_L3_DATA:
-        if ( !cdp_enabled )
-            *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
-        else
-            *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].data;
-        break;
-
-    default:
-        ASSERT_UNREACHABLE();
-    }
-
-    return 0;
-}
-
-static bool_t psr_check_cbm(unsigned int cbm_len, uint64_t cbm)
-{
-    unsigned int first_bit, zero_bit;
-
-    /* Set bits should only in the range of [0, cbm_len). */
-    if ( cbm & (~0ull << cbm_len) )
-        return 0;
-
-    /* At least one bit need to be set. */
-    if ( cbm == 0 )
-        return 0;
-
-    first_bit = find_first_bit(&cbm, cbm_len);
-    zero_bit = find_next_zero_bit(&cbm, cbm_len, first_bit);
-
-    /* Set bits should be contiguous. */
-    if ( zero_bit < cbm_len &&
-         find_next_bit(&cbm, cbm_len, zero_bit) < cbm_len )
-        return 0;
-
-    return 1;
-}
-
-struct cos_cbm_info
-{
-    unsigned int cos;
-    bool_t cdp;
-    uint64_t cbm_code;
-    uint64_t cbm_data;
-};
-
-static void do_write_l3_cbm(void *data)
-{
-    struct cos_cbm_info *info = data;
-
-    if ( info->cdp )
-    {
-        wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(info->cos), info->cbm_code);
-        wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(info->cos), info->cbm_data);
-    }
-    else
-        wrmsrl(MSR_IA32_PSR_L3_MASK(info->cos), info->cbm_code);
-}
-
-static int write_l3_cbm(unsigned int socket, unsigned int cos,
-                        uint64_t cbm_code, uint64_t cbm_data, bool_t cdp)
-{
-    struct cos_cbm_info info =
-    {
-        .cos = cos,
-        .cbm_code = cbm_code,
-        .cbm_data = cbm_data,
-        .cdp = cdp,
-    };
-
-    if ( socket == cpu_to_socket(smp_processor_id()) )
-        do_write_l3_cbm(&info);
-    else
-    {
-        unsigned int cpu = get_socket_cpu(socket);
-
-        if ( cpu >= nr_cpu_ids )
-            return -ENOTSOCK;
-        on_selected_cpus(cpumask_of(cpu), do_write_l3_cbm, &info, 1);
-    }
-
     return 0;
 }
 
-static int find_cos(struct psr_cat_cbm *map, unsigned int cos_max,
-                    uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
-{
-    unsigned int cos;
-
-    for ( cos = 0; cos <= cos_max; cos++ )
-    {
-        if ( (map[cos].ref || cos == 0) &&
-             ((!cdp_enabled && map[cos].cbm == cbm_code) ||
-              (cdp_enabled && map[cos].code == cbm_code &&
-                              map[cos].data == cbm_data)) )
-            return cos;
-    }
-
-    return -ENOENT;
-}
-
-static int pick_avail_cos(struct psr_cat_cbm *map, unsigned int cos_max,
-                          unsigned int old_cos)
-{
-    unsigned int cos;
-
-    /* If old cos is referred only by the domain, then use it. */
-    if ( map[old_cos].ref == 1 && old_cos != 0 )
-        return old_cos;
-
-    /* Find an unused one other than cos0. */
-    for ( cos = 1; cos <= cos_max; cos++ )
-        if ( map[cos].ref == 0 )
-            return cos;
-
-    return -ENOENT;
-}
-
 int psr_set_l3_cbm(struct domain *d, unsigned int socket,
                    uint64_t cbm, enum cbm_type type)
 {
-    unsigned int old_cos, cos_max;
-    int cos, ret;
-    uint64_t cbm_data, cbm_code;
-    bool_t cdp_enabled = cdp_is_enabled(socket);
-    struct psr_cat_cbm *map;
-    struct psr_cat_socket_info *info = get_cat_socket_info(socket);
-
-    if ( IS_ERR(info) )
-        return PTR_ERR(info);
-
-    if ( !psr_check_cbm(info->cbm_len, cbm) )
-        return -EINVAL;
-
-    if ( !cdp_enabled && (type == PSR_CBM_TYPE_L3_CODE ||
-                          type == PSR_CBM_TYPE_L3_DATA) )
-        return -ENXIO;
-
-    cos_max = info->cos_max;
-    old_cos = d->arch.psr_cos_ids[socket];
-    map = info->cos_to_cbm;
-
-    switch ( type )
-    {
-    case PSR_CBM_TYPE_L3:
-        cbm_code = cbm;
-        cbm_data = cbm;
-        break;
-
-    case PSR_CBM_TYPE_L3_CODE:
-        cbm_code = cbm;
-        cbm_data = map[old_cos].data;
-        break;
-
-    case PSR_CBM_TYPE_L3_DATA:
-        cbm_code = map[old_cos].code;
-        cbm_data = cbm;
-        break;
-
-    default:
-        ASSERT_UNREACHABLE();
-        return -EINVAL;
-    }
-
-    spin_lock(&info->cbm_lock);
-    cos = find_cos(map, cos_max, cbm_code, cbm_data, cdp_enabled);
-    if ( cos >= 0 )
-    {
-        if ( cos == old_cos )
-        {
-            spin_unlock(&info->cbm_lock);
-            return 0;
-        }
-    }
-    else
-    {
-        cos = pick_avail_cos(map, cos_max, old_cos);
-        if ( cos < 0 )
-        {
-            spin_unlock(&info->cbm_lock);
-            return cos;
-        }
-
-        /* We try to avoid writing MSR. */
-        if ( (cdp_enabled &&
-             (map[cos].code != cbm_code || map[cos].data != cbm_data)) ||
-             (!cdp_enabled && map[cos].cbm != cbm_code) )
-        {
-            ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
-            if ( ret )
-            {
-                spin_unlock(&info->cbm_lock);
-                return ret;
-            }
-            map[cos].code = cbm_code;
-            map[cos].data = cbm_data;
-        }
-    }
-
-    map[cos].ref++;
-    map[old_cos].ref--;
-    spin_unlock(&info->cbm_lock);
-
-    d->arch.psr_cos_ids[socket] = cos;
-
     return 0;
 }
 
-/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
-static void psr_free_cos(struct domain *d)
-{
-    unsigned int socket;
-    unsigned int cos;
-    struct psr_cat_socket_info *info;
-
-    if( !d->arch.psr_cos_ids )
-        return;
-
-    for_each_set_bit(socket, cat_socket_enable, nr_sockets)
-    {
-        if ( (cos = d->arch.psr_cos_ids[socket]) == 0 )
-            continue;
-
-        info = cat_socket_info + socket;
-        spin_lock(&info->cbm_lock);
-        info->cos_to_cbm[cos].ref--;
-        spin_unlock(&info->cbm_lock);
-    }
-
-    xfree(d->arch.psr_cos_ids);
-    d->arch.psr_cos_ids = NULL;
-}
-
 int psr_domain_init(struct domain *d)
 {
-    if ( cat_socket_info )
-    {
-        d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
-        if ( !d->arch.psr_cos_ids )
-            return -ENOMEM;
-    }
-
     return 0;
 }
 
 void psr_domain_free(struct domain *d)
 {
     psr_free_rmid(d);
-    psr_free_cos(d);
-}
-
-static int cat_cpu_prepare(unsigned int cpu)
-{
-    if ( !cat_socket_info )
-        return 0;
-
-    if ( temp_cos_to_cbm == NULL &&
-         (temp_cos_to_cbm = xzalloc_array(struct psr_cat_cbm,
-                                          opt_cos_max + 1UL)) == NULL )
-        return -ENOMEM;
-
-    return 0;
-}
-
-static void cat_cpu_init(void)
-{
-    unsigned int eax, ebx, ecx, edx;
-    struct psr_cat_socket_info *info;
-    unsigned int socket;
-    unsigned int cpu = smp_processor_id();
-    uint64_t val;
-    const struct cpuinfo_x86 *c = cpu_data + cpu;
-
-    if ( !cpu_has(c, X86_FEATURE_PQE) || c->cpuid_level < PSR_CPUID_LEVEL_CAT )
-        return;
-
-    socket = cpu_to_socket(cpu);
-    if ( test_bit(socket, cat_socket_enable) )
-        return;
-
-    cpuid_count(PSR_CPUID_LEVEL_CAT, 0, &eax, &ebx, &ecx, &edx);
-    if ( ebx & PSR_RESOURCE_TYPE_L3 )
-    {
-        cpuid_count(PSR_CPUID_LEVEL_CAT, 1, &eax, &ebx, &ecx, &edx);
-        info = cat_socket_info + socket;
-        info->cbm_len = (eax & 0x1f) + 1;
-        info->cos_max = min(opt_cos_max, edx & 0xffff);
-
-        info->cos_to_cbm = temp_cos_to_cbm;
-        temp_cos_to_cbm = NULL;
-        /* cos=0 is reserved as default cbm(all ones). */
-        info->cos_to_cbm[0].cbm = (1ull << info->cbm_len) - 1;
-
-        spin_lock_init(&info->cbm_lock);
-
-        set_bit(socket, cat_socket_enable);
-
-        if ( (ecx & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
-             cdp_socket_enable && !test_bit(socket, cdp_socket_enable) )
-        {
-            info->cos_to_cbm[0].code = (1ull << info->cbm_len) - 1;
-            info->cos_to_cbm[0].data = (1ull << info->cbm_len) - 1;
-
-            /* We only write mask1 since mask0 is always all ones by default. */
-            wrmsrl(MSR_IA32_PSR_L3_MASK(1), (1ull << info->cbm_len) - 1);
-
-            rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
-            wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
-
-            /* Cut half of cos_max when CDP is enabled. */
-            info->cos_max >>= 1;
-
-            set_bit(socket, cdp_socket_enable);
-        }
-        printk(XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n",
-               socket, info->cos_max, info->cbm_len,
-               cdp_is_enabled(socket) ? "on" : "off");
-    }
-}
-
-static void cat_cpu_fini(unsigned int cpu)
-{
-    unsigned int socket = cpu_to_socket(cpu);
-
-    if ( !socket_cpumask[socket] || cpumask_empty(socket_cpumask[socket]) )
-    {
-        struct psr_cat_socket_info *info = cat_socket_info + socket;
-
-        if ( info->cos_to_cbm )
-        {
-            xfree(info->cos_to_cbm);
-            info->cos_to_cbm = NULL;
-        }
-
-        if ( cdp_is_enabled(socket) )
-            clear_bit(socket, cdp_socket_enable);
-
-        clear_bit(socket, cat_socket_enable);
-    }
-}
-
-static void __init psr_cat_free(void)
-{
-    xfree(cat_socket_enable);
-    cat_socket_enable = NULL;
-    xfree(cat_socket_info);
-    cat_socket_info = NULL;
-}
-
-static void __init init_psr_cat(void)
-{
-    if ( opt_cos_max < 1 )
-    {
-        printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
-        return;
-    }
-
-    cat_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
-    cat_socket_info = xzalloc_array(struct psr_cat_socket_info, nr_sockets);
-    cdp_socket_enable = xzalloc_array(unsigned long, BITS_TO_LONGS(nr_sockets));
-
-    if ( !cat_socket_enable || !cat_socket_info )
-        psr_cat_free();
 }
 
 static int psr_cpu_prepare(unsigned int cpu)
 {
-    return cat_cpu_prepare(cpu);
+    return 0;
 }
 
 static void psr_cpu_init(void)
 {
-    if ( cat_socket_info )
-        cat_cpu_init();
-
     psr_assoc_init();
 }
 
 static void psr_cpu_fini(unsigned int cpu)
 {
-    if ( cat_socket_info )
-        cat_cpu_fini(cpu);
+    return;
 }
 
 static int cpu_callback(
@@ -738,14 +282,10 @@ static int __init psr_presmp_init(void)
     if ( (opt_psr & PSR_CMT) && opt_rmid_max )
         init_psr_cmt(opt_rmid_max);
 
-    if ( opt_psr & PSR_CAT )
-        init_psr_cat();
-
-    if ( psr_cpu_prepare(0) )
-        psr_cat_free();
+    psr_cpu_prepare(0);
 
     psr_cpu_init();
-    if ( psr_cmt_enabled() || cat_socket_info )
+    if ( psr_cmt_enabled() )
         register_cpu_notifier(&cpu_nfb);
 
     return 0;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 03/25] x86: refactor psr: implement main data structures.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
  2017-03-16 11:07 ` [PATCH v9 01/25] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
  2017-03-16 11:07 ` [PATCH v9 02/25] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
@ 2017-03-16 11:07 ` Yi Sun
  2017-03-24 16:19   ` Jan Beulich
  2017-03-16 11:07 ` [PATCH v9 04/25] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
                   ` (22 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

To construct an extendible framework, we need analyze PSR features
and abstract the common things and feature specific things. Then,
encapsulate them into different data structures.

By analyzing PSR features, we can get below map.
                +------+------+------+
      --------->| Dom0 | Dom1 | ...  |
      |         +------+------+------+
      |            |
      |Dom ID      | cos_id of domain
      |            V
      |        +-----------------------------------------------------------------------------+
User --------->| PSR                                                                         |
     Socket ID |  +--------------+---------------+---------------+                           |
               |  | Socket0 Info | Socket 1 Info |    ...        |                           |
               |  +--------------+---------------+---------------+                           |
               |    |                   cos_id=0               cos_id=1          ...         |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |->Ref   : |         ref 0         |         ref 1         | ...       | |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |->L3 CAT: |         cos 0         |         cos 1         | ...       | |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |->L2 CAT: |         cos 0         |         cos 1         | ...       | |
               |    |          +-----------------------+-----------------------+-----------+ |
               |    |          +-----------+-----------+-----------+-----------+-----------+ |
               |    |->CDP   : | cos0 code | cos0 data | cos1 code | cos1 data | ...       | |
               |               +-----------+-----------+-----------+-----------+-----------+ |
               +-----------------------------------------------------------------------------+

So, we need define a socket info data structure, 'struct
psr_socket_info' to manage information per socket. It contains a
reference count array according to COS ID and a feature array to
manage all features enabled. Every entry of the reference count
array is used to record how many domains are using the COS registers
according to the COS ID. For example, L3 CAT and L2 CAT are enabled,
Dom1 uses COS_ID=1 registers of both features to save CBM values, like
below.
        +-------+-------+-------+-----+
        | COS 0 | COS 1 | COS 2 | ... |
        +-------+-------+-------+-----+
L3 CAT  | 0x7ff | 0x1ff | ...   | ... |
        +-------+-------+-------+-----+
L2 CAT  | 0xff  | 0xff  | ...   | ... |
        +-------+-------+-------+-----+

If Dom2 has same CBM values, it can reuse these registers which COS_ID=1.
That means, both Dom1 and Dom2 use same COS registers(ID=1) to save same
L3/L2 values. So, the value ref[1] is 2 which means 2 domains are using
COS_ID 1.

To manage a feature, we need define a feature node data structure,
'struct feat_node', to manage feature's specific HW info, its callback
functions (all feature's specific behaviors are encapsulated into these
callback functions), and an array of all COS registers values of this
feature.

CDP is a special feature which uses two entries of the array
for one COS ID. So, the number of CDP COS registers is the half of L3
CAT. E.g. L3 CAT has 16 COS registers, then CDP has 8 COS registers if
it is enabled. CDP uses the COS registers array as below.

                         +-----------+-----------+-----------+-----------+-----------+
CDP cos_reg_val[] index: |     0     |     1     |     2     |     3     |    ...    |
                         +-----------+-----------+-----------+-----------+-----------+
                  value: | cos0 code | cos0 data | cos1 code | cos1 data |    ...    |
                         +-----------+-----------+-----------+-----------+-----------+

For more details, please refer SDM and patches to implement 'get value' and
'set value'.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
v9:
    - replace feature list to a feature pointer array.
      (suggested by Roger Pau)
    - add 'PSR_SOCKET_MAX_FEAT' in 'enum psr_feat_type' to know features
      account.
      (suggested by Roger Pau)
    - move 'feat_ops' declaration into 'feat_node' structure.
      (suggested by Roger Pau)
    - directly use uninon for feature HW info and move its declaration into
      'feat_node' structure.
      (suggested by Roger Pau)
    - remove 'enum psr_feat_type feature' declared in 'feat_ops' because it is
      not useful after using feature pointer array.
      (suggested by Roger Pau)
    - rename 'l3_cat_info' to 'cat_info' to be used by all CAT/CDP features.
    - remove 'nr_feat' which is only for a record.
      (suggested by Jan Beulich)
    - add 'cos_num' to record how many COS registers are used by a feature in
      one time access.
      (suggested by Jan Beulich)
    - replace 'uint64_t' to 'uint32_t' for cbm value because SDM specifies the
      max 32 bits for it.
      (suggested by Jan Beulich)
v7:
    - sort inclusion files position.
      (suggested by Wei Liu)
v6:
    - make commit message be clearer.
      (suggested by Konrad Rzeszutek Wilk)
    - fix wordings.
      (suggested by Konrad Rzeszutek Wilk)
    - add comments to explain relationship between 'feat_mask' and
      'enum psr_feat_type'.
      (suggested by Konrad Rzeszutek Wilk)
v5:
    - remove section number.
      (suggested by Jan Beulich)
    - remove double blank.
      (suggested by Jan Beulich)
v4:
    - create this patch because of removing all old CAT/CDP codes to make
      implementation be more easily understood.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 97 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 96a8589..822f1c0 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -13,16 +13,112 @@
  * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
  * more details.
  */
-#include <xen/init.h>
 #include <xen/cpu.h>
 #include <xen/err.h>
+#include <xen/init.h>
 #include <xen/sched.h>
 #include <asm/psr.h>
 
+/*
+ * Terminology:
+ * - CAT         Cache Allocation Technology
+ * - CBM         Capacity BitMasks
+ * - CDP         Code and Data Prioritization
+ * - COS/CLOS    Class of Service. Also mean COS registers.
+ * - COS_MAX     Max number of COS for the feature (minus 1)
+ * - MSRs        Machine Specific Registers
+ * - PSR         Intel Platform Shared Resource
+ */
+
 #define PSR_CMT        (1<<0)
 #define PSR_CAT        (1<<1)
 #define PSR_CDP        (1<<2)
 
+/*
+ * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
+ * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
+ * up to 128 L3 CAT Classes of Service. The COS_ID=[0,127].
+ *
+ * The MSRs ranging from 0D10H through 0D4FH (inclusive), enables support for
+ * up to 64 L2 CAT COS. The COS_ID=[0,63].
+ *
+ * So, the maximum COS register count of one feature is 128.
+ */
+#define MAX_COS_REG_CNT  128
+
+enum psr_feat_type {
+    PSR_SOCKET_L3_CAT = 0,
+    PSR_SOCKET_L3_CDP,
+    PSR_SOCKET_L2_CAT,
+    PSR_SOCKET_MAX_FEAT,
+};
+
+/* CAT/CDP HW info data structure. */
+struct psr_cat_hw_info {
+    unsigned int cbm_len;
+    unsigned int cos_max;
+};
+
+/*
+ * This structure represents one feature.
+ * feat_ops    - Feature operation callback functions.
+ * info        - Feature HW info.
+ * cos_reg_val - Array to store the values of COS registers. One entry stores
+ *               the value of one COS register.
+ *               For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
+ *               For CDP, two entries correspond to one COS_ID. E.g.
+ *               COS_ID=0 corresponds to cos_reg_val[0] (Data) and
+ *               cos_reg_val[1] (Code).
+ * cos_num     - COS registers number that feature uses in one time access.
+ */
+struct feat_node {
+    /*
+     * This structure defines feature operation callback functions. Every feature
+     * enabled MUST implement such callback functions and register them to ops.
+     *
+     * Feature specific behaviors will be encapsulated into these callback
+     * functions. Then, the main flows will not be changed when introducing a new
+     * feature.
+     */
+    struct feat_ops {
+        /* get_cos_max is used to get feature's cos_max. */
+        unsigned int (*get_cos_max)(const struct feat_node *feat);
+    } ops;
+
+    /* Encapsulate feature specific HW info here. */
+    union {
+        struct psr_cat_hw_info cat_info;
+    } info;
+
+    uint32_t cos_reg_val[MAX_COS_REG_CNT];
+    unsigned int cos_num;
+};
+
+/*
+ * PSR features are managed per socket. Below structure defines the members
+ * used to manage these features.
+ * feat_mask - Mask used to record features enabled on socket. There may be
+ *             some features enabled at same time.
+ * features  - An feature node array used to manage all features enabled.
+ * cos_ref   - A reference count array to record how many domains are using the
+ *             COS_ID.
+ *             Every entry of cos_ref corresponds to one COS ID.
+ * ref_lock  - A lock to protect cos_ref.
+ */
+struct psr_socket_info {
+    /*
+     * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
+     * psr_feat_type' means the bit position.
+     * bit 0:   L3 CAT
+     * bit 1:   L3 CDP
+     * bit 2:   L2 CAT
+     */
+    unsigned int feat_mask;
+    struct feat_node *features[PSR_SOCKET_MAX_FEAT];
+    unsigned int cos_ref[MAX_COS_REG_CNT];
+    spinlock_t ref_lock;
+};
+
 struct psr_assoc {
     uint64_t val;
     uint64_t cos_mask;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 04/25] x86: move cpuid_count_leaf from cpuid.c to processor.h.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (2 preceding siblings ...)
  2017-03-16 11:07 ` [PATCH v9 03/25] x86: refactor psr: implement main data structures Yi Sun
@ 2017-03-16 11:07 ` Yi Sun
  2017-03-24 16:22   ` Jan Beulich
  2017-03-16 11:07 ` [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow Yi Sun
                   ` (21 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch moves 'cpuid_count_leaf' from cpuid.c to processor.h to
make it available to external codes.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - create this patch alone to move 'cpuid_count_leaf'.
      (suggested by Wei Liu)
v6:
    - use 'struct cpuid_leaf' in psr.c. So we have to access 'cpuid_count_leaf'
      which has to be moved to processor.h.
      (suggested by Andrew Cooper)
---
 xen/arch/x86/cpuid.c            | 6 ------
 xen/include/asm-x86/processor.h | 7 +++++++
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/xen/arch/x86/cpuid.c b/xen/arch/x86/cpuid.c
index d6f6b88..13a28ca 100644
--- a/xen/arch/x86/cpuid.c
+++ b/xen/arch/x86/cpuid.c
@@ -35,12 +35,6 @@ static void cpuid_leaf(uint32_t leaf, struct cpuid_leaf *data)
     cpuid(leaf, &data->a, &data->b, &data->c, &data->d);
 }
 
-static void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
-                             struct cpuid_leaf *data)
-{
-    cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
-}
-
 static void sanitise_featureset(uint32_t *fs)
 {
     /* for_each_set_bit() uses unsigned longs.  Extend with zeroes. */
diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h
index dda8b83..2588a1b 100644
--- a/xen/include/asm-x86/processor.h
+++ b/xen/include/asm-x86/processor.h
@@ -13,6 +13,7 @@
 #include <asm/types.h>
 #include <asm/cpufeature.h>
 #include <asm/desc.h>
+#include <asm/x86_emulate.h>
 #endif
 
 #include <asm/x86-defns.h>
@@ -261,6 +262,12 @@ static always_inline unsigned int cpuid_count_ebx(
     return ebx;
 }
 
+static always_inline void cpuid_count_leaf(uint32_t leaf, uint32_t subleaf,
+                                           struct cpuid_leaf *data)
+{
+    cpuid_count(leaf, subleaf, &data->a, &data->b, &data->c, &data->d);
+}
+
 static inline unsigned long read_cr0(void)
 {
     unsigned long cr0;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (3 preceding siblings ...)
  2017-03-16 11:07 ` [PATCH v9 04/25] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
@ 2017-03-16 11:07 ` Yi Sun
  2017-03-24 16:52   ` Jan Beulich
  2017-03-16 11:07 ` [PATCH v9 06/25] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
                   ` (20 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements the CPU init and free flow including L3 CAT
initialization and feature array free. It includes below flows:
1. presmp init:
    - parse command line parameter.
    - allocate socket info for every socket.
    - allocate feature resource.
    - initialize socket info, get feature info and add feature into feature
      array per cpuid result.
    - free resources allocated if error happens.
    - register cpu notifier to handle cpu events.
2. cpu notifier:
    - handle cpu online events, if initialization work has been done before,
      do nothing.
    - handle cpu offline events, if it is the last cpu offline, free feature
      resources.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - add commit message to explain the flows.
    - handle cpu offline and online again case to read MSRs registers values
      back and save them into cos array to make user can get real data.
    - create a new patch about moving 'cpuid_count_leaf'.
      (suggested by Wei Liu)
    - modify comment to explain why not free some resource in 'free_feature'.
      (suggested by Wei Liu)
    - implement 'psr_alloc_feat_enabled' to check if allocation feature is
      enabled in cmdline and some initialization work done.
      (suggested by Wei Liu)
    - implement 'cat_default_val' to set default value for CAT features.
      (suggested by Wei Liu)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - implement a common 'cat_init_feature' to replace L3 CAT/L2 CAT specific
      init functions.
      (suggested by Roger Pau)
    - modify comments for global feature node.
      (suggested by Jan Beulich)
    - remove unnecessary comments.
      (suggested by Jan Beulich)
    - remove unnecessary 'else'.
      (suggested by Jan Beulich)
    - remove 'nr_feat'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - check global flag with boot cpu operations.
      (suggested by Jan Beulich)
    - remove 'cpu_init_work' and move codes into 'psr_cpu_init'.
      (suggested by Jan Beulich)
    - remove 'cpu_fini_work' and move codes into 'psr_cpu_fini'.
      (suggested by Jan Beulich)
    - assign value for 'cos_num'.
      (suggested by Jan Beulich)
    - change about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v8:
    - fix format issue.
      (suggested by Konrad Rzeszutek Wilk)
    - add comments to explain why we care about cpumask_empty when the last
      cpu on socket is offline.
      (suggested by Konrad Rzeszutek Wilk)
v7:
    - initialize structure objects for avoiding surprise.
      (suggested by Konrad Rzeszutek Wilk)
    - fix typo.
      (suggested by Konrad Rzeszutek Wilk)
    - fix a logical mistake when handling the last cpu offline event.
      (suggested by Konrad Rzeszutek Wilk)
v6:
    - use 'struct cpuid_leaf' introduced in Andrew's patch.
      (suggested by Konrad Rzeszutek Wilk)
    - add comments about cpu_add_remove_lock.
      (suggested by Konrad Rzeszutek Wilk)
    - change 'clear_bit' to '__clear_bit'.
      (suggested by Konrad Rzeszutek Wilk)
    - add 'ASSERT' check when setting 'feat_mask'.
      (suggested by Konrad Rzeszutek Wilk)
    - adjust 'printk' position to avoid odd spacing.
      (suggested by Konrad Rzeszutek Wilk)
    - add comment to explain usage of 'feat_l3_cat'.
      (suggested by Konrad Rzeszutek Wilk)
    - fix wording.
      (suggested by Konrad Rzeszutek Wilk)
    - move 'cpuid_count_leaf' helper function to 'asm-x86/processor.h'.
      It cannot be moved to 'cpuid.h' which causes compilation error because
      of header file loop reference.
      (suggested by Andrew Cooper)
v5:
    - add comment to explain the reason to define 'feat_l3_cat'.
      (suggested by Jan Beulich)
    - use 'list_for_each_entry_safe'.
      (suggested by Jan Beulich)
    - remove codes to free 'feat_l3_cat' in 'free_feature' to avoid the need
      for an allocation the next time a CPU comes online.
      (suggested by Jan Beulich)
    - define 'struct cpuid_leaf_regs' to encapsulate eax~edx.
      (suggested by Jan Beulich)
    - print feature info on a socket only when 'opt_cpu_info' is true.
      (suggested by Jan Beulich)
    - declare global variable 'l3_cat_ops' to 'static const'.
      (suggested by Jan Beulich)
    - use 'current_cpu_data'.
      (suggested by Jan Beulich)
    - rename 'feat_tmp' to 'feat'.
      (suggested by Jan Beulich)
    - clear PQE feature bit when the maximum CPUID level is too low.
      (suggested by Jan Beulich)
    - directly call 'l3_cat_init_feature'. No need to make it a callback
      function.
      (suggested by Jan Beulich)
    - remove local variable 'info'.
      (suggested by Jan Beulich)
    - move 'INIT_LIST_HEAD' into 'cpu_init_work' to be together with
      spin_lock_init().
      (suggested by Jan Beulich)
    - remove 'cpu_prepare_work' and move its content into 'psr_cpu_prepare'.
      (suggested by Jan Beulich)
v4:
    - create this patch because of removing all CAT/CDP old codes to make
      implementation be more easily understood.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 202 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 197 insertions(+), 5 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 822f1c0..66a9ce8 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -18,6 +18,7 @@
 #include <xen/init.h>
 #include <xen/sched.h>
 #include <asm/psr.h>
+#include <asm/x86_emulate.h>
 
 /*
  * Terminology:
@@ -34,6 +35,9 @@
 #define PSR_CAT        (1<<1)
 #define PSR_CDP        (1<<2)
 
+#define CAT_CBM_LEN_MASK 0x1f
+#define CAT_COS_MAX_MASK 0xffff
+
 /*
  * Per SDM chapter 'Cache Allocation Technology: Cache Mask Configuration',
  * the MSRs ranging from 0C90H through 0D0FH (inclusive), enables support for
@@ -46,6 +50,9 @@
  */
 #define MAX_COS_REG_CNT  128
 
+/* CAT features use 1 COS register in one access. */
+#define CAT_COS_NUM      1
+
 enum psr_feat_type {
     PSR_SOCKET_L3_CAT = 0,
     PSR_SOCKET_L3_CDP,
@@ -126,11 +133,110 @@ struct psr_assoc {
 
 struct psr_cmt *__read_mostly psr_cmt;
 
+static struct psr_socket_info *__read_mostly socket_info;
+
 static unsigned int opt_psr;
 static unsigned int __initdata opt_rmid_max = 255;
+static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
 static uint64_t rmid_mask;
 static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
 
+/*
+ * Declare global feature node for every feature to facilitate the feature
+ * array creation. It is used to transiently store a spare node.
+ */
+static struct feat_node *feat_l3_cat;
+
+/* Common functions */
+#define cat_default_val(len)                 \
+            ( (uint32_t)((1ul << len) - 1) )
+
+/*
+ * Use this function to check if any allocation feature has been enabled
+ * in cmdline.
+ */
+static bool psr_alloc_feat_enabled(void)
+{
+    return ((!socket_info) ? false : true );
+}
+
+static void free_feature(struct psr_socket_info *info)
+{
+    unsigned int i;
+
+    if ( !info )
+        return;
+
+    /*
+     * Free resources of features. The global feature object, e.g. feat_l3_cat,
+     * may not be freed here if it is not added into array. It is simply being
+     * kept until the next CPU online attempt.
+     */
+    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
+    {
+        if ( !info->features[i] )
+            continue;
+
+        xfree(info->features[i]);
+        info->features[i] = NULL;
+        __clear_bit(i, &info->feat_mask);
+    }
+}
+
+/* CAT common functions implementation. */
+static void cat_init_feature(struct cpuid_leaf regs,
+                             struct feat_node *feat,
+                             struct psr_socket_info *info,
+                             enum psr_feat_type type)
+{
+    unsigned int socket, i;
+    struct psr_cat_hw_info cat = { };
+    uint64_t val;
+
+    /* No valid value so do not enable feature. */
+    if ( !regs.a || !regs.d )
+        return;
+
+    cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
+    cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
+
+    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+    feat->cos_reg_val[0] = cat_default_val(cat.cbm_len);
+    /*
+     * To handle cpu offline and then online case, we need read MSRs back to
+     * save values into cos_reg_val array.
+     */
+    for ( i = 1; i <= cat.cos_max; i++ )
+    {
+        rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
+        feat->cos_reg_val[i] = (uint32_t)val;
+    }
+
+    feat->info.cat_info = cat;
+    feat->cos_num = CAT_COS_NUM;
+
+    /* Add this feature into array. */
+    info->features[type] = feat;
+
+    ASSERT(!test_bit(type, &info->feat_mask));
+    __set_bit(type, &info->feat_mask);
+
+    socket = cpu_to_socket(smp_processor_id());
+    if ( !opt_cpu_info )
+        return;
+
+    printk(XENLOG_INFO "%s CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+           ((type == PSR_SOCKET_L3_CAT) ? "L3" : "L2"),
+           socket, feat->info.cat_info.cos_max,
+           feat->info.cat_info.cbm_len);
+
+    return;
+}
+
+/* L3 CAT ops */
+static const struct feat_ops l3_cat_ops = {
+};
+
 static void __init parse_psr_bool(char *s, char *value, char *feature,
                                   unsigned int mask)
 {
@@ -170,6 +276,9 @@ static void __init parse_psr_param(char *s)
         if ( val_str && !strcmp(s, "rmid_max") )
             opt_rmid_max = simple_strtoul(val_str, NULL, 0);
 
+        if ( val_str && !strcmp(s, "cos_max") )
+            opt_cos_max = simple_strtoul(val_str, NULL, 0);
+
         s = ss + 1;
     } while ( ss );
 }
@@ -325,19 +434,98 @@ void psr_domain_free(struct domain *d)
     psr_free_rmid(d);
 }
 
-static int psr_cpu_prepare(unsigned int cpu)
+static void __init init_psr(void)
 {
+    if ( opt_cos_max < 1 )
+    {
+        printk(XENLOG_INFO "CAT: disabled, cos_max is too small\n");
+        return;
+    }
+
+    socket_info = xzalloc_array(struct psr_socket_info, nr_sockets);
+
+    if ( !socket_info )
+    {
+        printk(XENLOG_INFO "Failed to alloc socket_info!\n");
+        return;
+    }
+}
+
+static void __init psr_free(void)
+{
+    xfree(socket_info);
+    socket_info = NULL;
+}
+
+static int psr_cpu_prepare(void)
+{
+    if ( !psr_alloc_feat_enabled() )
+        return 0;
+
+    /* Malloc memory for the global feature head here. */
+    if ( feat_l3_cat == NULL &&
+         (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
+        return -ENOMEM;
+
     return 0;
 }
 
 static void psr_cpu_init(void)
 {
+    struct psr_socket_info *info;
+    unsigned int socket, i;
+    unsigned int cpu = smp_processor_id();
+    struct feat_node *feat;
+    struct cpuid_leaf regs;
+
+    if ( !psr_alloc_feat_enabled() || !boot_cpu_has(X86_FEATURE_PQE) )
+        goto assoc_init;
+
+    if ( boot_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
+    {
+        setup_clear_cpu_cap(X86_FEATURE_PQE);
+        goto assoc_init;
+    }
+
+    socket = cpu_to_socket(cpu);
+    info = socket_info + socket;
+    if ( info->feat_mask )
+        goto assoc_init;
+
+    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
+        info->features[i] = NULL;
+
+    spin_lock_init(&info->ref_lock);
+
+    cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, &regs);
+    if ( regs.b & PSR_RESOURCE_TYPE_L3 )
+    {
+        cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, &regs);
+
+        feat = feat_l3_cat;
+        feat_l3_cat = NULL;
+        feat->ops = l3_cat_ops;
+
+        cat_init_feature(regs, feat, info, PSR_SOCKET_L3_CAT);
+    }
+
+assoc_init:
     psr_assoc_init();
 }
 
 static void psr_cpu_fini(unsigned int cpu)
 {
-    return;
+    unsigned int socket = cpu_to_socket(cpu);
+
+    if ( !psr_alloc_feat_enabled() )
+        return;
+
+    /*
+     * We only free when we are the last CPU in the socket. The socket_cpumask
+     * is cleared prior to this notification code by remove_siblinginfo().
+     */
+    if ( socket_cpumask[socket] && cpumask_empty(socket_cpumask[socket]) )
+        free_feature(socket_info + socket);
 }
 
 static int cpu_callback(
@@ -349,7 +537,7 @@ static int cpu_callback(
     switch ( action )
     {
     case CPU_UP_PREPARE:
-        rc = psr_cpu_prepare(cpu);
+        rc = psr_cpu_prepare();
         break;
     case CPU_STARTING:
         psr_cpu_init();
@@ -378,10 +566,14 @@ static int __init psr_presmp_init(void)
     if ( (opt_psr & PSR_CMT) && opt_rmid_max )
         init_psr_cmt(opt_rmid_max);
 
-    psr_cpu_prepare(0);
+    if ( opt_psr & PSR_CAT )
+        init_psr();
+
+    if ( psr_cpu_prepare() )
+        psr_free();
 
     psr_cpu_init();
-    if ( psr_cmt_enabled() )
+    if ( psr_cmt_enabled() || psr_alloc_feat_enabled() )
         register_cpu_notifier(&cpu_nfb);
 
     return 0;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 06/25] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (4 preceding siblings ...)
  2017-03-16 11:07 ` [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow Yi Sun
@ 2017-03-16 11:07 ` Yi Sun
  2017-03-16 11:07 ` [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
                   ` (19 subsequent siblings)
  25 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements the Domain init/free and schedule flows.
- When domain init, its psr resource should be allocated.
- When domain free, its psr resource should be freed too.
- When domain is scheduled, its COS ID on the socket should be
  set into ASSOC register to make corresponding COS MSR value
  work.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
v9:
    - rename 'l3_cat_get_cos_max' to 'cat_get_cos_max' to cover all CAT/CDP
      features.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - implement 'psr_alloc_cos' to match 'psr_free_cos'.
      (suggested by Wei Liu)
    - use 'psr_alloc_feat_enabled'.
      (suggested by Wei Liu)
    - fix coding style issue.
      (suggested by Wei Liu)
    - remove 'inline'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - remove 'psr_cos_ids' check in 'psr_free_cos'.
      (suggested by Jan Beulich)
v6:
    - change 'PSR_ASSOC_REG_POS' to 'PSR_ASSOC_REG_SHIFT'.
      (suggested by Konrad Rzeszutek Wilk)
v5:
    - rename 'feat_tmp' to 'feat'.
      (suggested by Jan Beulich)
    - define 'PSR_ASSOC_REG_POS'.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 74 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 66a9ce8..a6a8081 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -50,6 +50,8 @@
  */
 #define MAX_COS_REG_CNT  128
 
+#define PSR_ASSOC_REG_SHIFT 32
+
 /* CAT features use 1 COS register in one access. */
 #define CAT_COS_NUM      1
 
@@ -233,8 +235,14 @@ static void cat_init_feature(struct cpuid_leaf regs,
     return;
 }
 
+static unsigned int cat_get_cos_max(const struct feat_node *feat)
+{
+    return feat->info.cat_info.cos_max;
+}
+
 /* L3 CAT ops */
 static const struct feat_ops l3_cat_ops = {
+    .get_cos_max = cat_get_cos_max,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -378,11 +386,39 @@ void psr_free_rmid(struct domain *d)
     d->arch.psr_rmid = 0;
 }
 
-static inline void psr_assoc_init(void)
+static unsigned int get_max_cos_max(const struct psr_socket_info *info)
+{
+    const struct feat_node *feat;
+    unsigned int cos_max = 0, i;
+
+    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
+    {
+        feat = info->features[i];
+        if ( !feat )
+            continue;
+
+        cos_max = max(feat->ops.get_cos_max(feat), cos_max);
+    }
+
+    return cos_max;
+}
+
+static void psr_assoc_init(void)
 {
     struct psr_assoc *psra = &this_cpu(psr_assoc);
 
-    if ( psr_cmt_enabled() )
+    if ( psr_alloc_feat_enabled() )
+    {
+        unsigned int socket = cpu_to_socket(smp_processor_id());
+        const struct psr_socket_info *info = socket_info + socket;
+        unsigned int cos_max = get_max_cos_max(info);
+
+        if ( info->feat_mask )
+            psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
+                             PSR_ASSOC_REG_SHIFT;
+    }
+
+    if ( psr_cmt_enabled() || psra->cos_mask )
         rdmsrl(MSR_IA32_PSR_ASSOC, psra->val);
 }
 
@@ -391,6 +427,13 @@ static inline void psr_assoc_rmid(uint64_t *reg, unsigned int rmid)
     *reg = (*reg & ~rmid_mask) | (rmid & rmid_mask);
 }
 
+static void psr_assoc_cos(uint64_t *reg, unsigned int cos,
+                          uint64_t cos_mask)
+{
+    *reg = (*reg & ~cos_mask) |
+            (((uint64_t)cos << PSR_ASSOC_REG_SHIFT) & cos_mask);
+}
+
 void psr_ctxt_switch_to(struct domain *d)
 {
     struct psr_assoc *psra = &this_cpu(psr_assoc);
@@ -399,6 +442,11 @@ void psr_ctxt_switch_to(struct domain *d)
     if ( psr_cmt_enabled() )
         psr_assoc_rmid(&reg, d->arch.psr_rmid);
 
+    if ( psra->cos_mask )
+        psr_assoc_cos(&reg, d->arch.psr_cos_ids ?
+                      d->arch.psr_cos_ids[cpu_to_socket(smp_processor_id())] :
+                      0, psra->cos_mask);
+
     if ( reg != psra->val )
     {
         wrmsrl(MSR_IA32_PSR_ASSOC, reg);
@@ -424,14 +472,37 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket,
     return 0;
 }
 
-int psr_domain_init(struct domain *d)
+/* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
+static void psr_free_cos(struct domain *d)
+{
+    xfree(d->arch.psr_cos_ids);
+    d->arch.psr_cos_ids = NULL;
+}
+
+static int psr_alloc_cos(struct domain *d)
 {
+    d->arch.psr_cos_ids = xzalloc_array(unsigned int, nr_sockets);
+    if ( !d->arch.psr_cos_ids )
+        return -ENOMEM;
+
     return 0;
 }
 
+int psr_domain_init(struct domain *d)
+{
+    /* Init to success value */
+    int ret = 0;
+
+    if ( psr_alloc_feat_enabled() )
+        ret = psr_alloc_cos(d);
+
+    return ret;
+}
+
 void psr_domain_free(struct domain *d)
 {
     psr_free_rmid(d);
+    psr_free_cos(d);
 }
 
 static void __init init_psr(void)
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (5 preceding siblings ...)
  2017-03-16 11:07 ` [PATCH v9 06/25] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
@ 2017-03-16 11:07 ` Yi Sun
  2017-03-27  9:07   ` Jan Beulich
  2017-03-16 11:07 ` [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
                   ` (18 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements get HW info flow including L3 CAT callback
function.

It also changes sysctl interface to make it more general.

With this patch, 'psr-hwinfo' can work for L3 CAT.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v9:
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - define 'PSR_INFO_SIZE'.
      (suggested by Roger Pau)
    - fix coding style issue.
      (suggested by Roger Pau and Jan Beulich)
    - use 'ARRAY_SIZE'.
      (suggested by Roger Pau)
    - rename 'l3_cat_get_feat_info' to 'cat_get_feat_info' to make it a common
      function for both L3/L2 CAT.
      (suggested by Roger Pau)
    - move constant to the right of comparison.
      (suggested by Wei Liu)
    - remove wrong comment.
      (suggested by Jan Beulich)
    - rename macros used by psr_get_info to make them meaningful.
      (suggested by Jan Beulich)
    - remove assignment for 'PSR_SOCKET_UNKNOWN'.
      (suggested by Jan Beulich)
    - retain blank line after 'case XEN_SYSCTL_PSR_CAT_get_l3_info'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - move common data check into common function.
      (suggested by Jan Beulich)
v6:
    - fix coding style issue.
      (suggested by Konrad Rzeszutek Wilk)
    - define 'PSR_SOCKET_UNKNOWN' in 'psr_feat_type'.
      (suggested by Konrad Rzeszutek Wilk)
    - change '-ENOTSOCK' to 'ERANGE'.
      (suggested by Konrad Rzeszutek Wilk)
    - modify position of macros to remove odd spacing in psr.h.
      (suggested by Konrad Rzeszutek Wilk)
v5:
    - change 'dat[]' to 'data[]'.
      (suggested by Jan Beulich)
    - modify parameter type to avoid fixed width type when there is no such
      intention.
      (suggested by Jan Beulich)
    - use 'const' when it is possible.
      (suggested by Jan Beulich)
    - check feature type outside callback function.
      (suggested by Jan Beulich)
    - modify macros names to add prefix 'PSR_' and change 'CDP_FLAG' to
      'PSR_FLAG'.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c        | 74 +++++++++++++++++++++++++++++++++++++++++++++--
 xen/arch/x86/sysctl.c     | 14 ++++++---
 xen/include/asm-x86/psr.h | 17 +++++++----
 3 files changed, 93 insertions(+), 12 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index a6a8081..83358e3 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -60,6 +60,7 @@ enum psr_feat_type {
     PSR_SOCKET_L3_CDP,
     PSR_SOCKET_L2_CAT,
     PSR_SOCKET_MAX_FEAT,
+    PSR_SOCKET_UNKNOWN,
 };
 
 /* CAT/CDP HW info data structure. */
@@ -92,6 +93,10 @@ struct feat_node {
     struct feat_ops {
         /* get_cos_max is used to get feature's cos_max. */
         unsigned int (*get_cos_max)(const struct feat_node *feat);
+
+        /* get_feat_info is used to get feature HW info. */
+        bool (*get_feat_info)(const struct feat_node *feat,
+                              uint32_t data[], unsigned int array_len);
     } ops;
 
     /* Encapsulate feature specific HW info here. */
@@ -185,6 +190,23 @@ static void free_feature(struct psr_socket_info *info)
     }
 }
 
+static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
+{
+    enum psr_feat_type feat_type;
+
+    switch ( type )
+    {
+    case PSR_CBM_TYPE_L3:
+        feat_type = PSR_SOCKET_L3_CAT;
+        break;
+    default:
+        feat_type = PSR_SOCKET_UNKNOWN;
+        break;
+    }
+
+    return feat_type;
+}
+
 /* CAT common functions implementation. */
 static void cat_init_feature(struct cpuid_leaf regs,
                              struct feat_node *feat,
@@ -240,9 +262,23 @@ static unsigned int cat_get_cos_max(const struct feat_node *feat)
     return feat->info.cat_info.cos_max;
 }
 
+static bool cat_get_feat_info(const struct feat_node *feat,
+                              uint32_t data[], unsigned int array_len)
+{
+    if ( array_len != PSR_INFO_CAT_SIZE )
+        return false;
+
+    data[PSR_INFO_IDX_CBM_LEN] = feat->info.cat_info.cbm_len;
+    data[PSR_INFO_IDX_COS_MAX] = feat->info.cat_info.cos_max;
+    data[PSR_INFO_IDX_FLAG] = 0;
+
+    return true;
+}
+
 /* L3 CAT ops */
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = cat_get_cos_max,
+    .get_feat_info = cat_get_feat_info,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -454,10 +490,42 @@ void psr_ctxt_switch_to(struct domain *d)
     }
 }
 
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
-                        uint32_t *cos_max, uint32_t *flags)
+static struct psr_socket_info *get_socket_info(unsigned int socket)
 {
-    return 0;
+    if ( !socket_info )
+        return ERR_PTR(-ENODEV);
+
+    if ( socket >= nr_sockets )
+        return ERR_PTR(-ERANGE);
+
+    if ( !socket_info[socket].feat_mask )
+        return ERR_PTR(-ENOENT);
+
+    return socket_info + socket;
+}
+
+int psr_get_info(unsigned int socket, enum cbm_type type,
+                 uint32_t data[], unsigned int array_len)
+{
+    const struct psr_socket_info *info = get_socket_info(socket);
+    const struct feat_node *feat;
+    enum psr_feat_type feat_type;
+
+    if ( IS_ERR(info) )
+        return PTR_ERR(info);
+
+    if ( !data )
+        return -EINVAL;
+
+    feat_type = psr_cbm_type_to_feat_type(type);
+    feat = info->features[feat_type];
+    if ( !feat )
+        return -ENOENT;
+
+    if ( feat->ops.get_feat_info(feat, data, array_len) )
+        return 0;
+
+    return -EINVAL;
 }
 
 int psr_get_l3_cbm(struct domain *d, unsigned int socket,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 2f7056e..aab3d2d 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -175,14 +175,20 @@ long arch_do_sysctl(
         switch ( sysctl->u.psr_cat_op.cmd )
         {
         case XEN_SYSCTL_PSR_CAT_get_l3_info:
-            ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
-                                      &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
-                                      &sysctl->u.psr_cat_op.u.l3_info.cos_max,
-                                      &sysctl->u.psr_cat_op.u.l3_info.flags);
+        {
+            uint32_t data[PSR_INFO_CAT_SIZE];
+
+            ret = psr_get_info(sysctl->u.psr_cat_op.target,
+                               PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
+
+            sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[PSR_INFO_IDX_CBM_LEN];
+            sysctl->u.psr_cat_op.u.l3_info.cos_max = data[PSR_INFO_IDX_COS_MAX];
+            sysctl->u.psr_cat_op.u.l3_info.flags   = data[PSR_INFO_IDX_FLAG];
 
             if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
                 ret = -EFAULT;
             break;
+        }
 
         default:
             ret = -EOPNOTSUPP;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 57f47e9..e27e685 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -19,20 +19,27 @@
 #include <xen/types.h>
 
 /* CAT cpuid level */
-#define PSR_CPUID_LEVEL_CAT   0x10
+#define PSR_CPUID_LEVEL_CAT             0x10
 
 /* Resource Type Enumeration */
 #define PSR_RESOURCE_TYPE_L3            0x2
 
 /* L3 Monitoring Features */
-#define PSR_CMT_L3_OCCUPANCY           0x1
+#define PSR_CMT_L3_OCCUPANCY            0x1
 
 /* CDP Capability */
-#define PSR_CAT_CDP_CAPABILITY       (1u << 2)
+#define PSR_CAT_CDP_CAPABILITY          (1u << 2)
 
 /* L3 CDP Enable bit*/
 #define PSR_L3_QOS_CDP_ENABLE_BIT       0x0
 
+/* Used by psr_get_info() */
+#define PSR_INFO_IDX_CBM_LEN            0
+#define PSR_INFO_IDX_COS_MAX            1
+#define PSR_INFO_IDX_FLAG               2
+#define PSR_INFO_CAT_SIZE               3
+
+
 struct psr_cmt_l3 {
     unsigned int features;
     unsigned int upscaling_factor;
@@ -63,8 +70,8 @@ int psr_alloc_rmid(struct domain *d);
 void psr_free_rmid(struct domain *d);
 void psr_ctxt_switch_to(struct domain *d);
 
-int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
-                        uint32_t *cos_max, uint32_t *flags);
+int psr_get_info(unsigned int socket, enum cbm_type type,
+                 uint32_t data[], unsigned int array_len);
 int psr_get_l3_cbm(struct domain *d, unsigned int socket,
                    uint64_t *cbm, enum cbm_type type);
 int psr_set_l3_cbm(struct domain *d, unsigned int socket,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (6 preceding siblings ...)
  2017-03-16 11:07 ` [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
@ 2017-03-16 11:07 ` Yi Sun
  2017-03-27  9:23   ` Jan Beulich
  2017-03-16 11:07 ` [PATCH v9 09/25] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
                   ` (17 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

There is an interface in user space to show feature value of
domains.

This patch implements get value flow in hypervisor including
L3 CAT callback function.

It also changes domctl interface to make it more general.

With this patch, 'psr-cat-show' can work for L3 CAT but not for
L3 code/data which is implemented in patch "x86: refactor psr:
implement get value flow for CDP.".

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
v9:
    - add commit message to explain there is an user space interface.
    - rename 'l3_cat_get_val' to 'cat_get_val' to cover all L3/L2 CAT features.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - change parameter of 'psr_get'. Use 'psr_cos_ids' directly to replace
      domain. Also declare it to 'const'.
      (suggested by Jan Beulich)
    - change code flow to remove 'psr_get' but add 'psr_get_feat' to make codes
      more reasonable.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - move cos check into common function because this check is required by all
      features.
      (suggested by Jan Beulich)
    - fix coding style issue.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'..
      (suggested by Jan Beulich)
v7:
    - rename '__psr_get' to 'psr_get'.
      (suggested by Wei Liu)
v6:
    - modify commit message to make it clearer.
      (suggested by Konrad Rzeszutek Wilk)
    - remove one extra space in code.
      (suggested by Konrad Rzeszutek Wilk)
    - remove unnecessary comment.
      (suggested by Konrad Rzeszutek Wilk)
    - write a helper function to move get info and get val functions into
      it. Because most codes of 'get_info' and 'get_val' are same.
      (suggested by Konrad Rzeszutek Wilk)
v5:
    - rename 'dat[]' to 'data[]'
      (suggested by Jan Beulich)
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
      (suggested by Jan Beulich)
    - check if feature type match in caller of feature callback function.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/domctl.c     | 21 +++++++++--------
 xen/arch/x86/psr.c        | 58 +++++++++++++++++++++++++++++++++++++++++------
 xen/include/asm-x86/psr.h |  4 ++--
 3 files changed, 65 insertions(+), 18 deletions(-)

diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 02b48e8..e714d1d 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1455,23 +1455,26 @@ long arch_do_domctl(
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
-            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 &domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3);
+            domctl->u.psr_cat_op.data = 0;
+            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+                              (uint32_t *)&domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3);
             copyback = 1;
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
-            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 &domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3_CODE);
+            domctl->u.psr_cat_op.data = 0;
+            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+                              (uint32_t *)&domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3_CODE);
             copyback = 1;
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
-            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 &domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3_DATA);
+            domctl->u.psr_cat_op.data = 0;
+            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+                              (uint32_t *)&domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3_DATA);
             copyback = 1;
             break;
 
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 83358e3..53105f5 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -97,6 +97,10 @@ struct feat_node {
         /* get_feat_info is used to get feature HW info. */
         bool (*get_feat_info)(const struct feat_node *feat,
                               uint32_t data[], unsigned int array_len);
+
+        /* get_val is used to get feature COS register value. */
+        void (*get_val)(const struct feat_node *feat, unsigned int cos,
+                        enum cbm_type type, uint32_t *val);
     } ops;
 
     /* Encapsulate feature specific HW info here. */
@@ -275,10 +279,17 @@ static bool cat_get_feat_info(const struct feat_node *feat,
     return true;
 }
 
+static void cat_get_val(const struct feat_node *feat, unsigned int cos,
+                        enum cbm_type type, uint32_t *val)
+{
+    *val = feat->cos_reg_val[cos];
+}
+
 /* L3 CAT ops */
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = cat_get_cos_max,
     .get_feat_info = cat_get_feat_info,
+    .get_val = cat_get_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -504,21 +515,30 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
     return socket_info + socket;
 }
 
-int psr_get_info(unsigned int socket, enum cbm_type type,
-                 uint32_t data[], unsigned int array_len)
+static const struct feat_node * psr_get_feat(unsigned int socket,
+                                             enum cbm_type type)
 {
     const struct psr_socket_info *info = get_socket_info(socket);
     const struct feat_node *feat;
     enum psr_feat_type feat_type;
 
     if ( IS_ERR(info) )
-        return PTR_ERR(info);
+        return NULL;
+
+    feat_type = psr_cbm_type_to_feat_type(type);
+    feat = info->features[feat_type];
+    return feat;
+}
+
+int psr_get_info(unsigned int socket, enum cbm_type type,
+                 uint32_t data[], unsigned int array_len)
+{
+    const struct feat_node *feat;
 
     if ( !data )
         return -EINVAL;
 
-    feat_type = psr_cbm_type_to_feat_type(type);
-    feat = info->features[feat_type];
+    feat = psr_get_feat(socket, type);
     if ( !feat )
         return -ENOENT;
 
@@ -528,9 +548,33 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
     return -EINVAL;
 }
 
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
-                   uint64_t *cbm, enum cbm_type type)
+int psr_get_val(struct domain *d, unsigned int socket,
+                uint32_t *val, enum cbm_type type)
 {
+    const struct feat_node *feat;
+    unsigned int cos;
+
+    if ( !d || !val )
+        return -EINVAL;
+
+    feat = psr_get_feat(socket, type);
+    if ( !feat )
+        return -ENOENT;
+
+    cos = d->arch.psr_cos_ids[socket];
+    /*
+     * If input cos exceeds current feature's cos_max, we should return its
+     * default value which is stored in cos 0. This case only happens
+     * when more than two features enabled concurrently and at least one
+     * features's cos_max is bigger than others. When a domain's working cos
+     * id is bigger than some features' cos_max, HW automatically works as
+     * default value for those features which cos_max is smaller.
+     */
+    if ( cos > feat->ops.get_cos_max(feat) )
+        cos = 0;
+
+    feat->ops.get_val(feat, cos, type, val);
+
     return 0;
 }
 
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index e27e685..cd00733 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -72,8 +72,8 @@ void psr_ctxt_switch_to(struct domain *d);
 
 int psr_get_info(unsigned int socket, enum cbm_type type,
                  uint32_t data[], unsigned int array_len);
-int psr_get_l3_cbm(struct domain *d, unsigned int socket,
-                   uint64_t *cbm, enum cbm_type type);
+int psr_get_val(struct domain *d, unsigned int socket,
+                uint32_t *val, enum cbm_type type);
 int psr_set_l3_cbm(struct domain *d, unsigned int socket,
                    uint64_t cbm, enum cbm_type type);
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 09/25] x86: refactor psr: L3 CAT: set value: implement framework.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (7 preceding siblings ...)
  2017-03-16 11:07 ` [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
@ 2017-03-16 11:07 ` Yi Sun
  2017-03-27  9:59   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
                   ` (16 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:07 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

As set value flow is the most complicated one in psr, it will be
divided to some patches to make things clearer. This patch
implements the set value framework to show a whole picture firstly.

It also changes domctl interface to make it more general.

To make the set value flow be general and can support multiple features
at same time, it includes below steps:
1. Get COS ID that current domain is using.
2. Gather a value array to store all features current value
   into it and replace the current value of the feature which is
   being set to the new input value.
3. Find if there is already a COS ID on which all features'
   values are same as the array. Then, we can reuse this COS
   ID.
4. If fail to find, we need pick an available COS ID. Only COS ID which ref
   is 0 or 1 can be picked.
5. Write all features MSRs according to the COS ID.
6. Update ref according to COS ID.
7. Save the COS ID into current domain's psr_cos_ids[socket] so that we
   can know which COS the domain is using on the socket.

So, some functions are abstracted and the callback functions will be
implemented in next patches.

Here is an example to understand the process. The CPU supports
two featuers, e.g. L3 CAT and L2 CAT. User wants to set L3 CAT
of Dom1 to 0x1ff.
1. Get the old_cos of Dom1 which is 0. L3 CAT is the first
element of feature list. The COS registers values are below at
this time.
        -------------------------------
        | COS 0 | COS 1 | COS 2 | ... |
        -------------------------------
L3 CAT  | 0x7ff | ...   | ...   | ... |
        -------------------------------
L2 CAT  | 0xff  | ...   | ...   | ... |
        -------------------------------

2. Gather the value array and set new value into it:
val[0]: 0x1ff
val[1]: 0xff

3. It cannot find a matching COS.

4. Pick COS 1 to store the value set.

5. Write the COS 1 registers. The COS registers values are
changed to below now.
        -------------------------------
        | COS 0 | COS 1 | COS 2 | ... |
        -------------------------------
L3 CAT  | 0x7ff | 0x1ff | ...   | ... |
        -------------------------------
L2 CAT  | 0xff  | 0xff  | ...   | ... |
        -------------------------------

6. The ref[1] is increased to 1 because Dom1 is using it now.

7. Save 1 to Dom1's psr_cos_ids[socket].

Then, user wants to set L3 CAT of Dom2 to 0x1ff too. The old_cos
of Dom2 is 0 too. Repeat above flow.

The val array assembled is:
val[0]: 0x1ff
val[1]: 0xff

So, it can find a matching COS, COS 1. Then, it can reuse COS 1
for Dom2.

The ref[1] is increased to 2 now because both Dom1 and Dom2 are
using this COS ID. Set 1 to Dom2's psr_cos_ids[socket].

Another thing need to emphasize is the context switch. When context
switch happens, 'psr_ctxt_switch_to' is called by system to get
domain's COS ID from 'psr_cos_ids[socket]'. But 'psr_cos_ids[socket]'
is set at step 7 above. So, there are three scenarios, e.g.:
1. User calls domctl interface on Dom0 to set a COS ID 1 for Dom1 into its
   psr_cos_ids[]. Then, Dom1 is scheduled so that 'psr_ctxt_switch_to()' is
   called which makes COS ID 1 work. For this case, we do not any action.

2. Dom1 runs on CPU 1 and COS ID 1 is working. At same time, user calls domctl
   interface on Dom0 to set a new COS ID 2 for Dom1 into psr_cos_ids[]. After
   time slice ends, the Dom1 is scheduled again, the new COS ID 2 will work.

3. When a new COS ID is being set to psr_cos_ids[], 'psr_ctxt_switch_to()'
   is called to access the same psr_cos_ids[] member through 'psr_assoc_cos'.
   The COS ID is constrained by cos_mask so that it cannot exceeds the cos_max.
   So even the COS ID got here is wrong, it is still a workable ID (within
   cos_max). The functionality is still workable, only another valid CBM be
   effective for a short time. In next schedule, the correct CBM will take
   effect.

All these cases will not cause race condition and no harm to system. The PSR
features are to set cache capacity for a domain. The setting to cache is
progressively effective. When the cache setting becomes really effective, the
time slice to schedule a domain may have passed. So, even if a wrong COS ID is
used to set ASSOC, only another valid CBM be effective for a short time during
cache preparation time. The correct COS ID will take effect in a short time.
This does not affect cache capacity setting much.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - use goto style error handling in 'psr_set_val'.
      (suggested by Wei Liu)
    - use ASSERT for checking old_cos.
      (suggested by Wei Liu and Jan Beulich)
    - fix coding style issue.
      (suggested by Wei Liu)
    - rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
      (suggested by Wei Liu)
    - use 'spin_is_locked' to check ref_lock.
      (suggested by Roger Pau)
    - add an input parameter 'array_len' for 'write_psr_msr'.
    - check 'socket_info' and 'psr_cos_ids' in this patch.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - fix commit message words.
      (suggested by Jan Beulich)
    - change 'assemble_val_array' to 'gather_val_array'.
      (suggested by Jan Beulich)
    - change 'set_new_val_to_array' to 'insert_new_val_to_array'.
      (suggested by Jan Beulich)
    - change parameter 'm' of 'insert_new_val_to_array' to 'new_val'.
      (suggested by Jan Beulich)
    - change 'write_psr_msr' to 'write_psr_msrs'.
      (suggested by Jan Beulich)
    - correct comments.
      (suggested by Jan Beulich)
    - remove unnecessary comments.
      (suggested by Jan Beulich)
    - adjust conditions after 'find_cos' to save a level of indentation.
      (suggested by Jan Beulich)
    - add 'ASSERT(!old_cos || ref[old_cos])'.
      (suggested by Jan Beulich)
    - move ASSERT() check into locked region.
      (suggested by Jan Beulich)
    - replace parameter '*val' to 'val[]' in some functions.
      (suggested by Jan Beulich)
    - change 'write_psr_msr' parameters to prepare to only set one new value
      for one feature.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
    - add explanation about context switch.
      (suggested by Jan Beulich)
v5:
    - modify commit message.
      (suggested by Jan Beulich)
    - return an error for all helper functions in set flow.
      (suggested by Jan Beulich)
    - remove unnecessary cast.
      (suggested by Jan Beulich)
    - divide 'get_old_set_new' to two functions, 'assemble_val_array' and
      'set_new_val_to_array'.
      (suggested by Jan Beulich)
    - modify comments.
      (suggested by Jan Beulich)
    - adjust code format.
      (suggested by Jan Beulich)
    - change 'alloc_new_cos' to 'pick_avail_cos' to make name accurate.
      (suggested by Jan Beulich)
    - check feature type when entering 'psr_set_val'.
      (suggested by Jan Beulich)
    - use ASSERT to check ref.
      (suggested by Jan Beulich)
    - rename 'dat[]' to 'data[]'.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/domctl.c     |  18 ++---
 xen/arch/x86/psr.c        | 192 +++++++++++++++++++++++++++++++++++++++++++++-
 xen/include/asm-x86/psr.h |   4 +-
 3 files changed, 201 insertions(+), 13 deletions(-)

diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index e714d1d..76923c8 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1437,21 +1437,21 @@ long arch_do_domctl(
         switch ( domctl->u.psr_cat_op.cmd )
         {
         case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
-            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3);
+            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+                              (uint32_t)domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3);
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE:
-            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3_CODE);
+            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+                              (uint32_t)domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3_CODE);
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA:
-            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 domctl->u.psr_cat_op.data,
-                                 PSR_CBM_TYPE_L3_DATA);
+            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+                              (uint32_t)domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L3_DATA);
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 53105f5..f8d4be4 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -578,15 +578,203 @@ int psr_get_val(struct domain *d, unsigned int socket,
     return 0;
 }
 
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
-                   uint64_t cbm, enum cbm_type type)
+/* Set value functions */
+static unsigned int get_cos_num(const struct psr_socket_info *info)
 {
     return 0;
 }
 
+static int gather_val_array(uint32_t val[],
+                            uint32_t array_len,
+                            const struct psr_socket_info *info,
+                            unsigned int old_cos)
+{
+    return -EINVAL;
+}
+
+static int insert_new_val_to_array(uint32_t val[],
+                                   uint32_t array_len,
+                                   const struct psr_socket_info *info,
+                                   enum psr_feat_type feat_type,
+                                   enum cbm_type type,
+                                   uint32_t new_val)
+{
+    return -EINVAL;
+}
+
+static int find_cos(const uint32_t val[], uint32_t array_len,
+                    enum psr_feat_type feat_type,
+                    const struct psr_socket_info *info)
+{
+    ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
+    return -ENOENT;
+}
+
+static int pick_avail_cos(const struct psr_socket_info *info,
+                          const uint32_t val[], uint32_t array_len,
+                          unsigned int old_cos,
+                          enum psr_feat_type feat_type)
+{
+    ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
+    return -ENOENT;
+}
+
+static int write_psr_msr(unsigned int socket, unsigned int cos,
+                         uint32_t val, enum cbm_type type,
+                         enum psr_feat_type feat_type)
+{
+    return -ENOENT;
+}
+
+int psr_set_val(struct domain *d, unsigned int socket,
+                uint32_t val, enum cbm_type type)
+{
+    unsigned int old_cos;
+    int cos, ret;
+    unsigned int *ref;
+    uint32_t *val_array;
+    struct psr_socket_info *info = get_socket_info(socket);
+    uint32_t array_len;
+    enum psr_feat_type feat_type;
+
+    if ( IS_ERR(info) )
+        return PTR_ERR(info);
+
+    feat_type = psr_cbm_type_to_feat_type(type);
+    if ( !test_bit(feat_type, &info->feat_mask) )
+        return -ENOENT;
+
+    /*
+     * Step 0:
+     * old_cos means the COS ID current domain is using. By default, it is 0.
+     *
+     * For every COS ID, there is a reference count to record how many domains
+     * are using the COS register corresponding to this COS ID.
+     * - If ref[old_cos] is 0, that means this COS is not used by any domain.
+     * - If ref[old_cos] is 1, that means this COS is only used by current
+     *   domain.
+     * - If ref[old_cos] is more than 1, that mean multiple domains are using
+     *   this COS.
+     */
+    old_cos = d->arch.psr_cos_ids[socket];
+    ASSERT(old_cos < MAX_COS_REG_CNT && old_cos >= 0);
+
+    ref = info->cos_ref;
+
+    /*
+     * Step 1:
+     * Gather a value array to store all features cos_reg_val[old_cos].
+     * And, set the input new val into array according to the feature's
+     * position in array.
+     */
+    array_len = get_cos_num(info);
+    val_array = xzalloc_array(uint32_t, array_len);
+    if ( !val_array )
+        return -ENOMEM;
+
+    if ( (ret = gather_val_array(val_array, array_len, info, old_cos)) != 0 )
+        goto free_array;
+
+    if ( (ret = insert_new_val_to_array(val_array, array_len, info,
+                                        feat_type, type, val)) != 0 )
+        goto free_array;
+
+    spin_lock(&info->ref_lock);
+
+    /*
+     * Step 2:
+     * Try to find if there is already a COS ID on which all features' values
+     * are same as the array. Then, we can reuse this COS ID.
+     */
+    cos = find_cos(val_array, array_len, feat_type, info);
+    if ( cos == old_cos )
+    {
+        ret = 0;
+        goto unlock_free_array;
+    }
+    else if ( cos >= 0 )
+        goto cos_found;
+
+    /*
+     * Step 3:
+     * If fail to find, we need pick an available COS ID.
+     * In fact, only COS ID which ref is 1 or 0 can be picked for current
+     * domain. If old_cos is not 0 and its ref==1, that means only current
+     * domain is using this old_cos ID. So, this old_cos ID certainly can
+     * be reused by current domain. Ref==0 means there is no any domain
+     * using this COS ID. So it can be used for current domain too.
+     */
+    cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
+    if ( cos < 0 )
+    {
+        ret = cos;
+        goto unlock_free_array;
+    }
+
+    /*
+     * Step 4:
+     * Write all features MSRs according to the COS ID.
+     */
+    ret = write_psr_msr(socket, cos, val, type, feat_type);
+    if ( ret )
+        goto unlock_free_array;
+
+cos_found:
+    /*
+     * Step 5:
+     * Update ref according to COS ID.
+     */
+    ref[cos]++;
+    ASSERT(!cos || ref[cos]);
+    ASSERT(!old_cos || ref[old_cos]);
+    ref[old_cos]--;
+    spin_unlock(&info->ref_lock);
+
+    /*
+     * Step 6:
+     * Save the COS ID into current domain's psr_cos_ids[] so that we can know
+     * which COS the domain is using on the socket. One domain can only use
+     * one COS ID at same time on each socket.
+     */
+    d->arch.psr_cos_ids[socket] = cos;
+    goto free_array;
+
+unlock_free_array:
+    spin_unlock(&info->ref_lock);
+free_array:
+    xfree(val_array);
+    return ret;
+}
+
 /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
 static void psr_free_cos(struct domain *d)
 {
+    unsigned int socket, cos;
+
+    if ( !socket_info || !d->arch.psr_cos_ids )
+        return;
+
+    /* Domain is destroied so its cos_ref should be decreased. */
+    for ( socket = 0; socket < nr_sockets; socket++ )
+    {
+        struct psr_socket_info *info;
+
+        /* cos 0 is default one which does not need be handled. */
+        cos = d->arch.psr_cos_ids[socket];
+        if ( cos == 0 )
+            continue;
+
+        /*
+         * If domain uses other cos ids, all corresponding refs must have been
+         * increased 1 for this domain. So, we need decrease them.
+         */
+        info = socket_info + socket;
+        spin_lock(&info->ref_lock);
+        ASSERT(!cos || info->cos_ref[cos]);
+        info->cos_ref[cos]--;
+        spin_unlock(&info->ref_lock);
+    }
+
     xfree(d->arch.psr_cos_ids);
     d->arch.psr_cos_ids = NULL;
 }
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index cd00733..d2262d9 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -74,8 +74,8 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
                  uint32_t data[], unsigned int array_len);
 int psr_get_val(struct domain *d, unsigned int socket,
                 uint32_t *val, enum cbm_type type);
-int psr_set_l3_cbm(struct domain *d, unsigned int socket,
-                   uint64_t cbm, enum cbm_type type);
+int psr_set_val(struct domain *d, unsigned int socket,
+                uint32_t val, enum cbm_type type);
 
 int psr_domain_init(struct domain *d);
 void psr_domain_free(struct domain *d);
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (8 preceding siblings ...)
  2017-03-16 11:07 ` [PATCH v9 09/25] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 10:17   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 11/25] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
                   ` (15 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

Only can one COS ID be used by one domain at one time. That means all enabled
features' COS registers at this COS ID are valid for this domain at that time.

When user updates a feature's value, we need make sure all other features'
values are not affected. So, we firstly need gather an array which contains
all features current values and replace the setting feature's value in array
to new value.

Then, we can try to find if there is a COS ID on which all features' COS
registers values are same as the array. If we can find, we just use this COS
ID. If fail to find, we need pick a new COS ID.

This patch implements value array assembling flow.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - add comments about boundary checking.
      (suggested by Wei Liu)
    - rename 'assemble_val_array' to 'combine_val_array' in pervious patch.
      (suggested by Wei Liu)
    - rename 'l3_cat_get_cos_num' to 'cat_get_cos_num' to cover all L3/L2 CAT
      features.
      (suggested by Roger Pau)
    - rename 'l3_cat_get_old_val' to 'cat_get_old_val' to cover all L3/L2 CAT
      features and reuse cat_get_val in it.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - replace 'm' to 'new_val'.
      (suggested by Jan Beulich)
    - move cos check outside callback function.
      (suggested by Jan Beulich)
    - remove 'get_cos_num' callback function.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v6:
    - change 'assemble_val_array' to 'combine_val_array'.
      (suggested by Konrad Rzeszutek Wilk)
    - check return value of 'get_old_val'.
      (suggested by Konrad Rzeszutek Wilk)
    - replace some 'EINVAL' to 'ENOSPC'.
      (suggested by Konrad Rzeszutek Wilk)
v5:
    - modify comments according to changes of codes.
      (suggested by Jan Beulich)
    - change 'bool_t' to 'bool'.
      (suggested by Jan Beulich)
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
      (suggested by Jan Beulich)
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 149 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index f8d4be4..18aad8e 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -101,6 +101,28 @@ struct feat_node {
         /* get_val is used to get feature COS register value. */
         void (*get_val)(const struct feat_node *feat, unsigned int cos,
                         enum cbm_type type, uint32_t *val);
+
+        /*
+         * get_old_val and set_new_val are a pair of functions called in order.
+         * The caller will traverse all features in the array and call
+         * 'get_old_val' to get old_cos register value of all supported
+         * features. Then, call 'set_new_val' to set the new value for the
+         * designated feature.
+         *
+         * All the values are set into value array according to the traversal
+         * order, meaning the same order of feature array members.
+         *
+         * The return value meaning of set_new_val:
+         * 0 - success.
+         * negative - error.
+         */
+        void (*get_old_val)(uint32_t val[],
+                            const struct feat_node *feat,
+                            unsigned int old_cos);
+        int (*set_new_val)(uint32_t val[],
+                           const struct feat_node *feat,
+                           enum cbm_type type,
+                           uint32_t new_val);
     } ops;
 
     /* Encapsulate feature specific HW info here. */
@@ -212,6 +234,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
 }
 
 /* CAT common functions implementation. */
+static bool psr_check_cbm(unsigned int cbm_len, uint32_t cbm)
+{
+    unsigned int first_bit, zero_bit;
+
+    /* Set bits should only in the range of [0, cbm_len]. */
+    if ( cbm & (~0ul << cbm_len) )
+        return false;
+
+    /* At least one bit need to be set. */
+    if ( cbm == 0 )
+        return false;
+
+    first_bit = find_first_bit((uint64_t *)&cbm, cbm_len);
+    zero_bit = find_next_zero_bit((uint64_t *)&cbm, cbm_len, first_bit);
+
+    /* Set bits should be contiguous. */
+    if ( zero_bit < cbm_len &&
+         find_next_bit((uint64_t *)&cbm, cbm_len, zero_bit) < cbm_len )
+        return false;
+
+    return true;
+}
+
 static void cat_init_feature(struct cpuid_leaf regs,
                              struct feat_node *feat,
                              struct psr_socket_info *info,
@@ -285,11 +330,35 @@ static void cat_get_val(const struct feat_node *feat, unsigned int cos,
     *val = feat->cos_reg_val[cos];
 }
 
+/* val[] len checking is done by caller. */
+static void cat_get_old_val(uint32_t val[],
+                            const struct feat_node *feat,
+                            unsigned int old_cos)
+{
+    cat_get_val(feat, old_cos, 0, &val[0]);
+}
+
+/* val[] len checking is done by caller. */
+static int cat_set_new_val(uint32_t val[],
+                           const struct feat_node *feat,
+                           enum cbm_type type,
+                           uint32_t new_val)
+{
+    if ( !psr_check_cbm(feat->info.cat_info.cbm_len, new_val) )
+        return -EINVAL;
+
+    val[0] = new_val;
+
+    return 0;
+}
+
 /* L3 CAT ops */
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = cat_get_cos_max,
     .get_feat_info = cat_get_feat_info,
     .get_val = cat_get_val,
+    .get_old_val = cat_get_old_val,
+    .set_new_val = cat_set_new_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -581,7 +650,21 @@ int psr_get_val(struct domain *d, unsigned int socket,
 /* Set value functions */
 static unsigned int get_cos_num(const struct psr_socket_info *info)
 {
-    return 0;
+    const struct feat_node *feat;
+    unsigned int num = 0, i;
+
+    /* Get all features total amount. */
+    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
+    {
+        if ( !info->features[i] )
+            continue;
+
+        feat = info->features[i];
+
+        num += feat->cos_num;
+    }
+
+    return num;
 }
 
 static int gather_val_array(uint32_t val[],
@@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
                             const struct psr_socket_info *info,
                             unsigned int old_cos)
 {
-    return -EINVAL;
+    const struct feat_node *feat;
+    unsigned int i;
+
+    if ( !val )
+        return -EINVAL;
+
+    /* Get all features current values according to old_cos. */
+    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
+    {
+        if ( !info->features[i] )
+            continue;
+
+        feat = info->features[i];
+
+        if ( old_cos > feat->ops.get_cos_max(feat) )
+            old_cos = 0;
+
+        /* value getting order is same as feature array */
+        feat->ops.get_old_val(val, feat, old_cos);
+
+        array_len -= feat->cos_num;
+        if ( array_len < 0 )
+            return -ENOSPC;
+
+        val += feat->cos_num;
+    }
+
+    return 0;
 }
 
 static int insert_new_val_to_array(uint32_t val[],
@@ -599,7 +709,43 @@ static int insert_new_val_to_array(uint32_t val[],
                                    enum cbm_type type,
                                    uint32_t new_val)
 {
-    return -EINVAL;
+    const struct feat_node *feat;
+    int ret;
+    unsigned int i;
+
+    ASSERT(feat_type < PSR_SOCKET_MAX_FEAT);
+
+    /* Set new value into array according to feature's position in array. */
+    for ( i = 0; i < feat_type; i++ )
+    {
+        if ( !info->features[i] )
+            continue;
+
+        feat = info->features[i];
+
+        array_len -= feat->cos_num;
+        if ( array_len <= 0 )
+            return -ENOSPC;
+
+        val += feat->cos_num;
+    }
+
+    feat = info->features[feat_type];
+
+    array_len -= feat->cos_num;
+    if ( array_len < 0 )
+        return -ENOSPC;
+
+    /*
+     * Value setting position is same as feature array.
+     * Different features may have different setting behaviors, e.g. CDP
+     * has two values (DATA/CODE) which need us to save input value to
+     * different position in the array according to type, so we have to
+     * maintain a callback function.
+     */
+    ret = feat->ops.set_new_val(val, feat, type, new_val);
+
+    return ret;
 }
 
 static int find_cos(const uint32_t val[], uint32_t array_len,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 11/25] x86: refactor psr: L3 CAT: set value: implement cos finding flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (9 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 10:28   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
                   ` (14 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

Continue from patch:
'x86: refactor psr: L3 CAT: set value: assemble features value array'

We can try to find if there is a COS ID on which all features' COS registers
values are same as the array assembled before.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - modify comments of 'compare_val' to be same as current implementation.
      (suggested by Wei Liu)
    - fix indentation issue.
      (suggested by Wei Liu)
    - rename 'l3_cat_compare_val' to 'cat_compare_val' to cover all L3/L2 CAT
      features.
      (suggested by Roger Pau)
    - remove parameter 'found' from 'cat_compare_val' and modify the return
      values to let caller know if the id is found or not.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - replace 'get_cos_num' to 'feat->cos_num'.
      (suggested by Jan Beulich)
    - directly use 'cos_reg_val[0]' as default value.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v5:
    - modify commit message to provide exact patch name to continue from.
      (suggested by Jan Beulich)
    - remove 'get_cos_max_from_type' because it can be replaced by
      'get_cos_max'.
    - move type check out from callback functions to caller.
      (suggested by Jan Beulich)
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
      (suggested by Jan Beulich)
    - modify comments according to changes of codes.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 18aad8e..f2c2614 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -123,6 +123,19 @@ struct feat_node {
                            const struct feat_node *feat,
                            enum cbm_type type,
                            uint32_t new_val);
+
+        /*
+         * compare_val is used in set value process to compare if the
+         * input value array can match all the features' COS registers values
+         * according to input cos id.
+         *
+         * The return value is:
+         * 1 - find the entry in value array.
+         * 0 - not find the entry in value array.
+         * negative - error.
+         */
+        int (*compare_val)(const uint32_t val[], const struct feat_node *feat,
+                           unsigned int cos);
     } ops;
 
     /* Encapsulate feature specific HW info here. */
@@ -352,6 +365,33 @@ static int cat_set_new_val(uint32_t val[],
     return 0;
 }
 
+static int cat_compare_val(const uint32_t val[],
+                           const struct feat_node *feat,
+                           unsigned int cos)
+{
+    /*
+     * Different features' cos_max are different. If cos id of the feature
+     * being set exceeds other feature's cos_max, the val of other feature
+     * must be default value. HW supports such case.
+     */
+    if ( cos > feat->info.cat_info.cos_max )
+    {
+        /* cos_reg_val[0] is the default value. */
+        if ( val[0] != feat->cos_reg_val[0] )
+            return -EINVAL;
+
+        /* Find */
+        return 1;
+    }
+
+    if ( val[0] == feat->cos_reg_val[cos] )
+        /* Find */
+        return 1;
+
+    /* Not find */
+    return 0;
+}
+
 /* L3 CAT ops */
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = cat_get_cos_max,
@@ -359,6 +399,7 @@ static const struct feat_ops l3_cat_ops = {
     .get_val = cat_get_val,
     .get_old_val = cat_get_old_val,
     .set_new_val = cat_set_new_val,
+    .compare_val = cat_compare_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -752,7 +793,61 @@ static int find_cos(const uint32_t val[], uint32_t array_len,
                     enum psr_feat_type feat_type,
                     const struct psr_socket_info *info)
 {
+    unsigned int cos, i;
+    const unsigned int *ref = info->cos_ref;
+    const struct feat_node *feat;
+    const uint32_t *val_array = val;
+    int find = 0;
+    unsigned int cos_max;
+
     ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
+
+    /* cos_max is the one of the feature which is being set. */
+    feat = info->features[feat_type];
+    if ( !feat )
+        return -ENOENT;
+
+    cos_max = feat->ops.get_cos_max(feat);
+
+    for ( cos = 0; cos <= cos_max; cos++ )
+    {
+        if ( cos && !ref[cos] )
+            continue;
+
+        /*
+         * If fail to find cos in below loop, need find whole feature array
+         * again from beginning.
+         */
+        val_array = val;
+        for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
+        {
+            if ( !info->features[i] )
+                continue;
+
+            feat = info->features[i];
+            /*
+             * Compare value according to feature array order.
+             * We must follow this order because value array is assembled
+             * as this order.
+             */
+            find = feat->ops.compare_val(val_array, feat, cos);
+            if ( find < 0 )
+                return find;
+
+            /* If fail to match, go to next cos to compare. */
+            if ( !find )
+                break;
+
+            val_array += feat->cos_num;
+            if ( val_array - val > array_len )
+                return -ENOSPC;
+        }
+
+        /* For this COS ID all entries in the values array did match. Use it. */
+        if ( find )
+            return cos;
+    }
+
     return -ENOENT;
 }
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (10 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 11/25] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 10:37   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 13/25] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
                   ` (13 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

Continue from previous patch:
'x86: refactor psr: L3 CAT: set value: implement cos finding flow.'

If fail to find a COS ID, we need pick a new COS ID for domain. Only COS ID
that ref[COS_ID] is 1 or 0 can be picked to input a new set feature values.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - modify return value of 'pick_avail_cos' to make it more accurate.
    - rename 'l3_cat_fits_cos_max' to 'cat_fits_cos_max' to cover L3/L2 CAT
      features.
      (suggested by Roger Pau)
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - fix comment.
      (suggested by Wei Liu)
    - directly use 'cos_reg_val[0]' as default value.
      (suggested by Jan Beulich)
    - replace 'get_cos_num' to 'feat->cos_num'.
      (suggested by Jan Beulich)
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v5:
    - modify commit message to provide exact patch name to continue from.
      (suggested by Jan Beulich)
    - change 'exceeds_cos_max' to 'fits_cos_max' to be accurate.
      (suggested by Jan Beulich)
    - modify comments according to changes of codes.
      (suggested by Jan Beulich)
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
      (suggested by Jan Beulich)
    - move type check out from callback functions to caller.
      (suggested by Jan Beulich)
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
      (suggested by Jan Beulich)
    - modify code format.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 98 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index f2c2614..d4db407 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -136,6 +136,19 @@ struct feat_node {
          */
         int (*compare_val)(const uint32_t val[], const struct feat_node *feat,
                            unsigned int cos);
+
+        /*
+         * fits_cos_max is used to check if the input cos id exceeds the
+         * feature's cos_max and if the input value is not the default one.
+         * Even if the associated cos exceeds the cos_max, HW can work with
+         * default value. That is the reason we need check if input value is
+         * default one. If both criteria are fulfilled, that means the input
+         * exceeds the range. If not, that means the input fits the
+         * requirements.
+         */
+        bool (*fits_cos_max)(const uint32_t val[],
+                             const struct feat_node *feat,
+                             unsigned int cos);
     } ops;
 
     /* Encapsulate feature specific HW info here. */
@@ -392,6 +405,21 @@ static int cat_compare_val(const uint32_t val[],
     return 0;
 }
 
+static bool cat_fits_cos_max(const uint32_t val[],
+                             const struct feat_node *feat,
+                             unsigned int cos)
+{
+    if ( cos > feat->info.cat_info.cos_max &&
+         val[0] != feat->cos_reg_val[0] )
+            /*
+             * Exceed cos_max and value to set is not default,
+             * return error.
+             */
+            return false;
+
+    return true;
+}
+
 /* L3 CAT ops */
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = cat_get_cos_max,
@@ -400,6 +428,7 @@ static const struct feat_ops l3_cat_ops = {
     .get_old_val = cat_get_old_val,
     .set_new_val = cat_set_new_val,
     .compare_val = cat_compare_val,
+    .fits_cos_max = cat_fits_cos_max,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -851,13 +880,81 @@ static int find_cos(const uint32_t val[], uint32_t array_len,
     return -ENOENT;
 }
 
+static bool fits_cos_max(const uint32_t val[],
+                         uint32_t array_len,
+                         const struct psr_socket_info *info,
+                         unsigned int cos)
+{
+    unsigned int ret, i;
+    const struct feat_node *feat;
+
+    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
+    {
+        if ( !info->features[i] )
+            continue;
+
+        feat = info->features[i];
+
+        ret = feat->ops.fits_cos_max(val, feat, cos);
+        if ( !ret )
+            return false;
+
+        array_len -= feat->cos_num;
+        if ( array_len < 0 )
+            return false;
+
+        val += feat->cos_num;
+    }
+
+    return true;
+}
+
 static int pick_avail_cos(const struct psr_socket_info *info,
                           const uint32_t val[], uint32_t array_len,
                           unsigned int old_cos,
                           enum psr_feat_type feat_type)
 {
+    unsigned int cos;
+    unsigned int cos_max = 0;
+    const struct feat_node *feat;
+    const unsigned int *ref = info->cos_ref;
+
     ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
-    return -ENOENT;
+
+    /* cos_max is the one of the feature which is being set. */
+    feat = info->features[feat_type];
+    if ( !feat )
+        return -ENOENT;
+
+    cos_max = feat->ops.get_cos_max(feat);
+    if ( !cos_max )
+        return -ENOENT;
+
+    /*
+     * If old cos is referred only by the domain, then use it. And, we cannot
+     * use id 0 because it stores the default values.
+     */
+    if ( old_cos && ref[old_cos] == 1 &&
+         fits_cos_max(val, array_len, info, old_cos) )
+            return old_cos;
+
+    /* Find an unused one other than cos0. */
+    for ( cos = 1; cos <= cos_max; cos++ )
+    {
+        /*
+         * ref is 0 means this COS is not used by other domain and
+         * can be used for current setting.
+         */
+        if ( !ref[cos] )
+        {
+            if ( !fits_cos_max(val, array_len, info, cos) )
+                return -EOVERFLOW;
+
+            return cos;
+        }
+    }
+
+    return -EOVERFLOW;
 }
 
 static int write_psr_msr(unsigned int socket, unsigned int cos,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 13/25] x86: refactor psr: L3 CAT: set value: implement write msr flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (11 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 10:46   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 14/25] x86: refactor psr: CDP: implement CPU init and free flow Yi Sun
                   ` (12 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

Continue from previous patch:
'x86: refactor psr: L3 CAT: set value: implement cos id picking flow.'

We have got the feature value and COS ID to set. Then, we write MSR of the
designated feature.

Till now, set value process is completed.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - replace feature list handling to feature array handling.
      (suggested by Roger Pau)
    - add 'array_len' in 'struct cos_write_info' and check if val array
      exceeds it.
    - modify 'write_psr_msr' flow only to set one value a time. No need to
      set whole feature array values.
    - modify patch title to indicate 'L3 CAT'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v8:
    - modify 'write_msr' callback function to 'void' because we have to set
      all features' cbm. When input cos exceeds some features' cos_max, just
      skip them but not break the iteration.
v5:
    - modify commit message to provide exact patch name to continue from.
      (suggested by Jan Beulich)
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
      (suggested by Jan Beulich)
    - move type check out from callback functions to caller.
      (suggested by Jan Beulich)
    - modify variables names to make them better, e.g. 'feat_tmp' to 'feat'.
      (suggested by Jan Beulich)
    - correct code format.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 69 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index d4db407..2bc7f3c 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -149,6 +149,10 @@ struct feat_node {
         bool (*fits_cos_max)(const uint32_t val[],
                              const struct feat_node *feat,
                              unsigned int cos);
+
+        /* write_msr is used to write out feature MSR register. */
+        void (*write_msr)(unsigned int cos, uint32_t val,
+                          enum cbm_type type, struct feat_node *feat);
     } ops;
 
     /* Encapsulate feature specific HW info here. */
@@ -421,6 +425,18 @@ static bool cat_fits_cos_max(const uint32_t val[],
 }
 
 /* L3 CAT ops */
+static void l3_cat_write_msr(unsigned int cos, uint32_t val,
+                             enum cbm_type type, struct feat_node *feat)
+{
+    if ( feat->cos_reg_val[cos] != val )
+    {
+        feat->cos_reg_val[cos] = val;
+        wrmsrl(MSR_IA32_PSR_L3_MASK(cos), (uint64_t)val);
+    }
+
+    return;
+}
+
 static const struct feat_ops l3_cat_ops = {
     .get_cos_max = cat_get_cos_max,
     .get_feat_info = cat_get_feat_info,
@@ -429,6 +445,7 @@ static const struct feat_ops l3_cat_ops = {
     .set_new_val = cat_set_new_val,
     .compare_val = cat_compare_val,
     .fits_cos_max = cat_fits_cos_max,
+    .write_msr = l3_cat_write_msr,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
@@ -957,11 +974,62 @@ static int pick_avail_cos(const struct psr_socket_info *info,
     return -EOVERFLOW;
 }
 
+static unsigned int get_socket_cpu(unsigned int socket)
+{
+    if ( likely(socket < nr_sockets) )
+        return cpumask_any(socket_cpumask[socket]);
+
+    return nr_cpu_ids;
+}
+
+struct cos_write_info
+{
+    unsigned int cos;
+    struct feat_node *feature;
+    uint32_t val;
+    enum cbm_type type;
+};
+
+static void do_write_psr_msr(void *data)
+{
+    struct cos_write_info *info = (struct cos_write_info *)data;
+    unsigned int cos            = info->cos;
+    struct feat_node *feat      = info->feature;
+
+    if ( !feat )
+        return;
+
+    if ( cos > feat->ops.get_cos_max(feat) )
+        return;
+
+    feat->ops.write_msr(cos, info->val, info->type, feat);
+}
+
 static int write_psr_msr(unsigned int socket, unsigned int cos,
                          uint32_t val, enum cbm_type type,
                          enum psr_feat_type feat_type)
 {
-    return -ENOENT;
+    struct psr_socket_info *info = get_socket_info(socket);
+    struct cos_write_info data =
+    {
+        .cos = cos,
+        .feature = info->features[feat_type],
+        .val = val,
+        .type = type,
+    };
+
+    if ( socket == cpu_to_socket(smp_processor_id()) )
+        do_write_psr_msr(&data);
+    else
+    {
+        unsigned int cpu = get_socket_cpu(socket);
+
+        if ( cpu >= nr_cpu_ids )
+            return -ENOTSOCK;
+        on_selected_cpus(cpumask_of(cpu), do_write_psr_msr, &data, 1);
+    }
+
+    return 0;
 }
 
 int psr_set_val(struct domain *d, unsigned int socket,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 14/25] x86: refactor psr: CDP: implement CPU init and free flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (12 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 13/25] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 13:58   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 15/25] x86: refactor psr: CDP: implement get hw info flow Yi Sun
                   ` (11 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements the CPU init and free flow for CDP including L3 CDP
initialization callback function. The flow is almost same as L3 CAT.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - modify commit message to describe flow clearer.
    - handle cpu offline and online again case to read MSRs registers values
      back and save them into cos array to make user can get real data.
    - modify error handling process in 'psr_cpu_prepare' to reduce redundant
      codes.
    - modify 'get_cdp_data' and 'get_cdp_code' to make them standard.
      (suggested by Roger Pau and Jan Beulich)
    - encapsulate CDP operations into 'cat_init_feature' to reduce redundant
      codes.
      (suggested by Roger Pau)
    - reuse 'cat_get_cos_max' for CDP.
      (suggested by Roger Pau)
    - handle 'PSR_CDP' in psr_presmp_init to make init work can be done when
      there is only 'psr=cdp' in cmdline.
    - remove unnecessary comment.
      (suggested by Jan Beulich)
    - move CDP related codes in 'cpu_init_work' into 'psr_cpu_init'.
      (suggested by Jan Beulich)
    - add codes to handle CDP's 'cos_num'.
      (suggested by Jan Beulich)
    - fix coding style issue.
      (suggested by Jan Beulich)
    - do not free resources when allocation fails in 'psr_cpu_prepare'.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v7:
    - initialize 'l3_cdp'.
      (suggested by Konrad Rzeszutek Wilk)
v6:
    - use 'cpuid_leaf'.
      (suggested by Konrad Rzeszutek Wilk and Jan Beulich)
v5:
    - remove codes to free 'feat_l3_cdp' in 'free_feature'.
      (suggested by Jan Beulich)
    - encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
      (suggested by Jan Beulich)
    - print socket info when 'opt_cpu_info' is true.
      (suggested by Jan Beulich)
    - rename 'l3_cdp_get_max_cos_max' to 'l3_cdp_get_cos_max'.
      (suggested by Jan Beulich)
    - rename 'dat[]' to 'data[]'.
      (suggested by Jan Beulich)
    - move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 112 ++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 94 insertions(+), 18 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 2bc7f3c..e2a2643 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -55,6 +55,9 @@
 /* CAT features use 1 COS register in one access. */
 #define CAT_COS_NUM      1
 
+/* CDP uses 2 COS registers in one access. */
+#define CDP_COS_NUM      2
+
 enum psr_feat_type {
     PSR_SOCKET_L3_CAT = 0,
     PSR_SOCKET_L3_CDP,
@@ -209,12 +212,29 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
  * array creation. It is used to transiently store a spare node.
  */
 static struct feat_node *feat_l3_cat;
+static struct feat_node *feat_l3_cdp;
 
 /* Common functions */
 #define cat_default_val(len)                 \
             ( (uint32_t)((1ul << len) - 1) )
 
 /*
+ * get_data - get DATA COS register value from input COS ID.
+ * @feat:        the feature node.
+ * @cos:         the COS ID.
+ */
+#define get_cdp_data(feat, cos)              \
+            ( (feat)->cos_reg_val[(cos) * 2] )
+
+/*
+ * get_cdp_code - get CODE COS register value from input COS ID.
+ * @feat:        the feature node.
+ * @cos:         the COS ID.
+ */
+#define get_cdp_code(feat, cos)              \
+            ( (feat)->cos_reg_val[(cos) * 2 + 1] )
+
+/*
  * Use this function to check if any allocation feature has been enabled
  * in cmdline.
  */
@@ -303,20 +323,56 @@ static void cat_init_feature(struct cpuid_leaf regs,
     cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
     cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
 
-    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
-    feat->cos_reg_val[0] = cat_default_val(cat.cbm_len);
-    /*
-     * To handle cpu offline and then online case, we need read MSRs back to
-     * save values into cos_reg_val array.
-     */
-    for ( i = 1; i <= cat.cos_max; i++ )
+    if ( type == PSR_SOCKET_L3_CDP )
+    {
+        uint64_t val_code;
+
+        /* Cut half of cos_max when CDP is enabled. */
+        cat.cos_max >>= 1;
+
+        /* We only write mask1 since mask0 is always all ones by default. */
+        wrmsrl(MSR_IA32_PSR_L3_MASK(1), cat_default_val(cat.cbm_len));
+        rdmsrl(MSR_IA32_PSR_L3_QOS_CFG, val);
+        wrmsrl(MSR_IA32_PSR_L3_QOS_CFG, val | (1 << PSR_L3_QOS_CDP_ENABLE_BIT));
+
+        /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+        get_cdp_code(feat, 0) = cat_default_val(cat.cbm_len);
+        get_cdp_data(feat, 0) = cat_default_val(cat.cbm_len);
+
+        feat->cos_num = CDP_COS_NUM;
+
+        /*
+         * To handle cpu offline and then online case, we need read MSRs back to
+         * save values into cos_reg_val array. All cpus MSRs values on same socket
+         * are same.
+         */
+        for ( i = 1; i <= cat.cos_max; i++ )
+        {
+            rdmsrl(MSR_IA32_PSR_L3_MASK_DATA(i), val);
+            rdmsrl(MSR_IA32_PSR_L3_MASK_CODE(i), val_code);
+            get_cdp_code(feat, i) = (uint32_t)val_code;
+            get_cdp_data(feat, i) = (uint32_t)val;
+        }
+    }
+    else
     {
-        rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
-        feat->cos_reg_val[i] = (uint32_t)val;
+        /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
+        feat->cos_reg_val[0] = cat_default_val(cat.cbm_len);
+
+        feat->cos_num = CAT_COS_NUM;
+
+        /*
+         * To handle cpu offline and then online case, we need read MSRs back to
+         * save values into cos_reg_val array.
+         */
+        for ( i = 1; i <= cat.cos_max; i++ )
+        {
+            rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
+            feat->cos_reg_val[i] = (uint32_t)val;
+        }
     }
 
     feat->info.cat_info = cat;
-    feat->cos_num = CAT_COS_NUM;
 
     /* Add this feature into array. */
     info->features[type] = feat;
@@ -328,8 +384,9 @@ static void cat_init_feature(struct cpuid_leaf regs,
     if ( !opt_cpu_info )
         return;
 
-    printk(XENLOG_INFO "%s CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
-           ((type == PSR_SOCKET_L3_CAT) ? "L3" : "L2"),
+    printk(XENLOG_INFO "%s: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
+           ((type == PSR_SOCKET_L3_CDP) ? "CDP" :
+            ((type == PSR_SOCKET_L3_CAT) ? "L3 CAT": "L2 CAT")),
            socket, feat->info.cat_info.cos_max,
            feat->info.cat_info.cbm_len);
 
@@ -448,6 +505,11 @@ static const struct feat_ops l3_cat_ops = {
     .write_msr = l3_cat_write_msr,
 };
 
+/* L3 CDP ops */
+struct feat_ops l3_cdp_ops = {
+    .get_cos_max = cat_get_cos_max,
+};
+
 static void __init parse_psr_bool(char *s, char *value, char *feature,
                                   unsigned int mask)
 {
@@ -1244,6 +1306,10 @@ static int psr_cpu_prepare(void)
          (feat_l3_cat = xzalloc(struct feat_node)) == NULL )
         return -ENOMEM;
 
+    if ( feat_l3_cdp == NULL &&
+         (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
+        return -ENOMEM;
+
     return 0;
 }
 
@@ -1279,11 +1345,21 @@ static void psr_cpu_init(void)
     {
         cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, &regs);
 
-        feat = feat_l3_cat;
-        feat_l3_cat = NULL;
-        feat->ops = l3_cat_ops;
-
-        cat_init_feature(regs, feat, info, PSR_SOCKET_L3_CAT);
+        if ( (regs.c & PSR_CAT_CDP_CAPABILITY) && (opt_psr & PSR_CDP) &&
+             !test_bit(PSR_SOCKET_L3_CDP, &info->feat_mask) )
+        {
+            feat = feat_l3_cdp;
+            feat_l3_cdp = NULL;
+            feat->ops = l3_cdp_ops;
+            cat_init_feature(regs, feat, info, PSR_SOCKET_L3_CDP);
+        }
+        else
+        {
+            feat = feat_l3_cat;
+            feat_l3_cat = NULL;
+            feat->ops = l3_cat_ops;
+            cat_init_feature(regs, feat, info, PSR_SOCKET_L3_CAT);
+        }
     }
 
 assoc_init:
@@ -1343,7 +1419,7 @@ static int __init psr_presmp_init(void)
     if ( (opt_psr & PSR_CMT) && opt_rmid_max )
         init_psr_cmt(opt_rmid_max);
 
-    if ( opt_psr & PSR_CAT )
+    if ( opt_psr & (PSR_CAT | PSR_CDP) )
         init_psr();
 
     if ( psr_cpu_prepare() )
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 15/25] x86: refactor psr: CDP: implement get hw info flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (13 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 14/25] x86: refactor psr: CDP: implement CPU init and free flow Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 14:08   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 16/25] x86: refactor psr: CDP: implement get value flow Yi Sun
                   ` (10 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements get HW info flow for CDP including L3 CDP callback
function. The flow is almost same as L3 CAT.

It also changes sysctl function to make it work for CDP.

With this patch, 'psr-hwinfo' can work for L3 CDP.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - modify commit message to explain flow more clearly.
    - reuse 'cat_get_feat_info' for CDP to reduce redundant codes.
      (suggested by Roger Pau)
    - fix coding style issues.
      (suggested by Wei Liu and Roger Pau)
    - rename macros used by psr_get_info to make them meaningful.
      (suggested by Jan Beulich)
v5:
    - rename 'dat[]' to 'data[]'.
      (suggested by Jan Beulich)
    - remove type check in callback function.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c    | 16 ++++++++++++++++
 xen/arch/x86/sysctl.c | 34 ++++++++++++++++++++++++++++++----
 2 files changed, 46 insertions(+), 4 deletions(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index e2a2643..ba5c7a4 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -275,6 +275,10 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
     case PSR_CBM_TYPE_L3:
         feat_type = PSR_SOCKET_L3_CAT;
         break;
+    case PSR_CBM_TYPE_L3_DATA:
+    case PSR_CBM_TYPE_L3_CODE:
+        feat_type = PSR_SOCKET_L3_CDP;
+        break;
     default:
         feat_type = PSR_SOCKET_UNKNOWN;
         break;
@@ -506,8 +510,20 @@ static const struct feat_ops l3_cat_ops = {
 };
 
 /* L3 CDP ops */
+static bool l3_cdp_get_feat_info(const struct feat_node *feat,
+                                 uint32_t data[], uint32_t array_len)
+{
+    if ( !cat_get_feat_info(feat, data, array_len) )
+        return false;
+
+    data[PSR_INFO_IDX_FLAG] |= XEN_SYSCTL_PSR_CAT_L3_CDP;
+
+    return true;
+}
+
 struct feat_ops l3_cdp_ops = {
     .get_cos_max = cat_get_cos_max,
+    .get_feat_info = l3_cdp_get_feat_info,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index aab3d2d..3bf51a4 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -180,10 +180,36 @@ long arch_do_sysctl(
 
             ret = psr_get_info(sysctl->u.psr_cat_op.target,
                                PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
-
-            sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[PSR_INFO_IDX_CBM_LEN];
-            sysctl->u.psr_cat_op.u.l3_info.cos_max = data[PSR_INFO_IDX_COS_MAX];
-            sysctl->u.psr_cat_op.u.l3_info.flags   = data[PSR_INFO_IDX_FLAG];
+            if ( !ret )
+            {
+                sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+                                               data[PSR_INFO_IDX_CBM_LEN];
+                sysctl->u.psr_cat_op.u.l3_info.cos_max =
+                                               data[PSR_INFO_IDX_COS_MAX];
+                sysctl->u.psr_cat_op.u.l3_info.flags =
+                                               data[PSR_INFO_IDX_FLAG];
+            }
+            else
+            {
+                /*
+                 * Check if CDP is enabled.
+                 *
+                 * Per spec, L3 CAT and CDP cannot co-exist. So, we need replace
+                 * output values to CDP's if it is enabled.
+                 */
+                ret = psr_get_info(sysctl->u.psr_cat_op.target,
+                                   PSR_CBM_TYPE_L3_CODE, data,
+                                   ARRAY_SIZE(data));
+                if ( !ret )
+                {
+                    sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+                                               data[PSR_INFO_IDX_CBM_LEN];
+                    sysctl->u.psr_cat_op.u.l3_info.cos_max =
+                                               data[PSR_INFO_IDX_COS_MAX];
+                    sysctl->u.psr_cat_op.u.l3_info.flags =
+                                               data[PSR_INFO_IDX_FLAG];
+                }
+            }
 
             if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
                 ret = -EFAULT;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 16/25] x86: refactor psr: CDP: implement get value flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (14 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 15/25] x86: refactor psr: CDP: implement get hw info flow Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-16 11:08 ` [PATCH v9 17/25] x86: refactor psr: CDP: implement set value callback functions Yi Sun
                   ` (9 subsequent siblings)
  25 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements L3 CDP get value callback function.

With this patch, 'psr-cat-show' can work for L3 CDP.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - modify the type of 'l3_cdp_get_val' to 'void'.
    - cos checking has been done in common function so remove related codes
      in CDP callback function.
      (suggested by Jan Beulich)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v5:
    - remove type check in callback function.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index ba5c7a4..0ce8a6b 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -521,9 +521,19 @@ static bool l3_cdp_get_feat_info(const struct feat_node *feat,
     return true;
 }
 
+static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
+                           enum cbm_type type, uint32_t *val)
+{
+    if ( type == PSR_CBM_TYPE_L3_DATA )
+        *val = get_cdp_data(feat, cos);
+    else
+        *val = get_cdp_code(feat, cos);
+}
+
 struct feat_ops l3_cdp_ops = {
     .get_cos_max = cat_get_cos_max,
     .get_feat_info = l3_cdp_get_feat_info,
+    .get_val = l3_cdp_get_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 17/25] x86: refactor psr: CDP: implement set value callback functions.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (15 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 16/25] x86: refactor psr: CDP: implement get value flow Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 14:17   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 18/25] x86: L2 CAT: implement CPU init and free flow Yi Sun
                   ` (8 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements L3 CDP set value related callback functions.

With this patch, 'psr-cat-cbm-set' command can work for L3 CDP.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - add comment to explain why CDP uses 2 COSs.
      (suggested by Wei Liu)
    - use 'cat_default_val'.
      (suggested by Wei Liu)
    - remove 'l3_cdp_get_cos_num' because we can directly get cos_num from
      feat_node now.
      (suggested by Jan Beulich)
    - remove cos checking because it has been moved to common function.
      (suggested by Jan Beulich)
    - l3_cdp_set_new_val parameter 'm' is changed to 'new_val'.
      (suggested by Jan Beulich)
    - directly use get_cdp_data(feat, 0) and get_cdp_code(feat, 0) to get
      default value.
      (suggested by Jan Beulich)
    - modify 'l3_cdp_write_msr' flow to write value into register according
      to input type.
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v8:
    - modify 'l3_cdp_write_msr' type to 'void'.
v5:
    - remove type check in callback function.
      (suggested by Jan Beulich)
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
      (suggested by Jan Beulich)
    - remove 'l3_cdp_get_cos_max_from_type'.
    - rename 'l3_cdp_exceeds_cos_max' to 'l3_cdp_fits_cos_max'.
      (suggested by Jan Beulich)
v4:
    - create this patch to make codes easier to understand.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 0ce8a6b..7ede8a1 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -530,10 +530,101 @@ static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
         *val = get_cdp_code(feat, cos);
 }
 
+static void l3_cdp_get_old_val(uint32_t val[],
+                               const struct feat_node *feat,
+                               unsigned int old_cos)
+{
+    /* Data */
+    val[0] = get_cdp_data(feat, old_cos);
+    /* Code */
+    val[1] = get_cdp_code(feat, old_cos);
+}
+
+static int l3_cdp_set_new_val(uint32_t val[],
+                              const struct feat_node *feat,
+                              enum cbm_type type,
+                              uint32_t new_val)
+{
+    if ( !psr_check_cbm(feat->info.cat_info.cbm_len, new_val) )
+        return -EINVAL;
+
+    if ( type == PSR_CBM_TYPE_L3_DATA )
+        val[0] = new_val;
+    else
+        val[1] = new_val;
+
+    return 0;
+}
+
+static int l3_cdp_compare_val(const uint32_t val[],
+                              const struct feat_node *feat,
+                              unsigned int cos)
+{
+    /*
+     * Different features' cos_max are different. If cos id of the feature
+     * being set exceeds other feature's cos_max, the val of other feature
+     * must be default value. HW supports such case.
+    */
+    if ( cos > feat->info.cat_info.cos_max )
+    {
+        if ( val[0] != get_cdp_data(feat, 0) ||
+             val[1] != get_cdp_code(feat, 0) )
+            return -EINVAL;
+
+        /* Find */
+        return 1;
+    }
+
+    if ( val[0] == get_cdp_data(feat, cos) &&
+         val[1] == get_cdp_code(feat, cos) )
+        /* Find */
+        return 1;
+
+    /* Not find */
+    return 0;
+}
+
+static bool l3_cdp_fits_cos_max(const uint32_t val[],
+                                const struct feat_node *feat,
+                                unsigned int cos)
+{
+    if ( cos > feat->info.cat_info.cos_max &&
+         (val[0] != get_cdp_data(feat, 0) || val[1] != get_cdp_code(feat, 0)) )
+            /*
+             * Exceed cos_max and value to set is not default,
+             * return error.
+             */
+            return false;
+
+    return true;
+}
+
+static void l3_cdp_write_msr(unsigned int cos, uint32_t val,
+                             enum cbm_type type, struct feat_node *feat)
+{
+    /* Data */
+    if ( type == PSR_CBM_TYPE_L3_DATA && get_cdp_data(feat, cos) != val )
+    {
+        get_cdp_data(feat, cos) = val;
+        wrmsrl(MSR_IA32_PSR_L3_MASK_DATA(cos), (uint64_t)val);
+    }
+    /* Code */
+    if ( type == PSR_CBM_TYPE_L3_CODE && get_cdp_code(feat, cos) != val )
+    {
+        get_cdp_code(feat, cos) = val;
+        wrmsrl(MSR_IA32_PSR_L3_MASK_CODE(cos), (uint64_t)val);
+    }
+}
+
 struct feat_ops l3_cdp_ops = {
     .get_cos_max = cat_get_cos_max,
     .get_feat_info = l3_cdp_get_feat_info,
     .get_val = l3_cdp_get_val,
+    .get_old_val = l3_cdp_get_old_val,
+    .set_new_val = l3_cdp_set_new_val,
+    .compare_val = l3_cdp_compare_val,
+    .fits_cos_max = l3_cdp_fits_cos_max,
+    .write_msr = l3_cdp_write_msr,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 18/25] x86: L2 CAT: implement CPU init and free flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (16 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 17/25] x86: refactor psr: CDP: implement set value callback functions Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-16 11:08 ` [PATCH v9 19/25] x86: L2 CAT: implement get hw info flow Yi Sun
                   ` (7 subsequent siblings)
  25 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements the CPU init and free flow for L2 CAT including
L2 CAT initialization callback function.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - modify error handling process in 'psr_cpu_prepare' to reduce redundant
      codes.
    - reuse 'cat_init_feature' and 'cat_get_cos_max' for L2 CAT to reduce
      redundant codes.
      (suggested by Roger Pau)
    - remove unnecessary comment.
      (suggested by Jan Beulich)
    - move L2 CAT related codes from 'cpu_init_work' into 'psr_cpu_init'.
      (suggested by Jan Beulich)
    - do not free resource when allocation fails in 'psr_cpu_prepare'.
      (suggested by Jan Beulich)
v7:
    - initialize 'l2_cat'.
      (suggested by Konrad Rzeszutek Wilk)
v6:
    - use 'struct cpuid_leaf'.
      (suggested by Konrad Rzeszutek Wilk and Jan Beulich)
v5:
    - remove 'feat_l2_cat' free in 'free_feature'.
      (suggested by Jan Beulich)
    - encapsulate cpuid registers into 'struct cpuid_leaf_regs'.
      (suggested by Jan Beulich)
    - print socket info when 'opt_cpu_info' is true.
      (suggested by Jan Beulich)
    - rename 'l2_cat_get_max_cos_max' to 'l2_cat_get_cos_max'.
      (suggested by Jan Beulich)
    - rename 'dat[]' to 'data[]'
      (suggested by Jan Beulich)
    - move 'cpu_prepare_work' contents into 'psr_cpu_prepare'.
      (suggested by Jan Beulich)
v4:
    - create this patch because of codes architecture change.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c              | 27 ++++++++++++++++++++++++++-
 xen/include/asm-x86/msr-index.h |  1 +
 xen/include/asm-x86/psr.h       |  1 +
 3 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 7ede8a1..576914d 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -213,6 +213,7 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
  */
 static struct feat_node *feat_l3_cat;
 static struct feat_node *feat_l3_cdp;
+static struct feat_node *feat_l2_cat;
 
 /* Common functions */
 #define cat_default_val(len)                 \
@@ -371,7 +372,11 @@ static void cat_init_feature(struct cpuid_leaf regs,
          */
         for ( i = 1; i <= cat.cos_max; i++ )
         {
-            rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
+            if ( type == PSR_SOCKET_L3_CAT )
+                rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
+            else
+                rdmsrl(MSR_IA32_PSR_L2_MASK(i), val);
+
             feat->cos_reg_val[i] = (uint32_t)val;
         }
     }
@@ -627,6 +632,11 @@ struct feat_ops l3_cdp_ops = {
     .write_msr = l3_cdp_write_msr,
 };
 
+/* L2 CAT ops */
+struct feat_ops l2_cat_ops = {
+    .get_cos_max = cat_get_cos_max,
+};
+
 static void __init parse_psr_bool(char *s, char *value, char *feature,
                                   unsigned int mask)
 {
@@ -1427,6 +1437,10 @@ static int psr_cpu_prepare(void)
          (feat_l3_cdp = xzalloc(struct feat_node)) == NULL )
         return -ENOMEM;
 
+    if ( feat_l2_cat == NULL &&
+         (feat_l2_cat = xzalloc(struct feat_node)) == NULL )
+        return -ENOMEM;
+
     return 0;
 }
 
@@ -1479,6 +1493,17 @@ static void psr_cpu_init(void)
         }
     }
 
+    cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, &regs);
+    if ( regs.b & PSR_RESOURCE_TYPE_L2 )
+    {
+        cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 2, &regs);
+
+        feat = feat_l2_cat;
+        feat_l2_cat = NULL;
+        feat->ops = l2_cat_ops;
+        cat_init_feature(regs, feat, info, PSR_SOCKET_L2_CAT);
+    }
+
 assoc_init:
     psr_assoc_init();
 }
diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h
index 771e750..6c49c6d 100644
--- a/xen/include/asm-x86/msr-index.h
+++ b/xen/include/asm-x86/msr-index.h
@@ -345,6 +345,7 @@
 #define MSR_IA32_PSR_L3_MASK(n)	(0x00000c90 + (n))
 #define MSR_IA32_PSR_L3_MASK_CODE(n)	(0x00000c90 + (n) * 2 + 1)
 #define MSR_IA32_PSR_L3_MASK_DATA(n)	(0x00000c90 + (n) * 2)
+#define MSR_IA32_PSR_L2_MASK(n)		(0x00000d10 + (n))
 
 /* Intel Model 6 */
 #define MSR_P6_PERFCTR(n)		(0x000000c1 + (n))
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index d2262d9..4e392c8 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -23,6 +23,7 @@
 
 /* Resource Type Enumeration */
 #define PSR_RESOURCE_TYPE_L3            0x2
+#define PSR_RESOURCE_TYPE_L2            0x4
 
 /* L3 Monitoring Features */
 #define PSR_CMT_L3_OCCUPANCY            0x1
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 19/25] x86: L2 CAT: implement get hw info flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (17 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 18/25] x86: L2 CAT: implement CPU init and free flow Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 14:38   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 20/25] x86: L2 CAT: implement get value flow Yi Sun
                   ` (6 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements get HW info flow for L2 CAT including L2 CAT callback
function.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - reuse 'cat_get_feat_info' for L2 CAT to reduce redundant codes.
      (suggested by Roger Pau)
    - modify sysctl implementation of L2 CAT to input data[3] to use
      'cat_get_feat_info'.
      (suggested by Roger Pau)
    - modify macros names to newly defined ones.
      (suggested by Jan Beulich)
    - remove 'l2_info' to reuse 'l3_info'.
      (suggested by Jan Beulich)
    - modify macro name according to previous patch change.
      (suggested by Jan Beulich)
v5:
    - rename 'dat[]' to 'data[]'
      (suggested by Jan Beulich)
    - remove type check in callback function.
      (suggested by Jan Beulich)
v4:
    - create this patch because of codes architecture change.
      (suggested by Jan Beulich)
---
 xen/arch/x86/psr.c          |  4 ++++
 xen/arch/x86/sysctl.c       | 18 ++++++++++++++++++
 xen/include/asm-x86/psr.h   |  1 +
 xen/include/public/sysctl.h |  1 +
 4 files changed, 24 insertions(+)

diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 576914d..39f249e 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -280,6 +280,9 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
     case PSR_CBM_TYPE_L3_CODE:
         feat_type = PSR_SOCKET_L3_CDP;
         break;
+    case PSR_CBM_TYPE_L2:
+        feat_type = PSR_SOCKET_L2_CAT;
+        break;
     default:
         feat_type = PSR_SOCKET_UNKNOWN;
         break;
@@ -635,6 +638,7 @@ struct feat_ops l3_cdp_ops = {
 /* L2 CAT ops */
 struct feat_ops l2_cat_ops = {
     .get_cos_max = cat_get_cos_max,
+    .get_feat_info = cat_get_feat_info,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index 3bf51a4..f701910 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -216,6 +216,24 @@ long arch_do_sysctl(
             break;
         }
 
+        case XEN_SYSCTL_PSR_CAT_get_l2_info:
+        {
+            uint32_t data[PSR_INFO_CAT_SIZE];
+
+            ret = psr_get_info(sysctl->u.psr_cat_op.target,
+                               PSR_CBM_TYPE_L2, data, ARRAY_SIZE(data));
+            if ( ret )
+                break;
+
+            sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[PSR_INFO_IDX_CBM_LEN];
+            sysctl->u.psr_cat_op.u.l3_info.cos_max = data[PSR_INFO_IDX_COS_MAX];
+            sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_INFO_IDX_FLAG];
+
+            if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
+                ret = -EFAULT;
+            break;
+        }
+
         default:
             ret = -EOPNOTSUPP;
             break;
diff --git a/xen/include/asm-x86/psr.h b/xen/include/asm-x86/psr.h
index 4e392c8..a08bb94 100644
--- a/xen/include/asm-x86/psr.h
+++ b/xen/include/asm-x86/psr.h
@@ -58,6 +58,7 @@ enum cbm_type {
     PSR_CBM_TYPE_L3,
     PSR_CBM_TYPE_L3_CODE,
     PSR_CBM_TYPE_L3_DATA,
+    PSR_CBM_TYPE_L2,
 };
 
 extern struct psr_cmt *psr_cmt;
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 00f5e77..1fe8fe4 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
 DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
 
 #define XEN_SYSCTL_PSR_CAT_get_l3_info               0
+#define XEN_SYSCTL_PSR_CAT_get_l2_info               1
 struct xen_sysctl_psr_cat_op {
     uint32_t cmd;       /* IN: XEN_SYSCTL_PSR_CAT_* */
     uint32_t target;    /* IN */
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 20/25] x86: L2 CAT: implement get value flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (18 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 19/25] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 14:39   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 21/25] x86: L2 CAT: implement set " Yi Sun
                   ` (5 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements L2 CAT get value callback function and
interface in domctl.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - reuse 'cat_get_val' for L2 CAT to reduce redundant codes
      (suggested by Roger Pau)
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v5:
    - remove type check in callback function.
      (suggested by Jan Beulich)
v4:
    - create this patch because of codes architecture change.
      (suggested by Jan Beulich)
---
 xen/arch/x86/domctl.c       | 8 ++++++++
 xen/arch/x86/psr.c          | 1 +
 xen/include/public/domctl.h | 1 +
 3 files changed, 10 insertions(+)

diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 76923c8..7f55cfd 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1478,6 +1478,14 @@ long arch_do_domctl(
             copyback = 1;
             break;
 
+        case XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM:
+            domctl->u.psr_cat_op.data = 0;
+            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
+                              (uint32_t *)&domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L2);
+            copyback = 1;
+            break;
+
         default:
             ret = -EOPNOTSUPP;
             break;
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 39f249e..4cbcbe0 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -639,6 +639,7 @@ struct feat_ops l3_cdp_ops = {
 struct feat_ops l2_cat_ops = {
     .get_cos_max = cat_get_cos_max,
     .get_feat_info = cat_get_feat_info,
+    .get_val = cat_get_val,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 85cbb7c..8c183ba 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
 #define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA    3
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE    4
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA    5
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM     7
     uint32_t cmd;       /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
     uint32_t target;    /* IN */
     uint64_t data;      /* IN/OUT */
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 21/25] x86: L2 CAT: implement set value flow.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (19 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 20/25] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-27 14:40   ` Jan Beulich
  2017-03-16 11:08 ` [PATCH v9 22/25] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
                   ` (4 subsequent siblings)
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements L2 CAT set value related callback functions
and domctl interface.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
v9:
    - reuse some CAT common functions for L2 CAT to reduce redundant codes.
      (suggested by Roger Pau)
    - remove parameter 'found' from 'cat_compare_val' and modify the return
      values to let caller know if the id is found or not. These things are
      done in patch "x86: refactor psr: set value: implement cos finding flow."
      (suggested by Roger Pau and Dario Faggioli)
    - remove 'get_cos_num' related codes.
      (suggested by Jan Beulich)
    - modify 'l2_cat_write_msr' according to previous patch change.
    - changes about 'uint64_t' to 'uint32_t'.
      (suggested by Jan Beulich)
v8:
    - modify 'l2_cat_write_msr' to 'void'.
v5:
    - remove type check in callback function.
      (suggested by Jan Beulich)
    - modify return value of callback functions because we do not need them
      to return number of entries the feature uses. In caller, we call
      'get_cos_num' to get the number of entries the feature uses.
      (suggested by Jan Beulich)
    - remove 'l2_cat_get_cos_max_from_type'.
      (suggested by Jan Beulich)
    - rename 'l2_cat_exceeds_cos_max' to 'l2_cat_fits_cos_max'.
      (suggested by Jan Beulich)
v4:
    - create this patch because of codes architecture change.
      (suggested by Jan Beulich)
---
 xen/arch/x86/domctl.c       |  6 ++++++
 xen/arch/x86/psr.c          | 17 +++++++++++++++++
 xen/include/public/domctl.h |  1 +
 3 files changed, 24 insertions(+)

diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 7f55cfd..e732b99 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1454,6 +1454,12 @@ long arch_do_domctl(
                               PSR_CBM_TYPE_L3_DATA);
             break;
 
+        case XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM:
+            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
+                              (uint32_t)domctl->u.psr_cat_op.data,
+                              PSR_CBM_TYPE_L2);
+            break;
+
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
             domctl->u.psr_cat_op.data = 0;
             ret = psr_get_val(d, domctl->u.psr_cat_op.target,
diff --git a/xen/arch/x86/psr.c b/xen/arch/x86/psr.c
index 4cbcbe0..4059bbb 100644
--- a/xen/arch/x86/psr.c
+++ b/xen/arch/x86/psr.c
@@ -636,10 +636,27 @@ struct feat_ops l3_cdp_ops = {
 };
 
 /* L2 CAT ops */
+static void l2_cat_write_msr(unsigned int cos, uint32_t val,
+                             enum cbm_type type, struct feat_node *feat)
+{
+    if ( feat->cos_reg_val[cos] != val )
+    {
+        feat->cos_reg_val[cos] = val;
+        wrmsrl(MSR_IA32_PSR_L2_MASK(cos), (uint64_t)val);
+    }
+
+    return;
+}
+
 struct feat_ops l2_cat_ops = {
     .get_cos_max = cat_get_cos_max,
     .get_feat_info = cat_get_feat_info,
     .get_val = cat_get_val,
+    .get_old_val = cat_get_old_val,
+    .set_new_val = cat_set_new_val,
+    .compare_val = cat_compare_val,
+    .fits_cos_max = cat_fits_cos_max,
+    .write_msr = l2_cat_write_msr,
 };
 
 static void __init parse_psr_bool(char *s, char *value, char *feature,
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 8c183ba..523a2cd 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
 #define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA    3
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE    4
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA    5
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM     6
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM     7
     uint32_t cmd;       /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
     uint32_t target;    /* IN */
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 22/25] tools: L2 CAT: support get HW info for L2 CAT.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (20 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 21/25] x86: L2 CAT: implement set " Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-16 11:08 ` [PATCH v9 23/25] tools: L2 CAT: support show cbm " Yi Sun
                   ` (3 subsequent siblings)
  25 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements xl/xc changes to support get HW info
for L2 CAT.

'xl psr-hwinfo' is updated to show both L3 CAT and L2 CAT
info.

Example(on machine which only supports L2 CAT):
Cache Monitoring Technology (CMT):
Enabled         : 0
Cache Allocation Technology (CAT): L2
Socket ID       : 0
Maximum COS     : 3
CBM length      : 8
Default CBM     : 0xff

Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v9:
    - add some cases to handle return error no.
    - move xl_cmdimpl.c codes into xl/xl_psr.c.
    - change 'l3_info' to 'cat_info' to cover both L3 and L2 CAT.
v6:
    - adjust '{' position for 'switch'.
      (suggested by Wei Liu)
    - modify commit message to remove error log.
      (suggested by Dario Faggioli)
v5:
    - modify commit message to remove error log.
      (suggested by Wei Liu and Jan Beulich)
    - replace unnecessary 'return' to 'break'.
      (suggested by Wei Liu)
    - restore 'libxl_psr_cat_get_l3_info' to keep interface backward compatible
      but change codes in it to call new function to get hw info.
      (suggested by Wei Liu)
    - add 'L2_CBM' into 'psr_cbm_type' because it is interface change which
      should be in same patch with new 'LIBXL_HAVE_' macro.
      (suggested by Wei Liu)
    - addjust logs sentence to make unnecessary error logs not show.
      (suggested by Wei Liu and Jan Beulich)
v4:
    - create this patch to help reviewers better understand the codes.
---
 tools/libxc/include/xenctrl.h |  6 ++---
 tools/libxc/xc_psr.c          | 39 +++++++++++++++++++++++---------
 tools/libxl/libxl.h           |  9 ++++++++
 tools/libxl/libxl_psr.c       | 28 ++++++++++++++++++-----
 tools/libxl/libxl_types.idl   |  1 +
 tools/xl/xl_psr.c             | 52 +++++++++++++++++++++++++++++++++----------
 xen/arch/x86/sysctl.c         | 18 +++++++--------
 xen/include/public/sysctl.h   |  2 +-
 8 files changed, 114 insertions(+), 41 deletions(-)

diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index a48981a..99c6fa5 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2442,9 +2442,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
 int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
                                xc_psr_cat_type type, uint32_t target,
                                uint64_t *data);
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
-                           uint32_t *cos_max, uint32_t *cbm_len,
-                           bool *cdp_enabled);
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+                        uint32_t *cos_max, uint32_t *cbm_len,
+                        bool *cdp_enabled);
 
 int xc_get_cpu_levelling_caps(xc_interface *xch, uint32_t *caps);
 int xc_get_cpu_featureset(xc_interface *xch, uint32_t index,
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 43b3286..84a08c4 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -317,24 +317,41 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
     return rc;
 }
 
-int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
-                           uint32_t *cos_max, uint32_t *cbm_len,
-                           bool *cdp_enabled)
+int xc_psr_cat_get_info(xc_interface *xch, uint32_t socket, unsigned int lvl,
+                        uint32_t *cos_max, uint32_t *cbm_len, bool *cdp_enabled)
 {
-    int rc;
+    int rc = -1;
     DECLARE_SYSCTL;
 
     sysctl.cmd = XEN_SYSCTL_psr_cat_op;
-    sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
     sysctl.u.psr_cat_op.target = socket;
 
-    rc = xc_sysctl(xch, &sysctl);
-    if ( !rc )
+    switch ( lvl )
     {
-        *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
-        *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
-        *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
-                       XEN_SYSCTL_PSR_CAT_L3_CDP;
+    case 2:
+        sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l2_info;
+        rc = xc_sysctl(xch, &sysctl);
+        if ( !rc )
+        {
+            *cos_max = sysctl.u.psr_cat_op.u.cat_info.cos_max;
+            *cbm_len = sysctl.u.psr_cat_op.u.cat_info.cbm_len;
+            *cdp_enabled = false;
+        }
+        break;
+    case 3:
+        sysctl.u.psr_cat_op.cmd = XEN_SYSCTL_PSR_CAT_get_l3_info;
+        rc = xc_sysctl(xch, &sysctl);
+        if ( !rc )
+        {
+            *cos_max = sysctl.u.psr_cat_op.u.cat_info.cos_max;
+            *cbm_len = sysctl.u.psr_cat_op.u.cat_info.cbm_len;
+            *cdp_enabled = sysctl.u.psr_cat_op.u.cat_info.flags &
+                           XEN_SYSCTL_PSR_CAT_L3_CDP;
+        }
+        break;
+    default:
+        errno = EOPNOTSUPP;
+        break;
     }
 
     return rc;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 92f1751..6c6fb01 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -904,6 +904,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, const libxl_mac *src);
  * If this is defined, the Code and Data Prioritization feature is supported.
  */
 #define LIBXL_HAVE_PSR_CDP 1
+
+/*
+ * LIBXL_HAVE_PSR_L2_CAT
+ *
+ * If this is defined, the L2 Cache Allocation Technology feature is supported.
+ */
+#define LIBXL_HAVE_PSR_L2_CAT 1
 #endif
 
 /*
@@ -2172,6 +2179,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
  * On success, the function returns an array of elements in 'info',
  * and the length in 'nr'.
  */
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+                           int *nr, unsigned int lvl);
 int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
                               int *nr);
 void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr);
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index ec5c79d..f55ba1e 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -91,6 +91,15 @@ static void libxl__psr_cat_log_err_msg(libxl__gc *gc, int err)
     case ENXIO:
         msg = "Unable to set code or data CBM when CDP is disabled";
         break;
+    case EINVAL:
+        msg = "Invalid input or some internal values are not expected";
+        break;
+    case ERANGE:
+        msg = "Socket number is wrong";
+        break;
+    case ENOSPC:
+        msg = "Value array exceeds the range";
+        break;
 
     default:
         libxl__psr_log_err_msg(gc, err);
@@ -352,8 +361,8 @@ int libxl_psr_cat_get_cbm(libxl_ctx *ctx, uint32_t domid,
     return rc;
 }
 
-int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
-                              int *nr)
+int libxl_psr_cat_get_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+                           int *nr, unsigned int lvl)
 {
     GC_INIT(ctx);
     int rc;
@@ -380,9 +389,8 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
 
     libxl_for_each_set_bit(socketid, socketmap) {
         ptr[i].id = socketid;
-        if (xc_psr_cat_get_l3_info(ctx->xch, socketid, &ptr[i].cos_max,
-                                   &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
-            libxl__psr_cat_log_err_msg(gc, errno);
+        if (xc_psr_cat_get_info(ctx->xch, socketid, lvl, &ptr[i].cos_max,
+                                &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
             rc = ERROR_FAIL;
             free(ptr);
             goto out;
@@ -398,6 +406,16 @@ out:
     return rc;
 }
 
+int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
+                              int *nr)
+{
+    int rc;
+
+    rc = libxl_psr_cat_get_info(ctx, info, nr, 3);
+
+    return rc;
+}
+
 void libxl_psr_cat_info_list_free(libxl_psr_cat_info *list, int nr)
 {
     int i;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a612d1f..5a401b8 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -899,6 +899,7 @@ libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
     (1, "L3_CBM"),
     (2, "L3_CBM_CODE"),
     (3, "L3_CBM_DATA"),
+    (4, "L2_CBM"),
     ])
 
 libxl_psr_cat_info = Struct("psr_cat_info", [
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index c061b29..271b88f 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -294,21 +294,19 @@ int main_psr_cmt_show(int argc, char **argv)
 }
 #endif
 
-#ifdef LIBXL_HAVE_PSR_CAT
-static int psr_cat_hwinfo(void)
+#if defined(LIBXL_HAVE_PSR_CAT) || defined(LIBXL_HAVE_PSR_L2_CAT)
+static int psr_l3_cat_hwinfo(void)
 {
-    int rc;
-    int i, nr;
+    int rc, nr;
+    unsigned int i;
     uint32_t l3_cache_size;
     libxl_psr_cat_info *info;
 
-    printf("Cache Allocation Technology (CAT):\n");
-
-    rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
-    if (rc) {
-        fprintf(stderr, "Failed to get cat info\n");
+    rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+    if (rc)
         return rc;
-    }
+
+    printf("Cache Allocation Technology (CAT):\n");
 
     for (i = 0; i < nr; i++) {
         rc = libxl_psr_cmt_get_l3_cache_size(ctx, info[i].id, &l3_cache_size);
@@ -417,7 +415,7 @@ static int psr_cat_show(uint32_t domid)
     int rc;
     libxl_psr_cat_info *info;
 
-    rc = libxl_psr_cat_get_l3_info(ctx, &info, &nr);
+    rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
     if (rc) {
         fprintf(stderr, "Failed to get cat info\n");
         return rc;
@@ -434,6 +432,32 @@ out:
     return rc;
 }
 
+static int psr_l2_cat_hwinfo(void)
+{
+    int rc;
+    unsigned int i;
+    int nr;
+    libxl_psr_cat_info *info;
+
+    rc = libxl_psr_cat_get_info(ctx, &info, &nr, 2);
+    if (rc)
+        return rc;
+
+    printf("Cache Allocation Technology (CAT): L2\n");
+
+    for (i = 0; i < nr; i++) {
+        /* There is no CMT on L2 cache so far. */
+        printf("%-16s: %u\n", "Socket ID", info[i].id);
+        printf("%-16s: %u\n", "Maximum COS", info[i].cos_max);
+        printf("%-16s: %u\n", "CBM length", info[i].cbm_len);
+        printf("%-16s: %#llx\n", "Default CBM",
+               (1ull << info[i].cbm_len) - 1);
+    }
+
+    libxl_psr_cat_info_list_free(info, nr);
+    return rc;
+}
+
 int main_psr_cat_cbm_set(int argc, char **argv)
 {
     uint32_t domid;
@@ -551,7 +575,11 @@ int main_psr_hwinfo(int argc, char **argv)
         ret = psr_cmt_hwinfo();
 
     if (!ret && (all || cat))
-        ret = psr_cat_hwinfo();
+        ret = psr_l3_cat_hwinfo();
+
+    /* L2 CAT is independent of CMT and L3 CAT */
+    if (all || cat)
+        ret = psr_l2_cat_hwinfo();
 
     return ret;
 }
diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c
index f701910..c00c234 100644
--- a/xen/arch/x86/sysctl.c
+++ b/xen/arch/x86/sysctl.c
@@ -182,11 +182,11 @@ long arch_do_sysctl(
                                PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
             if ( !ret )
             {
-                sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+                sysctl->u.psr_cat_op.u.cat_info.cbm_len =
                                                data[PSR_INFO_IDX_CBM_LEN];
-                sysctl->u.psr_cat_op.u.l3_info.cos_max =
+                sysctl->u.psr_cat_op.u.cat_info.cos_max =
                                                data[PSR_INFO_IDX_COS_MAX];
-                sysctl->u.psr_cat_op.u.l3_info.flags =
+                sysctl->u.psr_cat_op.u.cat_info.flags =
                                                data[PSR_INFO_IDX_FLAG];
             }
             else
@@ -202,11 +202,11 @@ long arch_do_sysctl(
                                    ARRAY_SIZE(data));
                 if ( !ret )
                 {
-                    sysctl->u.psr_cat_op.u.l3_info.cbm_len =
+                    sysctl->u.psr_cat_op.u.cat_info.cbm_len =
                                                data[PSR_INFO_IDX_CBM_LEN];
-                    sysctl->u.psr_cat_op.u.l3_info.cos_max =
+                    sysctl->u.psr_cat_op.u.cat_info.cos_max =
                                                data[PSR_INFO_IDX_COS_MAX];
-                    sysctl->u.psr_cat_op.u.l3_info.flags =
+                    sysctl->u.psr_cat_op.u.cat_info.flags =
                                                data[PSR_INFO_IDX_FLAG];
                 }
             }
@@ -225,9 +225,9 @@ long arch_do_sysctl(
             if ( ret )
                 break;
 
-            sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[PSR_INFO_IDX_CBM_LEN];
-            sysctl->u.psr_cat_op.u.l3_info.cos_max = data[PSR_INFO_IDX_COS_MAX];
-            sysctl->u.psr_cat_op.u.l3_info.flags = data[PSR_INFO_IDX_FLAG];
+            sysctl->u.psr_cat_op.u.cat_info.cbm_len = data[PSR_INFO_IDX_CBM_LEN];
+            sysctl->u.psr_cat_op.u.cat_info.cos_max = data[PSR_INFO_IDX_COS_MAX];
+            sysctl->u.psr_cat_op.u.cat_info.flags = data[PSR_INFO_IDX_FLAG];
 
             if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
                 ret = -EFAULT;
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 1fe8fe4..a3998c6 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -754,7 +754,7 @@ struct xen_sysctl_psr_cat_op {
             uint32_t cos_max;   /* OUT: Maximum COS */
 #define XEN_SYSCTL_PSR_CAT_L3_CDP       (1u << 0)
             uint32_t flags;     /* OUT: CAT flags */
-        } l3_info;
+        } cat_info;
     } u;
 };
 typedef struct xen_sysctl_psr_cat_op xen_sysctl_psr_cat_op_t;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 23/25] tools: L2 CAT: support show cbm for L2 CAT.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (21 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 22/25] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-16 11:08 ` [PATCH v9 24/25] tools: L2 CAT: support set " Yi Sun
                   ` (2 subsequent siblings)
  25 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements changes in xl/xc changes to support
showing CBM of L2 CAT.

The new level option is introduced to original CAT showing
command in order to show CBM for specified level CAT.
- 'xl psr-cat-show' is updated to show CBM of a domain
  according to input cache level.

Examples:
root@:~$ xl psr-cat-show -l2 1
Socket ID       : 0
Default CBM     : 0xff
   ID                     NAME             CBM
    1                 ubuntu14            0x7f

Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v9:
    - move xl_cmdimpl.c changes into xl/xl_psr.c.
    - move xl_cmdtable.c changes into xl/xl_cmdtable.c.
v6:
    - check if input level is correct.
    - adjust '{' postion for 'if'.
      (suggested by Wei Liu)
v5:
    - remove 'L2_CBM' in idl because it has been moved to patch 21:
      "tools: L2 CAT: support get HW info for L2 CAT".
      (suggested by Wei Liu)
v4:
    - create this patch because of codes architecture change.
---
 tools/libxc/include/xenctrl.h |  1 +
 tools/libxc/xc_psr.c          |  3 ++
 tools/xl/xl_cmdtable.c        |  3 +-
 tools/xl/xl_psr.c             | 85 +++++++++++++++++++++++++++++--------------
 4 files changed, 63 insertions(+), 29 deletions(-)

diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 99c6fa5..0fd9326 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2418,6 +2418,7 @@ enum xc_psr_cat_type {
     XC_PSR_CAT_L3_CBM      = 1,
     XC_PSR_CAT_L3_CBM_CODE = 2,
     XC_PSR_CAT_L3_CBM_DATA = 3,
+    XC_PSR_CAT_L2_CBM      = 4,
 };
 typedef enum xc_psr_cat_type xc_psr_cat_type;
 
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 84a08c4..04f5927 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -299,6 +299,9 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
     case XC_PSR_CAT_L3_CBM_DATA:
         cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
         break;
+    case XC_PSR_CAT_L2_CBM:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM;
+        break;
     default:
         errno = EINVAL;
         return -1;
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 1219b33..ab7ad60 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -556,7 +556,8 @@ struct cmd_spec cmd_table[] = {
     { "psr-cat-show",
       &main_psr_cat_show, 0, 1,
       "Show Cache Allocation Technology information",
-      "<Domain>",
+      "[options] <Domain>",
+      "-l <level>        Specify the cache level to process, otherwise L3 cache is processed\n"
     },
 
 #endif
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index 271b88f..575f4a0 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -342,7 +342,7 @@ static void psr_cat_print_one_domain_cbm_type(uint32_t domid, uint32_t socketid,
 }
 
 static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
-                                         bool cdp_enabled)
+                                         bool cdp_enabled, unsigned int lvl)
 {
     char *domain_name;
 
@@ -350,27 +350,38 @@ static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid,
     printf("%5d%25s", domid, domain_name);
     free(domain_name);
 
-    if (!cdp_enabled) {
-        psr_cat_print_one_domain_cbm_type(domid, socketid,
-                                          LIBXL_PSR_CBM_TYPE_L3_CBM);
-    } else {
-        psr_cat_print_one_domain_cbm_type(domid, socketid,
-                                          LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+    switch (lvl) {
+    case 3:
+        if (!cdp_enabled) {
+            psr_cat_print_one_domain_cbm_type(domid, socketid,
+                                              LIBXL_PSR_CBM_TYPE_L3_CBM);
+        } else {
+            psr_cat_print_one_domain_cbm_type(domid, socketid,
+                                              LIBXL_PSR_CBM_TYPE_L3_CBM_CODE);
+            psr_cat_print_one_domain_cbm_type(domid, socketid,
+                                              LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+        }
+        break;
+    case 2:
         psr_cat_print_one_domain_cbm_type(domid, socketid,
-                                          LIBXL_PSR_CBM_TYPE_L3_CBM_DATA);
+                                          LIBXL_PSR_CBM_TYPE_L2_CBM);
+        break;
+    default:
+        printf("Input lvl %d is wrong!", lvl);
+        break;
     }
 
     printf("\n");
 }
 
 static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
-                                    bool cdp_enabled)
+                                    bool cdp_enabled, unsigned int lvl)
 {
     int i, nr_domains;
     libxl_dominfo *list;
 
     if (domid != INVALID_DOMID) {
-        psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled);
+        psr_cat_print_one_domain_cbm(domid, socketid, cdp_enabled, lvl);
         return 0;
     }
 
@@ -380,49 +391,59 @@ static int psr_cat_print_domain_cbm(uint32_t domid, uint32_t socketid,
     }
 
     for (i = 0; i < nr_domains; i++)
-        psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled);
+        psr_cat_print_one_domain_cbm(list[i].domid, socketid, cdp_enabled, lvl);
     libxl_dominfo_list_free(list, nr_domains);
 
     return 0;
 }
 
-static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info)
+static int psr_cat_print_socket(uint32_t domid, libxl_psr_cat_info *info,
+                                unsigned int lvl)
 {
     int rc;
     uint32_t l3_cache_size;
 
-    rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
-    if (rc) {
-        fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
-                info->id);
-        return -1;
+    printf("%-16s: %u\n", "Socket ID", info->id);
+
+    /* So far, CMT only supports L3 cache. */
+    if (lvl == 3) {
+        rc = libxl_psr_cmt_get_l3_cache_size(ctx, info->id, &l3_cache_size);
+        if (rc) {
+            fprintf(stderr, "Failed to get l3 cache size for socket:%d\n",
+                    info->id);
+            return -1;
+        }
+        printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
     }
 
-    printf("%-16s: %u\n", "Socket ID", info->id);
-    printf("%-16s: %uKB\n", "L3 Cache", l3_cache_size);
     printf("%-16s: %#llx\n", "Default CBM", (1ull << info->cbm_len) - 1);
     if (info->cdp_enabled)
         printf("%5s%25s%16s%16s\n", "ID", "NAME", "CBM (code)", "CBM (data)");
     else
         printf("%5s%25s%16s\n", "ID", "NAME", "CBM");
 
-    return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled);
+    return psr_cat_print_domain_cbm(domid, info->id, info->cdp_enabled, lvl);
 }
 
-static int psr_cat_show(uint32_t domid)
+static int psr_cat_show(uint32_t domid, unsigned int lvl)
 {
     int i, nr;
     int rc;
     libxl_psr_cat_info *info;
 
-    rc = libxl_psr_cat_get_info(ctx, &info, &nr, 3);
+    if (lvl != 2 && lvl != 3) {
+        fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+        return EXIT_FAILURE;
+    }
+
+    rc = libxl_psr_cat_get_info(ctx, &info, &nr, lvl);
     if (rc) {
-        fprintf(stderr, "Failed to get cat info\n");
+        fprintf(stderr, "Failed to get %s cat info\n", (lvl == 3)?"L3":"L2");
         return rc;
     }
 
     for (i = 0; i < nr; i++) {
-        rc = psr_cat_print_socket(domid, info + i);
+        rc = psr_cat_print_socket(domid, info + i, lvl);
         if (rc)
             goto out;
     }
@@ -533,11 +554,19 @@ int main_psr_cat_cbm_set(int argc, char **argv)
 
 int main_psr_cat_show(int argc, char **argv)
 {
-    int opt;
+    int opt = 0;
     uint32_t domid;
+    unsigned int lvl = 3;
 
-    SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cat-show", 0) {
-        /* No options */
+    static struct option opts[] = {
+        {"level", 1, 0, 'l'},
+        COMMON_LONG_OPTS
+    };
+
+    SWITCH_FOREACH_OPT(opt, "l:", opts, "psr-cat-show", 0) {
+    case 'l':
+        lvl = atoi(optarg);
+        break;
     }
 
     if (optind >= argc)
@@ -549,7 +578,7 @@ int main_psr_cat_show(int argc, char **argv)
         return 2;
     }
 
-    return psr_cat_show(domid);
+    return psr_cat_show(domid, lvl);
 }
 
 int main_psr_hwinfo(int argc, char **argv)
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 24/25] tools: L2 CAT: support set cbm for L2 CAT.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (22 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 23/25] tools: L2 CAT: support show cbm " Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-28 14:04   ` Wei Liu
  2017-03-16 11:08 ` [PATCH v9 25/25] docs: add L2 CAT description in docs Yi Sun
  2017-03-16 11:20 ` [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Jan Beulich
  25 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch implements the xl/xc changes to support set CBM
for L2 CAT.

The new level option is introduced to original CAT setting
command in order to set CBM for specified level CAT.
- 'xl psr-cat-set' is updated to set cache capacity bitmasks(CBM)
  for a domain according to input cache level.

root@:~$ xl psr-cat-set -l2 1 0x7f

root@:~$ xl psr-cat-show -l2 1
Socket ID       : 0
Default CBM     : 0xff
   ID                     NAME             CBM
    1                 ubuntu14            0x7f

Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
v9:
    - handle the case to set both CODE and DATA for CDP at same time.
      For such case, user does not input '-c' or '-d' to set CDP cbm.
    - move xl_cmdimpl.c changes into xl/xl_psr.c.
    - move xl_cmdtable.c changes into xl/xl_cmdtable.c.
v6:
    - rename 'psr-cat-cbm-set' to 'psr-cat-set'.
      (suggested by Kevin Tian)
    - return 'EXIT_FAILURE' for error case.
      (suggested by Dario Faggioli)
    - print error info when input level is wrong.
v4:
    - create this patch because of codes architecture change.
---
 tools/libxc/xc_psr.c    |  3 +++
 tools/libxl/libxl_psr.c | 42 ++++++++++++++++++++++++++++++++++++++----
 tools/xl/xl_cmdtable.c  |  3 ++-
 tools/xl/xl_psr.c       | 33 +++++++++++++++++++++++----------
 4 files changed, 66 insertions(+), 15 deletions(-)

diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
index 04f5927..039b920 100644
--- a/tools/libxc/xc_psr.c
+++ b/tools/libxc/xc_psr.c
@@ -266,6 +266,9 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
     case XC_PSR_CAT_L3_CBM_DATA:
         cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
         break;
+    case XC_PSR_CAT_L2_CBM:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM;
+        break;
     default:
         errno = EINVAL;
         return -1;
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
index f55ba1e..15e7e59 100644
--- a/tools/libxl/libxl_psr.c
+++ b/tools/libxl/libxl_psr.c
@@ -317,6 +317,7 @@ int libxl_psr_cat_set_cbm(libxl_ctx *ctx, uint32_t domid,
     GC_INIT(ctx);
     int rc;
     int socketid, nr_sockets;
+    libxl_psr_cat_info cat_info;
 
     rc = libxl__count_physical_sockets(gc, &nr_sockets);
     if (rc) {
@@ -331,10 +332,43 @@ int libxl_psr_cat_set_cbm(libxl_ctx *ctx, uint32_t domid,
             break;
 
         xc_type = libxl__psr_cbm_type_to_libxc_psr_cat_type(type);
-        if (xc_psr_cat_set_domain_data(ctx->xch, domid, xc_type,
-                                       socketid, cbm)) {
-            libxl__psr_cat_log_err_msg(gc, errno);
-            rc = ERROR_FAIL;
+
+        if (xc_type == XC_PSR_CAT_L3_CBM) {
+            if (xc_psr_cat_get_info(ctx->xch, socketid, 3, &cat_info.cos_max,
+                                    &cat_info.cbm_len, &cat_info.cdp_enabled)) {
+                libxl__psr_cat_log_err_msg(gc, errno);
+                rc = ERROR_FAIL;
+                goto out;
+            }
+        }
+
+        /*
+         * User can set both CDP CODE and DATA at same time when the '-c' or
+         * '-d' is not input. In such case, the input type is
+         * LIBXL_PSR_CBM_TYPE_L3_CBM. So, we need check if cdp_enabled is true.
+         * If it is true, we need set both CODE and DATA.
+         */
+        if (xc_type == XC_PSR_CAT_L3_CBM && cat_info.cdp_enabled) {
+            xc_type = XC_PSR_CAT_L3_CBM_CODE;
+            if (xc_psr_cat_set_domain_data(ctx->xch, domid, xc_type,
+                                           socketid, cbm)) {
+                libxl__psr_cat_log_err_msg(gc, errno);
+                rc = ERROR_FAIL;
+            }
+
+            xc_type = XC_PSR_CAT_L3_CBM_DATA;
+            if (rc != ERROR_FAIL &&
+                xc_psr_cat_set_domain_data(ctx->xch, domid, xc_type,
+                                           socketid, cbm)) {
+                libxl__psr_cat_log_err_msg(gc, errno);
+                rc = ERROR_FAIL;
+            }
+        } else {
+            if (xc_psr_cat_set_domain_data(ctx->xch, domid, xc_type,
+                                           socketid, cbm)) {
+                libxl__psr_cat_log_err_msg(gc, errno);
+                rc = ERROR_FAIL;
+            }
         }
     }
 
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index ab7ad60..d332e1a 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -545,11 +545,12 @@ struct cmd_spec cmd_table[] = {
     },
 #endif
 #ifdef LIBXL_HAVE_PSR_CAT
-    { "psr-cat-cbm-set",
+    { "psr-cat-set",
       &main_psr_cat_cbm_set, 0, 1,
       "Set cache capacity bitmasks(CBM) for a domain",
       "[options] <Domain> <CBM>",
       "-s <socket>       Specify the socket to process, otherwise all sockets are processed\n"
+      "-l <level>        Specify the cache level to process, otherwise L3 cache is processed\n"
       "-c                Set code CBM if CDP is supported\n"
       "-d                Set data CBM if CDP is supported\n"
     },
diff --git a/tools/xl/xl_psr.c b/tools/xl/xl_psr.c
index 575f4a0..7309d4f 100644
--- a/tools/xl/xl_psr.c
+++ b/tools/xl/xl_psr.c
@@ -490,19 +490,21 @@ int main_psr_cat_cbm_set(int argc, char **argv)
     char *value;
     libxl_string_list socket_list;
     unsigned long start, end;
-    int i, j, len;
+    unsigned int i, j, len;
+    unsigned int lvl = 3;
 
     static struct option opts[] = {
         {"socket", 1, 0, 's'},
         {"data", 0, 0, 'd'},
         {"code", 0, 0, 'c'},
+        {"level", 1, 0, 'l'},
         COMMON_LONG_OPTS
     };
 
     libxl_socket_bitmap_alloc(ctx, &target_map, 0);
     libxl_bitmap_set_none(&target_map);
 
-    SWITCH_FOREACH_OPT(opt, "s:cd", opts, "psr-cat-cbm-set", 2) {
+    SWITCH_FOREACH_OPT(opt, "s:l:cd", opts, "psr-cat-set", 2) {
     case 's':
         trim(isspace, optarg, &value);
         split_string_into_string_list(value, ",", &socket_list);
@@ -522,24 +524,35 @@ int main_psr_cat_cbm_set(int argc, char **argv)
     case 'c':
         opt_code = 1;
         break;
+    case 'l':
+        lvl = atoi(optarg);
+        break;
     }
 
-    if (opt_data && opt_code) {
-        fprintf(stderr, "Cannot handle -c and -d at the same time\n");
-        return -1;
-    } else if (opt_data) {
-        type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
-    } else if (opt_code) {
-        type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+    if (lvl == 2)
+        type = LIBXL_PSR_CBM_TYPE_L2_CBM;
+    else if (lvl == 3) {
+        if (opt_data && opt_code) {
+            fprintf(stderr, "Cannot handle -c and -d at the same time\n");
+            return EXIT_FAILURE;
+        } else if (opt_data) {
+            type = LIBXL_PSR_CBM_TYPE_L3_CBM_DATA;
+        } else if (opt_code) {
+            type = LIBXL_PSR_CBM_TYPE_L3_CBM_CODE;
+        } else {
+            type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+        }
     } else {
         type = LIBXL_PSR_CBM_TYPE_L3_CBM;
+        fprintf(stderr, "Input lvl %d is wrong\n", lvl);
+        return EXIT_FAILURE;
     }
 
     if (libxl_bitmap_is_empty(&target_map))
         libxl_bitmap_set_any(&target_map);
 
     if (argc != optind + 2) {
-        help("psr-cat-cbm-set");
+        help("psr-cat-set");
         return 2;
     }
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* [PATCH v9 25/25] docs: add L2 CAT description in docs.
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (23 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 24/25] tools: L2 CAT: support set " Yi Sun
@ 2017-03-16 11:08 ` Yi Sun
  2017-03-16 11:20 ` [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Jan Beulich
  25 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-16 11:08 UTC (permalink / raw)
  To: xen-devel
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, Yi Sun, mengxu, jbeulich, chao.p.peng, roger.pau

This patch adds L2 CAT description in related documents.

Signed-off-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
 docs/man/xl.pod.1.in      | 25 ++++++++++++++++++++++---
 docs/misc/xl-psr.markdown | 18 ++++++++++++------
 2 files changed, 34 insertions(+), 9 deletions(-)

diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 7caed08..5e7676e 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -1711,6 +1711,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
 a domain is attached to the monitoring service, monitoring data can be shown
 for any of these monitoring types.
 
+There is no cache monitoring and memory bandwidth monitoring on L2 cache so
+far.
+
 =over 4
 
 =item B<psr-cmt-attach> [I<domain-id>]
@@ -1735,7 +1738,7 @@ monitor types are:
 
 Intel Broadwell and later server platforms offer capabilities to configure and
 make use of the Cache Allocation Technology (CAT) mechanisms, which enable more
-cache resources (i.e. L3 cache) to be made available for high priority
+cache resources (i.e. L3/L2 cache) to be made available for high priority
 applications. In the Xen implementation, CAT is used to control cache allocation
 on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
 (CBM) for the domain.
@@ -1745,7 +1748,7 @@ Intel Broadwell and later server platforms also offer Code/Data Prioritization
 applications. CDP is used on a per VM basis in the Xen implementation. To
 specify code or data CBM for the domain, CDP feature must be enabled and CBM
 type options need to be specified when setting CBM, and the type options (code
-and data) are mutually exclusive.
+and data) are mutually exclusive. There is no CDP support on L2 so far.
 
 =over 4
 
@@ -1762,6 +1765,11 @@ B<OPTIONS>
 
 Specify the socket to process, otherwise all sockets are processed.
 
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
 =item B<-c>, B<--code>
 
 Set code CBM when CDP is enabled.
@@ -1772,10 +1780,21 @@ Set data CBM when CDP is enabled.
 
 =back
 
-=item B<psr-cat-show> [I<domain-id>]
+=item B<psr-cat-show> [I<OPTIONS>] [I<domain-id>]
 
 Show CAT settings for a certain domain or all domains.
 
+B<OPTIONS>
+
+=over 4
+
+=item B<-l LEVEL>, B<--level=LEVEL>
+
+Specify the cache level to process, otherwise the last level cache (L3) is
+processed.
+
+=back
+
 =back
 
 =head1 IGNORED FOR COMPATIBILITY WITH XM
diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
index c3c1e8e..04dd957 100644
--- a/docs/misc/xl-psr.markdown
+++ b/docs/misc/xl-psr.markdown
@@ -70,7 +70,7 @@ total-mem-bandwidth instead of cache-occupancy). E.g. after a `xl psr-cmt-attach
 
 Cache Allocation Technology (CAT) is a new feature available on Intel
 Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
-partition cache allocation (i.e. L3 cache) based on application priority or
+partition cache allocation (i.e. L3/L2 cache) based on application priority or
 Class of Service (COS). Each COS is configured using capacity bitmasks (CBM)
 which represent cache capacity and indicate the degree of overlap and
 isolation between classes. System cache resource is divided into numbers of
@@ -107,7 +107,7 @@ System CAT information such as maximum COS and CBM length can be obtained by:
 
 The simplest way to change a domain's CBM from its default is running:
 
-`xl psr-cat-cbm-set  [OPTIONS] <domid> <cbm>`
+`xl psr-cat-set  [OPTIONS] <domid> <cbm>`
 
 where cbm is a number to represent the corresponding cache subset can be used.
 A cbm is valid only when:
@@ -119,13 +119,19 @@ A cbm is valid only when:
 In a multi-socket system, the same cbm will be set on each socket by default.
 Per socket cbm can be specified with the `--socket SOCKET` option.
 
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
+
 Setting the CBM may not be successful if insufficient COS is available. In
 such case unused COS(es) may be freed by setting CBM of all related domains to
 its default value(all-ones).
 
 Per domain CBM settings can be shown by:
 
-`xl psr-cat-show`
+`xl psr-cat-show [OPTIONS] <domid>`
+
+In different systems, the different cache level is supported, e.g. L3 cache or
+L2 cache. Per cache level cbm can be specified with the `--level LEVEL` option.
 
 ## Code and Data Prioritization (CDP)
 
@@ -172,13 +178,13 @@ options is invalid.
 Example:
 
 Setting code CBM for a domain:
-`xl psr-cat-cbm-set -c <domid> <cbm>`
+`xl psr-cat-set -c <domid> <cbm>`
 
 Setting data CBM for a domain:
-`xl psr-cat-cbm-set -d <domid> <cbm>`
+`xl psr-cat-set -d <domid> <cbm>`
 
 Setting the same code and data CBM for a domain:
-`xl psr-cat-cbm-set <domid> <cbm>`
+`xl psr-cat-set <domid> <cbm>`
 
 ## Reference
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c
  2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
                   ` (24 preceding siblings ...)
  2017-03-16 11:08 ` [PATCH v9 25/25] docs: add L2 CAT description in docs Yi Sun
@ 2017-03-16 11:20 ` Jan Beulich
  2017-03-17  1:29   ` Yi Sun
  25 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> Acked and Reviewed list before V9:
> 
> a - Acked-by
> r - Reviewed-by
> 
>   r  patch 1  - docs: create Cache Allocation Technology (CAT) and Code and
>                 Data Prioritization (CDP) feature document
>   ar patch 2  - x86: refactor psr: remove L3 CAT/CDP codes.
>   r  patch 3  - x86: refactor psr: implement main data structures.
>   r  patch 6  - x86: refactor psr: L3 CAT: implement Domain init/free and
>                 schedule flows.
>   r  patch 7  - x86: refactor psr: L3 CAT: implement get hw info flow.
>   r  patch 8  - x86: refactor psr: L3 CAT: implement get value flow.

I didn't check whether you've kept them, but the changes you've
listed certainly are such that the last four of the above pretty likely
would need to have any prior tags dropped (in which case the
overview above is misleading).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c
  2017-03-16 11:20 ` [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Jan Beulich
@ 2017-03-17  1:29   ` Yi Sun
  2017-03-17  7:25     ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-17  1:29 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau

On 17-03-16 05:20:35, Jan Beulich wrote:
> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> > Acked and Reviewed list before V9:
> > 
> > a - Acked-by
> > r - Reviewed-by
> > 
> >   r  patch 1  - docs: create Cache Allocation Technology (CAT) and Code and
> >                 Data Prioritization (CDP) feature document
> >   ar patch 2  - x86: refactor psr: remove L3 CAT/CDP codes.
> >   r  patch 3  - x86: refactor psr: implement main data structures.
> >   r  patch 6  - x86: refactor psr: L3 CAT: implement Domain init/free and
> >                 schedule flows.
> >   r  patch 7  - x86: refactor psr: L3 CAT: implement get hw info flow.
> >   r  patch 8  - x86: refactor psr: L3 CAT: implement get value flow.
> 
> I didn't check whether you've kept them, but the changes you've
> listed certainly are such that the last four of the above pretty likely
> would need to have any prior tags dropped (in which case the
> overview above is misleading).
> 
Oh, sorry. I thought I should keep all ack/review history. But I am wrong.
For any reworked patch, I should remove a or r. But one question, if only
some minor changes added, shall I remove a or r? Thanks!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c
  2017-03-17  1:29   ` Yi Sun
@ 2017-03-17  7:25     ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-17  7:25 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 17.03.17 at 02:29, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-16 05:20:35, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
>> > Acked and Reviewed list before V9:
>> > 
>> > a - Acked-by
>> > r - Reviewed-by
>> > 
>> >   r  patch 1  - docs: create Cache Allocation Technology (CAT) and Code and
>> >                 Data Prioritization (CDP) feature document
>> >   ar patch 2  - x86: refactor psr: remove L3 CAT/CDP codes.
>> >   r  patch 3  - x86: refactor psr: implement main data structures.
>> >   r  patch 6  - x86: refactor psr: L3 CAT: implement Domain init/free and
>> >                 schedule flows.
>> >   r  patch 7  - x86: refactor psr: L3 CAT: implement get hw info flow.
>> >   r  patch 8  - x86: refactor psr: L3 CAT: implement get value flow.
>> 
>> I didn't check whether you've kept them, but the changes you've
>> listed certainly are such that the last four of the above pretty likely
>> would need to have any prior tags dropped (in which case the
>> overview above is misleading).
>> 
> Oh, sorry. I thought I should keep all ack/review history. But I am wrong.
> For any reworked patch, I should remove a or r. But one question, if only
> some minor changes added, shall I remove a or r? Thanks!

Cosmetic changes don't normally require dropping tags.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 03/25] x86: refactor psr: implement main data structures.
  2017-03-16 11:07 ` [PATCH v9 03/25] x86: refactor psr: implement main data structures Yi Sun
@ 2017-03-24 16:19   ` Jan Beulich
  2017-03-27  2:38     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-24 16:19 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> +enum psr_feat_type {
> +    PSR_SOCKET_L3_CAT = 0,

Pointless " = 0".

> +    PSR_SOCKET_L3_CDP,
> +    PSR_SOCKET_L2_CAT,
> +    PSR_SOCKET_MAX_FEAT,
> +};
> +
> +/* CAT/CDP HW info data structure. */
> +struct psr_cat_hw_info {
> +    unsigned int cbm_len;
> +    unsigned int cos_max;

So you have this field, and ...

> +};
> +
> +/*
> + * This structure represents one feature.
> + * feat_ops    - Feature operation callback functions.
> + * info        - Feature HW info.
> + * cos_reg_val - Array to store the values of COS registers. One entry stores
> + *               the value of one COS register.
> + *               For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
> + *               For CDP, two entries correspond to one COS_ID. E.g.
> + *               COS_ID=0 corresponds to cos_reg_val[0] (Data) and
> + *               cos_reg_val[1] (Code).
> + * cos_num     - COS registers number that feature uses in one time access.
> + */
> +struct feat_node {
> +    /*
> +     * This structure defines feature operation callback functions. Every feature
> +     * enabled MUST implement such callback functions and register them to ops.
> +     *
> +     * Feature specific behaviors will be encapsulated into these callback
> +     * functions. Then, the main flows will not be changed when introducing a new
> +     * feature.
> +     */
> +    struct feat_ops {
> +        /* get_cos_max is used to get feature's cos_max. */
> +        unsigned int (*get_cos_max)(const struct feat_node *feat);

... you have this op, suggesting that you expect all features
to have a cos_max. Why don't you then store the value in a
field which is not per-feature, just like ...

> +    } ops;
> +
> +    /* Encapsulate feature specific HW info here. */
> +    union {
> +        struct psr_cat_hw_info cat_info;
> +    } info;
> +
> +    uint32_t cos_reg_val[MAX_COS_REG_CNT];
> +    unsigned int cos_num;

... this. I'm pretty sure that during v8 review I did say that
this approach should be extended to all pieces of information
where it can be applied.

Also please place the array last, so that accesses to most/all
other fields have a better chance of working with 8-bit
displacements.

Furthermore, didn't we settle on ops being a pointer to a const
struct, initialized by taking the address of a static const object?
There is no reason to duplicate all the pointers in every node.

> +struct psr_socket_info {
> +    /*
> +     * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
> +     * psr_feat_type' means the bit position.
> +     * bit 0:   L3 CAT
> +     * bit 1:   L3 CDP
> +     * bit 2:   L2 CAT
> +     */
> +    unsigned int feat_mask;

Comment or not I don't understand what use this mask is, and
this is again something which I'm pretty sure I've mentioned in
v8 review, when the switch to ...

> +    struct feat_node *features[PSR_SOCKET_MAX_FEAT];
> +    unsigned int cos_ref[MAX_COS_REG_CNT];

... this array was suggested by Roger. The pointers in the
array being non-NULL can - afaict - easily fulfill the role of
the mask bits, so the latter are redundant.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 04/25] x86: move cpuid_count_leaf from cpuid.c to processor.h.
  2017-03-16 11:07 ` [PATCH v9 04/25] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
@ 2017-03-24 16:22   ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-24 16:22 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> This patch moves 'cpuid_count_leaf' from cpuid.c to processor.h to
> make it available to external codes.
> 
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>

Acked-by: Jan Beulich <jbeulich@suse.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow.
  2017-03-16 11:07 ` [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow Yi Sun
@ 2017-03-24 16:52   ` Jan Beulich
  2017-03-27  4:41     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-24 16:52 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -18,6 +18,7 @@
>  #include <xen/init.h>
>  #include <xen/sched.h>
>  #include <asm/psr.h>
> +#include <asm/x86_emulate.h>

I'm pretty sure you don't need this. If anything you need
processor.h (as that's where the previous patch put
cpuid_count_leaf()), but I'm rather convinced that the header was
already included indirectly at this point.

> @@ -46,6 +50,9 @@
>   */
>  #define MAX_COS_REG_CNT  128
>  
> +/* CAT features use 1 COS register in one access. */
> +#define CAT_COS_NUM      1

With it being stored into the feature node now I don't see why you
need this constant anymore. And indeed it's being used exactly
once.

> @@ -126,11 +133,110 @@ struct psr_assoc {
>  
>  struct psr_cmt *__read_mostly psr_cmt;
>  
> +static struct psr_socket_info *__read_mostly socket_info;
> +
>  static unsigned int opt_psr;
>  static unsigned int __initdata opt_rmid_max = 255;
> +static unsigned int __read_mostly opt_cos_max = MAX_COS_REG_CNT;
>  static uint64_t rmid_mask;
>  static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
>  
> +/*
> + * Declare global feature node for every feature to facilitate the feature
> + * array creation. It is used to transiently store a spare node.
> + */
> +static struct feat_node *feat_l3_cat;
> +
> +/* Common functions */
> +#define cat_default_val(len)                 \
> +            ( (uint32_t)((1ul << len) - 1) )

Pretty odd construct, which I guess you use to avoid the
undefined-ness when len == 32. But this can be had without
extra cast, assuming len is in [1,32]:

#define cat_default_val(len) (0xffffffff >> (32 - (len)))

Also - stray blanks and missing parentheses around the use of macro
parameter.

> +/*
> + * Use this function to check if any allocation feature has been enabled
> + * in cmdline.
> + */
> +static bool psr_alloc_feat_enabled(void)
> +{
> +    return ((!socket_info) ? false : true );

Stray parentheses (all of them actually) and blank. Even more, why
not simply

    return socket_info;

?

> +static void free_feature(struct psr_socket_info *info)
> +{
> +    unsigned int i;
> +
> +    if ( !info )
> +        return;
> +
> +    /*
> +     * Free resources of features. The global feature object, e.g. feat_l3_cat,
> +     * may not be freed here if it is not added into array. It is simply being
> +     * kept until the next CPU online attempt.
> +     */
> +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> +    {
> +        if ( !info->features[i] )
> +            continue;
> +
> +        xfree(info->features[i]);
> +        info->features[i] = NULL;
> +        __clear_bit(i, &info->feat_mask);
> +    }
> +}

What the function does suggests its name ought to be
free_features().

> +static void cat_init_feature(struct cpuid_leaf regs,

I'm sure I've asked before to not pass structures by value. And
once you switch to a pointer, please don't forget to constify it.

> +                             struct feat_node *feat,
> +                             struct psr_socket_info *info,
> +                             enum psr_feat_type type)
> +{
> +    unsigned int socket, i;
> +    struct psr_cat_hw_info cat = { };
> +    uint64_t val;
> +
> +    /* No valid value so do not enable feature. */
> +    if ( !regs.a || !regs.d )
> +        return;
> +
> +    cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
> +    cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
> +
> +    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> +    feat->cos_reg_val[0] = cat_default_val(cat.cbm_len);
> +    /*
> +     * To handle cpu offline and then online case, we need read MSRs back to
> +     * save values into cos_reg_val array.
> +     */
> +    for ( i = 1; i <= cat.cos_max; i++ )
> +    {
> +        rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
> +        feat->cos_reg_val[i] = (uint32_t)val;
> +    }

You mention this in the changes done, but I don't understand why
you do this. What meaning to these values have to you? If you
want hardware and cached values to match up, the much more
conventional way of enforcing this would be to write the values
you actually want (normally all zero).

> +    feat->info.cat_info = cat;
> +    feat->cos_num = CAT_COS_NUM;
> +
> +    /* Add this feature into array. */
> +    info->features[type] = feat;
> +
> +    ASSERT(!test_bit(type, &info->feat_mask));
> +    __set_bit(type, &info->feat_mask);

    if ( __test_and_set_bit(type, &info->feat_mask) )
        ASSERT_UNREACHABLE();

> +    socket = cpu_to_socket(smp_processor_id());
> +    if ( !opt_cpu_info )
> +        return;
> +
> +    printk(XENLOG_INFO "%s CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
> +           ((type == PSR_SOCKET_L3_CAT) ? "L3" : "L2"),
> +           socket, feat->info.cat_info.cos_max,
> +           feat->info.cat_info.cbm_len);
> +
> +    return;

Pointless statement at end of function.

> +/* L3 CAT ops */
> +static const struct feat_ops l3_cat_ops = {
> +};

Leaving an already declared function pointer as NULL? Please don't.

>  static void psr_cpu_init(void)
>  {
> +    struct psr_socket_info *info;
> +    unsigned int socket, i;
> +    unsigned int cpu = smp_processor_id();
> +    struct feat_node *feat;
> +    struct cpuid_leaf regs;
> +
> +    if ( !psr_alloc_feat_enabled() || !boot_cpu_has(X86_FEATURE_PQE) )
> +        goto assoc_init;
> +
> +    if ( boot_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
> +    {
> +        setup_clear_cpu_cap(X86_FEATURE_PQE);
> +        goto assoc_init;
> +    }
> +
> +    socket = cpu_to_socket(cpu);
> +    info = socket_info + socket;
> +    if ( info->feat_mask )
> +        goto assoc_init;
> +
> +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> +        info->features[i] = NULL;

You've xzalloc()ed this memory - why do you need this loop?

> +    spin_lock_init(&info->ref_lock);
> +
> +    cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, &regs);
> +    if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> +    {
> +        cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, &regs);
> +
> +        feat = feat_l3_cat;
> +        feat_l3_cat = NULL;
> +        feat->ops = l3_cat_ops;
> +
> +        cat_init_feature(regs, feat, info, PSR_SOCKET_L3_CAT);
> +    }
> +
> +assoc_init:

Labels indented by at least on space please.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 03/25] x86: refactor psr: implement main data structures.
  2017-03-24 16:19   ` Jan Beulich
@ 2017-03-27  2:38     ` Yi Sun
  2017-03-27  6:20       ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-27  2:38 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-24 10:19:30, Jan Beulich wrote:
> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> > +enum psr_feat_type {
> > +    PSR_SOCKET_L3_CAT = 0,
> 
> Pointless " = 0".
> 
Ok, will remove it.

> > +    PSR_SOCKET_L3_CDP,
> > +    PSR_SOCKET_L2_CAT,
> > +    PSR_SOCKET_MAX_FEAT,
> > +};
> > +
> > +/* CAT/CDP HW info data structure. */
> > +struct psr_cat_hw_info {
> > +    unsigned int cbm_len;
> > +    unsigned int cos_max;
> 
> So you have this field, and ...
> 
> > +};
> > +
> > +/*
> > + * This structure represents one feature.
> > + * feat_ops    - Feature operation callback functions.
> > + * info        - Feature HW info.
> > + * cos_reg_val - Array to store the values of COS registers. One entry stores
> > + *               the value of one COS register.
> > + *               For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
> > + *               For CDP, two entries correspond to one COS_ID. E.g.
> > + *               COS_ID=0 corresponds to cos_reg_val[0] (Data) and
> > + *               cos_reg_val[1] (Code).
> > + * cos_num     - COS registers number that feature uses in one time access.
> > + */
> > +struct feat_node {
> > +    /*
> > +     * This structure defines feature operation callback functions. Every feature
> > +     * enabled MUST implement such callback functions and register them to ops.
> > +     *
> > +     * Feature specific behaviors will be encapsulated into these callback
> > +     * functions. Then, the main flows will not be changed when introducing a new
> > +     * feature.
> > +     */
> > +    struct feat_ops {
> > +        /* get_cos_max is used to get feature's cos_max. */
> > +        unsigned int (*get_cos_max)(const struct feat_node *feat);
> 
> ... you have this op, suggesting that you expect all features
> to have a cos_max. Why don't you then store the value in a
> field which is not per-feature, just like ...
> 
> > +    } ops;
> > +
> > +    /* Encapsulate feature specific HW info here. */
> > +    union {
> > +        struct psr_cat_hw_info cat_info;
> > +    } info;
> > +
> > +    uint32_t cos_reg_val[MAX_COS_REG_CNT];
> > +    unsigned int cos_num;
> 
> ... this. I'm pretty sure that during v8 review I did say that
> this approach should be extended to all pieces of information
> where it can be applied.
> 
I thought this when implementing v9. As cos_max is part of feature HW info, I
thought it would be better to keep it in hw_info structure. Different features
may have different hw_info, so the callback function is needed to get cos_max.
Of course, we can keep a copy in feat_node but it is redundant. How do you
think?

> Also please place the array last, so that accesses to most/all
> other fields have a better chance of working with 8-bit
> displacements.
> 
Sure. Thanks!

> Furthermore, didn't we settle on ops being a pointer to a const
> struct, initialized by taking the address of a static const object?
> There is no reason to duplicate all the pointers in every node.
> 
Will correct this.

> > +struct psr_socket_info {
> > +    /*
> > +     * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
> > +     * psr_feat_type' means the bit position.
> > +     * bit 0:   L3 CAT
> > +     * bit 1:   L3 CDP
> > +     * bit 2:   L2 CAT
> > +     */
> > +    unsigned int feat_mask;
> 
> Comment or not I don't understand what use this mask is, and
> this is again something which I'm pretty sure I've mentioned in
> v8 review, when the switch to ...
> 
> > +    struct feat_node *features[PSR_SOCKET_MAX_FEAT];
> > +    unsigned int cos_ref[MAX_COS_REG_CNT];
> 
> ... this array was suggested by Roger. The pointers in the
> array being non-NULL can - afaict - easily fulfill the role of
> the mask bits, so the latter are redundant.
> 
I thought 'feat_mask' can facilitate things handling. If we check feature array
to know if any feature has been initialized, we have to iterate the array. But
it is simple to check 'feat_mask'.
1. In 'psr_cpu_init', we can use it to check if initialization on the socket
   has been done.
    if ( info->feat_mask )
        goto assoc_init;
2. Same purpose in 'psr_assoc_init'.
    if ( info->feat_mask )
        psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
                         PSR_ASSOC_REG_SHIFT;
3. Same purpose in 'get_socket_info'.
    if ( !socket_info[socket].feat_mask )
 
What is your advice? Thanks!

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow.
  2017-03-24 16:52   ` Jan Beulich
@ 2017-03-27  4:41     ` Yi Sun
  2017-03-27  6:34       ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-27  4:41 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-24 10:52:34, Jan Beulich wrote:
> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -18,6 +18,7 @@
> >  #include <xen/init.h>
> >  #include <xen/sched.h>
> >  #include <asm/psr.h>
> > +#include <asm/x86_emulate.h>
> 
> I'm pretty sure you don't need this. If anything you need
> processor.h (as that's where the previous patch put
> cpuid_count_leaf()), but I'm rather convinced that the header was
> already included indirectly at this point.
> 
Yes, you are right. It is indirectly included through 'sched.h'.

> > @@ -46,6 +50,9 @@
> >   */
> >  #define MAX_COS_REG_CNT  128
> >  
> > +/* CAT features use 1 COS register in one access. */
> > +#define CAT_COS_NUM      1
> 
> With it being stored into the feature node now I don't see why you
> need this constant anymore. And indeed it's being used exactly
> once.
> 
I remember somebody suggested me not to use constant but should define a
macro. As it is only used once, I will remove this and 'CDP_COS_NUM' in
later patch.

> > +/*
> > + * Declare global feature node for every feature to facilitate the feature
> > + * array creation. It is used to transiently store a spare node.
> > + */
> > +static struct feat_node *feat_l3_cat;
> > +
> > +/* Common functions */
> > +#define cat_default_val(len)                 \
> > +            ( (uint32_t)((1ul << len) - 1) )
> 
> Pretty odd construct, which I guess you use to avoid the
> undefined-ness when len == 32. But this can be had without
> extra cast, assuming len is in [1,32]:
> 
> #define cat_default_val(len) (0xffffffff >> (32 - (len)))
> 
> Also - stray blanks and missing parentheses around the use of macro
> parameter.
> 
Thanks for the suggestion! Will change it.

> > +/*
> > + * Use this function to check if any allocation feature has been enabled
> > + * in cmdline.
> > + */
> > +static bool psr_alloc_feat_enabled(void)
> > +{
> > +    return ((!socket_info) ? false : true );
> 
> Stray parentheses (all of them actually) and blank. Even more, why
> not simply
> 
>     return socket_info;
> 
> ?
> 
How about 'return !!socket_info'?

> > +static void free_feature(struct psr_socket_info *info)
> > +{
> > +    unsigned int i;
> > +
> > +    if ( !info )
> > +        return;
> > +
> > +    /*
> > +     * Free resources of features. The global feature object, e.g. feat_l3_cat,
> > +     * may not be freed here if it is not added into array. It is simply being
> > +     * kept until the next CPU online attempt.
> > +     */
> > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> > +    {
> > +        if ( !info->features[i] )
> > +            continue;
> > +
> > +        xfree(info->features[i]);
> > +        info->features[i] = NULL;
> > +        __clear_bit(i, &info->feat_mask);
> > +    }
> > +}
> 
> What the function does suggests its name ought to be
> free_features().
> 
Ok, will modify it.

> > +static void cat_init_feature(struct cpuid_leaf regs,
> 
> I'm sure I've asked before to not pass structures by value. And
> once you switch to a pointer, please don't forget to constify it.
> 
Will correct this, thanks!

> > +                             struct feat_node *feat,
> > +                             struct psr_socket_info *info,
> > +                             enum psr_feat_type type)
> > +{
> > +    unsigned int socket, i;
> > +    struct psr_cat_hw_info cat = { };
> > +    uint64_t val;
> > +
> > +    /* No valid value so do not enable feature. */
> > +    if ( !regs.a || !regs.d )
> > +        return;
> > +
> > +    cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
> > +    cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
> > +
> > +    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> > +    feat->cos_reg_val[0] = cat_default_val(cat.cbm_len);
> > +    /*
> > +     * To handle cpu offline and then online case, we need read MSRs back to
> > +     * save values into cos_reg_val array.
> > +     */
> > +    for ( i = 1; i <= cat.cos_max; i++ )
> > +    {
> > +        rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
> > +        feat->cos_reg_val[i] = (uint32_t)val;
> > +    }
> 
> You mention this in the changes done, but I don't understand why
> you do this. What meaning to these values have to you? If you
> want hardware and cached values to match up, the much more
> conventional way of enforcing this would be to write the values
> you actually want (normally all zero).
> 
When all cpus on a socket are offline, the free_feature() is called to free
features resources so that the values saved in cos_reg_val[] are lost. When the
socket is online again, features are allocated again so that cos_reg_val[]
members are all initialized to 0. Only is cos_reg_val[0] initialized to default
value in this function in old codes.

But domain is still alive so that its cos id on the socket is kept. The
corresponding MSR value is kept too per test. To make cos_reg_val[] values be
same as HW to not to mislead user, we should read back the valid values on HW
into cos_reg_val[].

> > +    feat->info.cat_info = cat;
> > +    feat->cos_num = CAT_COS_NUM;
> > +
> > +    /* Add this feature into array. */
> > +    info->features[type] = feat;
> > +
> > +    ASSERT(!test_bit(type, &info->feat_mask));
> > +    __set_bit(type, &info->feat_mask);
> 
>     if ( __test_and_set_bit(type, &info->feat_mask) )
>         ASSERT_UNREACHABLE();
> 
Thanks!

> > +    socket = cpu_to_socket(smp_processor_id());
> > +    if ( !opt_cpu_info )
> > +        return;
> > +
> > +    printk(XENLOG_INFO "%s CAT: enabled on socket %u, cos_max:%u, cbm_len:%u\n",
> > +           ((type == PSR_SOCKET_L3_CAT) ? "L3" : "L2"),
> > +           socket, feat->info.cat_info.cos_max,
> > +           feat->info.cat_info.cbm_len);
> > +
> > +    return;
> 
> Pointless statement at end of function.
> 
Will remove it.

> > +/* L3 CAT ops */
> > +static const struct feat_ops l3_cat_ops = {
> > +};
> 
> Leaving an already declared function pointer as NULL? Please don't.
> 
Ok, will consider to move it and below code into later patch.
    feat->ops = l3_cat_ops;

> >  static void psr_cpu_init(void)
> >  {
> > +    struct psr_socket_info *info;
> > +    unsigned int socket, i;
> > +    unsigned int cpu = smp_processor_id();
> > +    struct feat_node *feat;
> > +    struct cpuid_leaf regs;
> > +
> > +    if ( !psr_alloc_feat_enabled() || !boot_cpu_has(X86_FEATURE_PQE) )
> > +        goto assoc_init;
> > +
> > +    if ( boot_cpu_data.cpuid_level < PSR_CPUID_LEVEL_CAT )
> > +    {
> > +        setup_clear_cpu_cap(X86_FEATURE_PQE);
> > +        goto assoc_init;
> > +    }
> > +
> > +    socket = cpu_to_socket(cpu);
> > +    info = socket_info + socket;
> > +    if ( info->feat_mask )
> > +        goto assoc_init;
> > +
> > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> > +        info->features[i] = NULL;
> 
> You've xzalloc()ed this memory - why do you need this loop?
> 
Hmm, no need indeed. Will remove this.

> > +    spin_lock_init(&info->ref_lock);
> > +
> > +    cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 0, &regs);
> > +    if ( regs.b & PSR_RESOURCE_TYPE_L3 )
> > +    {
> > +        cpuid_count_leaf(PSR_CPUID_LEVEL_CAT, 1, &regs);
> > +
> > +        feat = feat_l3_cat;
> > +        feat_l3_cat = NULL;
> > +        feat->ops = l3_cat_ops;
> > +
> > +        cat_init_feature(regs, feat, info, PSR_SOCKET_L3_CAT);
> > +    }
> > +
> > +assoc_init:
> 
> Labels indented by at least on space please.
> 
Got it, thanks!

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 03/25] x86: refactor psr: implement main data structures.
  2017-03-27  2:38     ` Yi Sun
@ 2017-03-27  6:20       ` Jan Beulich
  2017-03-27  7:12         ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27  6:20 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 27.03.17 at 04:38, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-24 10:19:30, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
>> > +struct psr_cat_hw_info {
>> > +    unsigned int cbm_len;
>> > +    unsigned int cos_max;
>> 
>> So you have this field, and ...
>> 
>> > +};
>> > +
>> > +/*
>> > + * This structure represents one feature.
>> > + * feat_ops    - Feature operation callback functions.
>> > + * info        - Feature HW info.
>> > + * cos_reg_val - Array to store the values of COS registers. One entry stores
>> > + *               the value of one COS register.
>> > + *               For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
>> > + *               For CDP, two entries correspond to one COS_ID. E.g.
>> > + *               COS_ID=0 corresponds to cos_reg_val[0] (Data) and
>> > + *               cos_reg_val[1] (Code).
>> > + * cos_num     - COS registers number that feature uses in one time access.
>> > + */
>> > +struct feat_node {
>> > +    /*
>> > +     * This structure defines feature operation callback functions. Every feature
>> > +     * enabled MUST implement such callback functions and register them to ops.
>> > +     *
>> > +     * Feature specific behaviors will be encapsulated into these callback
>> > +     * functions. Then, the main flows will not be changed when introducing a new
>> > +     * feature.
>> > +     */
>> > +    struct feat_ops {
>> > +        /* get_cos_max is used to get feature's cos_max. */
>> > +        unsigned int (*get_cos_max)(const struct feat_node *feat);
>> 
>> ... you have this op, suggesting that you expect all features
>> to have a cos_max. Why don't you then store the value in a
>> field which is not per-feature, just like ...
>> 
>> > +    } ops;
>> > +
>> > +    /* Encapsulate feature specific HW info here. */
>> > +    union {
>> > +        struct psr_cat_hw_info cat_info;
>> > +    } info;
>> > +
>> > +    uint32_t cos_reg_val[MAX_COS_REG_CNT];
>> > +    unsigned int cos_num;
>> 
>> ... this. I'm pretty sure that during v8 review I did say that
>> this approach should be extended to all pieces of information
>> where it can be applied.
>> 
> I thought this when implementing v9. As cos_max is part of feature HW info, I
> thought it would be better to keep it in hw_info structure. Different features
> may have different hw_info, so the callback function is needed to get cos_max.
> Of course, we can keep a copy in feat_node but it is redundant. How do you
> think?

I don't follow - as long as you have a universal get_cos_max()
accesses, and as long as what that function returns depends
only on invariable things like CPUID output, I don't see why
this needs to be a function instead of a data field. If some
(perhaps future, theoretical) feature didn't want/need a
get_cos_max() function, the presence of that hook would
become questionable, yet it could surely become an optional
hook. However, the hook being optional could as well be
represented by the data field getting assigned a value of 0.

Bottom line: Data which can be calculated at initialization
time should be stored in a date object, rather than re-
calculating it over and over.

>> > +struct psr_socket_info {
>> > +    /*
>> > +     * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
>> > +     * psr_feat_type' means the bit position.
>> > +     * bit 0:   L3 CAT
>> > +     * bit 1:   L3 CDP
>> > +     * bit 2:   L2 CAT
>> > +     */
>> > +    unsigned int feat_mask;
>> 
>> Comment or not I don't understand what use this mask is, and
>> this is again something which I'm pretty sure I've mentioned in
>> v8 review, when the switch to ...
>> 
>> > +    struct feat_node *features[PSR_SOCKET_MAX_FEAT];
>> > +    unsigned int cos_ref[MAX_COS_REG_CNT];
>> 
>> ... this array was suggested by Roger. The pointers in the
>> array being non-NULL can - afaict - easily fulfill the role of
>> the mask bits, so the latter are redundant.
>> 
> I thought 'feat_mask' can facilitate things handling. If we check feature 
> array
> to know if any feature has been initialized, we have to iterate the array. 
> But
> it is simple to check 'feat_mask'.
> 1. In 'psr_cpu_init', we can use it to check if initialization on the socket
>    has been done.
>     if ( info->feat_mask )
>         goto assoc_init;
> 2. Same purpose in 'psr_assoc_init'.
>     if ( info->feat_mask )
>         psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
>                          PSR_ASSOC_REG_SHIFT;
> 3. Same purpose in 'get_socket_info'.
>     if ( !socket_info[socket].feat_mask )
>  
> What is your advice? Thanks!

My advice is: Avoid redundant data as much as possible. Any such
instance poses the risk of the two pieces of information going out
of sync. (That isn't to say that there aren't cases where redundancy
is almost unavoidable, e.g. in order to not overly complicate code,
but that's pretty clearly not the case here).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow.
  2017-03-27  4:41     ` Yi Sun
@ 2017-03-27  6:34       ` Jan Beulich
  2017-03-27  8:16         ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27  6:34 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 27.03.17 at 06:41, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-24 10:52:34, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
>> > @@ -46,6 +50,9 @@
>> >   */
>> >  #define MAX_COS_REG_CNT  128
>> >  
>> > +/* CAT features use 1 COS register in one access. */
>> > +#define CAT_COS_NUM      1
>> 
>> With it being stored into the feature node now I don't see why you
>> need this constant anymore. And indeed it's being used exactly
>> once.
>> 
> I remember somebody suggested me not to use constant but should define a
> macro. As it is only used once, I will remove this and 'CDP_COS_NUM' in
> later patch.

It may well have been me, back when this was used in multiple places.

>> > +/*
>> > + * Use this function to check if any allocation feature has been enabled
>> > + * in cmdline.
>> > + */
>> > +static bool psr_alloc_feat_enabled(void)
>> > +{
>> > +    return ((!socket_info) ? false : true );
>> 
>> Stray parentheses (all of them actually) and blank. Even more, why
>> not simply
>> 
>>     return socket_info;
>> 
>> ?
>> 
> How about 'return !!socket_info'?

And what would the !! be good for? Back when we were still using
bool_t that would have been a requirement (the code wouldn't
even have built without afaict), but now that we use bool I don't
see the point (other that cluttering code). In fact I consider the
presence of the function questionable as a whole, unless later
patches add to it.

>> > +                             struct feat_node *feat,
>> > +                             struct psr_socket_info *info,
>> > +                             enum psr_feat_type type)
>> > +{
>> > +    unsigned int socket, i;
>> > +    struct psr_cat_hw_info cat = { };
>> > +    uint64_t val;
>> > +
>> > +    /* No valid value so do not enable feature. */
>> > +    if ( !regs.a || !regs.d )
>> > +        return;
>> > +
>> > +    cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
>> > +    cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
>> > +
>> > +    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
>> > +    feat->cos_reg_val[0] = cat_default_val(cat.cbm_len);
>> > +    /*
>> > +     * To handle cpu offline and then online case, we need read MSRs back to
>> > +     * save values into cos_reg_val array.
>> > +     */
>> > +    for ( i = 1; i <= cat.cos_max; i++ )
>> > +    {
>> > +        rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
>> > +        feat->cos_reg_val[i] = (uint32_t)val;
>> > +    }
>> 
>> You mention this in the changes done, but I don't understand why
>> you do this. What meaning to these values have to you? If you
>> want hardware and cached values to match up, the much more
>> conventional way of enforcing this would be to write the values
>> you actually want (normally all zero).
>> 
> When all cpus on a socket are offline, the free_feature() is called to free
> features resources so that the values saved in cos_reg_val[] are lost. When the
> socket is online again, features are allocated again so that cos_reg_val[]
> members are all initialized to 0. Only is cos_reg_val[0] initialized to default
> value in this function in old codes.
> 
> But domain is still alive so that its cos id on the socket is kept. The
> corresponding MSR value is kept too per test. To make cos_reg_val[] values be
> same as HW to not to mislead user, we should read back the valid values on HW
> into cos_reg_val[].

Okay, I understand the background, but I don't view this solution
as viable: Once the last core on a socket goes offline, all
references to it should be cleaned up. After all what will be
brought back online may be a different physical CPU altogether;
you can't assume MSR values to have survived even if it is the
same CPU which comes back online, as it may have undergone
a reset cycle, or BIOS/SMM may have played with the MSRs.
That's even a possibility for a single core coming back online, so
you have to reload MSRs explicitly anyway if implicit reloading
(i.e. once vCPU-s get scheduled onto it) doesn't suffice.

>> > +/* L3 CAT ops */
>> > +static const struct feat_ops l3_cat_ops = {
>> > +};
>> 
>> Leaving an already declared function pointer as NULL? Please don't.
>> 
> Ok, will consider to move it and below code into later patch.
>     feat->ops = l3_cat_ops;

I don't mind the empty structure instance above, as long as the
structure doesn't have any function pointer members yet (data
members are almost always fine).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 03/25] x86: refactor psr: implement main data structures.
  2017-03-27  6:20       ` Jan Beulich
@ 2017-03-27  7:12         ` Yi Sun
  2017-03-27  7:37           ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-27  7:12 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 00:20:58, Jan Beulich wrote:
> >>> On 27.03.17 at 04:38, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-24 10:19:30, Jan Beulich wrote:
> >> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> >> > +struct psr_cat_hw_info {
> >> > +    unsigned int cbm_len;
> >> > +    unsigned int cos_max;
> >> 
> >> So you have this field, and ...
> >> 
> >> > +};
> >> > +
> >> > +/*
> >> > + * This structure represents one feature.
> >> > + * feat_ops    - Feature operation callback functions.
> >> > + * info        - Feature HW info.
> >> > + * cos_reg_val - Array to store the values of COS registers. One entry stores
> >> > + *               the value of one COS register.
> >> > + *               For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
> >> > + *               For CDP, two entries correspond to one COS_ID. E.g.
> >> > + *               COS_ID=0 corresponds to cos_reg_val[0] (Data) and
> >> > + *               cos_reg_val[1] (Code).
> >> > + * cos_num     - COS registers number that feature uses in one time access.
> >> > + */
> >> > +struct feat_node {
> >> > +    /*
> >> > +     * This structure defines feature operation callback functions. Every feature
> >> > +     * enabled MUST implement such callback functions and register them to ops.
> >> > +     *
> >> > +     * Feature specific behaviors will be encapsulated into these callback
> >> > +     * functions. Then, the main flows will not be changed when introducing a new
> >> > +     * feature.
> >> > +     */
> >> > +    struct feat_ops {
> >> > +        /* get_cos_max is used to get feature's cos_max. */
> >> > +        unsigned int (*get_cos_max)(const struct feat_node *feat);
> >> 
> >> ... you have this op, suggesting that you expect all features
> >> to have a cos_max. Why don't you then store the value in a
> >> field which is not per-feature, just like ...
> >> 
> >> > +    } ops;
> >> > +
> >> > +    /* Encapsulate feature specific HW info here. */
> >> > +    union {
> >> > +        struct psr_cat_hw_info cat_info;
> >> > +    } info;
> >> > +
> >> > +    uint32_t cos_reg_val[MAX_COS_REG_CNT];
> >> > +    unsigned int cos_num;
> >> 
> >> ... this. I'm pretty sure that during v8 review I did say that
> >> this approach should be extended to all pieces of information
> >> where it can be applied.
> >> 
> > I thought this when implementing v9. As cos_max is part of feature HW info, I
> > thought it would be better to keep it in hw_info structure. Different features
> > may have different hw_info, so the callback function is needed to get cos_max.
> > Of course, we can keep a copy in feat_node but it is redundant. How do you
> > think?
> 
> I don't follow - as long as you have a universal get_cos_max()
> accesses, and as long as what that function returns depends
> only on invariable things like CPUID output, I don't see why
> this needs to be a function instead of a data field. If some
> (perhaps future, theoretical) feature didn't want/need a
> get_cos_max() function, the presence of that hook would
> become questionable, yet it could surely become an optional
> hook. However, the hook being optional could as well be
> represented by the data field getting assigned a value of 0.
> 
> Bottom line: Data which can be calculated at initialization
> time should be stored in a date object, rather than re-
> calculating it over and over.
> 
The purpose to use the function is just not to define a redundant member
in 'struct feat_node'.

The cos_max is got in cat_init_feature in patch 5 and kept in the feature's
hw_info. The 'get_cos_max' only returns DIFFERENT features' cos_max without
recalculation. E.g:

CAT/CDP:
static unsigned int cat_get_cos_max(const struct feat_node *feat)
{
    return feat->info.cat_info.cos_max;
}

MBA:
static unsigned int mba_get_cos_max(const struct feat_node *feat)
{
    return feat->info.mba_info.cos_max;
}

But I think it is ok to add a new member in 'struct feat_node' to keep
cos_max for the feature.

What do you prefer? Thanks!

> >> > +struct psr_socket_info {
> >> > +    /*
> >> > +     * It maps to values defined in 'enum psr_feat_type' below. Value in 'enum
> >> > +     * psr_feat_type' means the bit position.
> >> > +     * bit 0:   L3 CAT
> >> > +     * bit 1:   L3 CDP
> >> > +     * bit 2:   L2 CAT
> >> > +     */
> >> > +    unsigned int feat_mask;
> >> 
> >> Comment or not I don't understand what use this mask is, and
> >> this is again something which I'm pretty sure I've mentioned in
> >> v8 review, when the switch to ...
> >> 
> >> > +    struct feat_node *features[PSR_SOCKET_MAX_FEAT];
> >> > +    unsigned int cos_ref[MAX_COS_REG_CNT];
> >> 
> >> ... this array was suggested by Roger. The pointers in the
> >> array being non-NULL can - afaict - easily fulfill the role of
> >> the mask bits, so the latter are redundant.
> >> 
> > I thought 'feat_mask' can facilitate things handling. If we check feature 
> > array
> > to know if any feature has been initialized, we have to iterate the array. 
> > But
> > it is simple to check 'feat_mask'.
> > 1. In 'psr_cpu_init', we can use it to check if initialization on the socket
> >    has been done.
> >     if ( info->feat_mask )
> >         goto assoc_init;
> > 2. Same purpose in 'psr_assoc_init'.
> >     if ( info->feat_mask )
> >         psra->cos_mask = ((1ull << get_count_order(cos_max)) - 1) <<
> >                          PSR_ASSOC_REG_SHIFT;
> > 3. Same purpose in 'get_socket_info'.
> >     if ( !socket_info[socket].feat_mask )
> >  
> > What is your advice? Thanks!
> 
> My advice is: Avoid redundant data as much as possible. Any such
> instance poses the risk of the two pieces of information going out
> of sync. (That isn't to say that there aren't cases where redundancy
> is almost unavoidable, e.g. in order to not overly complicate code,
> but that's pretty clearly not the case here).
> 
If so, I think I should define a function to iterate the function array
to return TRUE if any feature has been set into the array. Then, use
this function to replace above checking points.

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 03/25] x86: refactor psr: implement main data structures.
  2017-03-27  7:12         ` Yi Sun
@ 2017-03-27  7:37           ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-27  7:37 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 27.03.17 at 09:12, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 00:20:58, Jan Beulich wrote:
>> >>> On 27.03.17 at 04:38, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-24 10:19:30, Jan Beulich wrote:
>> >> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
>> >> > +struct psr_cat_hw_info {
>> >> > +    unsigned int cbm_len;
>> >> > +    unsigned int cos_max;
>> >> 
>> >> So you have this field, and ...
>> >> 
>> >> > +};
>> >> > +
>> >> > +/*
>> >> > + * This structure represents one feature.
>> >> > + * feat_ops    - Feature operation callback functions.
>> >> > + * info        - Feature HW info.
>> >> > + * cos_reg_val - Array to store the values of COS registers. One entry stores
>> >> > + *               the value of one COS register.
>> >> > + *               For L3 CAT and L2 CAT, one entry corresponds to one COS_ID.
>> >> > + *               For CDP, two entries correspond to one COS_ID. E.g.
>> >> > + *               COS_ID=0 corresponds to cos_reg_val[0] (Data) and
>> >> > + *               cos_reg_val[1] (Code).
>> >> > + * cos_num     - COS registers number that feature uses in one time access.
>> >> > + */
>> >> > +struct feat_node {
>> >> > +    /*
>> >> > +     * This structure defines feature operation callback functions. Every feature
>> >> > +     * enabled MUST implement such callback functions and register them to ops.
>> >> > +     *
>> >> > +     * Feature specific behaviors will be encapsulated into these callback
>> >> > +     * functions. Then, the main flows will not be changed when introducing a new
>> >> > +     * feature.
>> >> > +     */
>> >> > +    struct feat_ops {
>> >> > +        /* get_cos_max is used to get feature's cos_max. */
>> >> > +        unsigned int (*get_cos_max)(const struct feat_node *feat);
>> >> 
>> >> ... you have this op, suggesting that you expect all features
>> >> to have a cos_max. Why don't you then store the value in a
>> >> field which is not per-feature, just like ...
>> >> 
>> >> > +    } ops;
>> >> > +
>> >> > +    /* Encapsulate feature specific HW info here. */
>> >> > +    union {
>> >> > +        struct psr_cat_hw_info cat_info;
>> >> > +    } info;
>> >> > +
>> >> > +    uint32_t cos_reg_val[MAX_COS_REG_CNT];
>> >> > +    unsigned int cos_num;
>> >> 
>> >> ... this. I'm pretty sure that during v8 review I did say that
>> >> this approach should be extended to all pieces of information
>> >> where it can be applied.
>> >> 
>> > I thought this when implementing v9. As cos_max is part of feature HW info, I
>> > thought it would be better to keep it in hw_info structure. Different features
>> > may have different hw_info, so the callback function is needed to get cos_max.
>> > Of course, we can keep a copy in feat_node but it is redundant. How do you
>> > think?
>> 
>> I don't follow - as long as you have a universal get_cos_max()
>> accesses, and as long as what that function returns depends
>> only on invariable things like CPUID output, I don't see why
>> this needs to be a function instead of a data field. If some
>> (perhaps future, theoretical) feature didn't want/need a
>> get_cos_max() function, the presence of that hook would
>> become questionable, yet it could surely become an optional
>> hook. However, the hook being optional could as well be
>> represented by the data field getting assigned a value of 0.
>> 
>> Bottom line: Data which can be calculated at initialization
>> time should be stored in a date object, rather than re-
>> calculating it over and over.
>> 
> The purpose to use the function is just not to define a redundant member
> in 'struct feat_node'.
> 
> The cos_max is got in cat_init_feature in patch 5 and kept in the feature's
> hw_info. The 'get_cos_max' only returns DIFFERENT features' cos_max without
> recalculation. E.g:
> 
> CAT/CDP:
> static unsigned int cat_get_cos_max(const struct feat_node *feat)
> {
>     return feat->info.cat_info.cos_max;
> }
> 
> MBA:
> static unsigned int mba_get_cos_max(const struct feat_node *feat)
> {
>     return feat->info.mba_info.cos_max;
> }
> 
> But I think it is ok to add a new member in 'struct feat_node' to keep
> cos_max for the feature.
> 
> What do you prefer? Thanks!

Sigh. If you see the above two functions, and if you expect future
new features to have similar functions, then why in the world would
you want to make the field feature specific? If every feature is
expected to have some form of maximum COS, then this is a
property applicable to all features and hence should be a field in
the common part of the structure.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow.
  2017-03-27  6:34       ` Jan Beulich
@ 2017-03-27  8:16         ` Yi Sun
  2017-03-27  8:43           ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-27  8:16 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 00:34:29, Jan Beulich wrote:
> >>> On 27.03.17 at 06:41, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-24 10:52:34, Jan Beulich wrote:
> >> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> >> > @@ -46,6 +50,9 @@
> >> >   */
> >> >  #define MAX_COS_REG_CNT  128
> >> >  
> >> > +/* CAT features use 1 COS register in one access. */
> >> > +#define CAT_COS_NUM      1
> >> 
> >> With it being stored into the feature node now I don't see why you
> >> need this constant anymore. And indeed it's being used exactly
> >> once.
> >> 
> > I remember somebody suggested me not to use constant but should define a
> > macro. As it is only used once, I will remove this and 'CDP_COS_NUM' in
> > later patch.
> 
> It may well have been me, back when this was used in multiple places.
> 
Ok, I got it. Will remove such macros.

> >> > +/*
> >> > + * Use this function to check if any allocation feature has been enabled
> >> > + * in cmdline.
> >> > + */
> >> > +static bool psr_alloc_feat_enabled(void)
> >> > +{
> >> > +    return ((!socket_info) ? false : true );
> >> 
> >> Stray parentheses (all of them actually) and blank. Even more, why
> >> not simply
> >> 
> >>     return socket_info;
> >> 
> >> ?
> >> 
> > How about 'return !!socket_info'?
> 
> And what would the !! be good for? Back when we were still using
> bool_t that would have been a requirement (the code wouldn't
> even have built without afaict), but now that we use bool I don't
> see the point (other that cluttering code). In fact I consider the
> presence of the function questionable as a whole, unless later
> patches add to it.
> 
Per Wei's suggestion, I added this function to make readers clearly understand
the meaning of the code. In previous codes, we just check 'if ( !socket_info )'.

Per test, 'return socket_info' causes warning if function type is 'bool'.

> >> > +                             struct feat_node *feat,
> >> > +                             struct psr_socket_info *info,
> >> > +                             enum psr_feat_type type)
> >> > +{
> >> > +    unsigned int socket, i;
> >> > +    struct psr_cat_hw_info cat = { };
> >> > +    uint64_t val;
> >> > +
> >> > +    /* No valid value so do not enable feature. */
> >> > +    if ( !regs.a || !regs.d )
> >> > +        return;
> >> > +
> >> > +    cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
> >> > +    cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
> >> > +
> >> > +    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> >> > +    feat->cos_reg_val[0] = cat_default_val(cat.cbm_len);
> >> > +    /*
> >> > +     * To handle cpu offline and then online case, we need read MSRs back to
> >> > +     * save values into cos_reg_val array.
> >> > +     */
> >> > +    for ( i = 1; i <= cat.cos_max; i++ )
> >> > +    {
> >> > +        rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
> >> > +        feat->cos_reg_val[i] = (uint32_t)val;
> >> > +    }
> >> 
> >> You mention this in the changes done, but I don't understand why
> >> you do this. What meaning to these values have to you? If you
> >> want hardware and cached values to match up, the much more
> >> conventional way of enforcing this would be to write the values
> >> you actually want (normally all zero).
> >> 
> > When all cpus on a socket are offline, the free_feature() is called to free
> > features resources so that the values saved in cos_reg_val[] are lost. When the
> > socket is online again, features are allocated again so that cos_reg_val[]
> > members are all initialized to 0. Only is cos_reg_val[0] initialized to default
> > value in this function in old codes.
> > 
> > But domain is still alive so that its cos id on the socket is kept. The
> > corresponding MSR value is kept too per test. To make cos_reg_val[] values be
> > same as HW to not to mislead user, we should read back the valid values on HW
> > into cos_reg_val[].
> 
> Okay, I understand the background, but I don't view this solution
> as viable: Once the last core on a socket goes offline, all
> references to it should be cleaned up. After all what will be
> brought back online may be a different physical CPU altogether;
> you can't assume MSR values to have survived even if it is the
> same CPU which comes back online, as it may have undergone
> a reset cycle, or BIOS/SMM may have played with the MSRs.
> That's even a possibility for a single core coming back online, so
> you have to reload MSRs explicitly anyway if implicit reloading
> (i.e. once vCPU-s get scheduled onto it) doesn't suffice.
> 
So, you think the MSRs values may not be valid after such process and
reloading (write MSRs to default value) is needed. If so, I would like
to do more operations in 'free_feature()':
1. Iterate all domains working on the offline socket to change
   'd->arch.psr_cos_ids[socket]' to COS 0, i.e restore it back to init
   status.
2. Restore 'socket_info[socket].cos_ref[]' to all 0.

These can make the socket's info be totally restored back to init status.

How do you think? Thanks!

> >> > +/* L3 CAT ops */
> >> > +static const struct feat_ops l3_cat_ops = {
> >> > +};
> >> 
> >> Leaving an already declared function pointer as NULL? Please don't.
> >> 
> > Ok, will consider to move it and below code into later patch.
> >     feat->ops = l3_cat_ops;
> 
> I don't mind the empty structure instance above, as long as the
> structure doesn't have any function pointer members yet (data
> members are almost always fine).
> 
To explain how the data structures are, I declared '(*get_cos_max)' in
'struct feat_ops' in patch 3. So, do you mind I remove this declaration
and just keep an empty 'struct feat_ops' in patch 3 so that we can keep
current codes in this patch?

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow.
  2017-03-27  8:16         ` Yi Sun
@ 2017-03-27  8:43           ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-27  8:43 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 27.03.17 at 10:16, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 00:34:29, Jan Beulich wrote:
>> >>> On 27.03.17 at 06:41, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-24 10:52:34, Jan Beulich wrote:
>> >> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
>> >> > @@ -46,6 +50,9 @@
>> >> >   */
>> >> >  #define MAX_COS_REG_CNT  128
>> >> >  
>> >> > +/* CAT features use 1 COS register in one access. */
>> >> > +#define CAT_COS_NUM      1
>> >> 
>> >> With it being stored into the feature node now I don't see why you
>> >> need this constant anymore. And indeed it's being used exactly
>> >> once.
>> >> 
>> > I remember somebody suggested me not to use constant but should define a
>> > macro. As it is only used once, I will remove this and 'CDP_COS_NUM' in
>> > later patch.
>> 
>> It may well have been me, back when this was used in multiple places.
>> 
> Ok, I got it. Will remove such macros.
> 
>> >> > +/*
>> >> > + * Use this function to check if any allocation feature has been enabled
>> >> > + * in cmdline.
>> >> > + */
>> >> > +static bool psr_alloc_feat_enabled(void)
>> >> > +{
>> >> > +    return ((!socket_info) ? false : true );
>> >> 
>> >> Stray parentheses (all of them actually) and blank. Even more, why
>> >> not simply
>> >> 
>> >>     return socket_info;
>> >> 
>> >> ?
>> >> 
>> > How about 'return !!socket_info'?
>> 
>> And what would the !! be good for? Back when we were still using
>> bool_t that would have been a requirement (the code wouldn't
>> even have built without afaict), but now that we use bool I don't
>> see the point (other that cluttering code). In fact I consider the
>> presence of the function questionable as a whole, unless later
>> patches add to it.
>> 
> Per Wei's suggestion, I added this function to make readers clearly 
> understand
> the meaning of the code. In previous codes, we just check 'if ( !socket_info )'.
> 
> Per test, 'return socket_info' causes warning if function type is 'bool'.

Oh, that is unfortunate (and then indeed requires to use !!).
I would have expected that conversion here works just like in
if(), where no !! would be needed.

>> >> > +                             struct feat_node *feat,
>> >> > +                             struct psr_socket_info *info,
>> >> > +                             enum psr_feat_type type)
>> >> > +{
>> >> > +    unsigned int socket, i;
>> >> > +    struct psr_cat_hw_info cat = { };
>> >> > +    uint64_t val;
>> >> > +
>> >> > +    /* No valid value so do not enable feature. */
>> >> > +    if ( !regs.a || !regs.d )
>> >> > +        return;
>> >> > +
>> >> > +    cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
>> >> > +    cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
>> >> > +
>> >> > +    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
>> >> > +    feat->cos_reg_val[0] = cat_default_val(cat.cbm_len);
>> >> > +    /*
>> >> > +     * To handle cpu offline and then online case, we need read MSRs back to
>> >> > +     * save values into cos_reg_val array.
>> >> > +     */
>> >> > +    for ( i = 1; i <= cat.cos_max; i++ )
>> >> > +    {
>> >> > +        rdmsrl(MSR_IA32_PSR_L3_MASK(i), val);
>> >> > +        feat->cos_reg_val[i] = (uint32_t)val;
>> >> > +    }
>> >> 
>> >> You mention this in the changes done, but I don't understand why
>> >> you do this. What meaning to these values have to you? If you
>> >> want hardware and cached values to match up, the much more
>> >> conventional way of enforcing this would be to write the values
>> >> you actually want (normally all zero).
>> >> 
>> > When all cpus on a socket are offline, the free_feature() is called to free
>> > features resources so that the values saved in cos_reg_val[] are lost. When the
>> > socket is online again, features are allocated again so that cos_reg_val[]
>> > members are all initialized to 0. Only is cos_reg_val[0] initialized to default
>> > value in this function in old codes.
>> > 
>> > But domain is still alive so that its cos id on the socket is kept. The
>> > corresponding MSR value is kept too per test. To make cos_reg_val[] values be
>> > same as HW to not to mislead user, we should read back the valid values on HW
>> > into cos_reg_val[].
>> 
>> Okay, I understand the background, but I don't view this solution
>> as viable: Once the last core on a socket goes offline, all
>> references to it should be cleaned up. After all what will be
>> brought back online may be a different physical CPU altogether;
>> you can't assume MSR values to have survived even if it is the
>> same CPU which comes back online, as it may have undergone
>> a reset cycle, or BIOS/SMM may have played with the MSRs.
>> That's even a possibility for a single core coming back online, so
>> you have to reload MSRs explicitly anyway if implicit reloading
>> (i.e. once vCPU-s get scheduled onto it) doesn't suffice.
>> 
> So, you think the MSRs values may not be valid after such process and
> reloading (write MSRs to default value) is needed. If so, I would like
> to do more operations in 'free_feature()':
> 1. Iterate all domains working on the offline socket to change
>    'd->arch.psr_cos_ids[socket]' to COS 0, i.e restore it back to init
>    status.
> 2. Restore 'socket_info[socket].cos_ref[]' to all 0.
> 
> These can make the socket's info be totally restored back to init status.

Yes, that's what I think is needed.

>> >> > +/* L3 CAT ops */
>> >> > +static const struct feat_ops l3_cat_ops = {
>> >> > +};
>> >> 
>> >> Leaving an already declared function pointer as NULL? Please don't.
>> >> 
>> > Ok, will consider to move it and below code into later patch.
>> >     feat->ops = l3_cat_ops;
>> 
>> I don't mind the empty structure instance above, as long as the
>> structure doesn't have any function pointer members yet (data
>> members are almost always fine).
>> 
> To explain how the data structures are, I declared '(*get_cos_max)' in
> 'struct feat_ops' in patch 3. So, do you mind I remove this declaration
> and just keep an empty 'struct feat_ops' in patch 3 so that we can keep
> current codes in this patch?

As said, I have no problem with the structure remaining empty
until subsequent patches start filling it. No need to re-structure
several patches.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow.
  2017-03-16 11:07 ` [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
@ 2017-03-27  9:07   ` Jan Beulich
  2017-03-27 12:24     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27  9:07 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> +static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> +{
> +    enum psr_feat_type feat_type;
> +
> +    switch ( type )
> +    {
> +    case PSR_CBM_TYPE_L3:
> +        feat_type = PSR_SOCKET_L3_CAT;
> +        break;
> +    default:
> +        feat_type = PSR_SOCKET_UNKNOWN;
> +        break;

Is this actually reachable, if there are no bugs in the code? If not,
you will want to add ASSERT_UNREACHABLE().

> +int psr_get_info(unsigned int socket, enum cbm_type type,
> +                 uint32_t data[], unsigned int array_len)
> +{
> +    const struct psr_socket_info *info = get_socket_info(socket);
> +    const struct feat_node *feat;
> +    enum psr_feat_type feat_type;
> +
> +    if ( IS_ERR(info) )
> +        return PTR_ERR(info);
> +
> +    if ( !data )
> +        return -EINVAL;
> +
> +    feat_type = psr_cbm_type_to_feat_type(type);
> +    feat = info->features[feat_type];

You can't blindly use the return value here as array index, as (at
least in theory, see above) the function may return
PSR_SOCKET_UNKNOWN. IOW you need to check against
ARRAY_SIZE(info->features) first.

> +/* Used by psr_get_info() */
> +#define PSR_INFO_IDX_CBM_LEN            0
> +#define PSR_INFO_IDX_COS_MAX            1
> +#define PSR_INFO_IDX_FLAG               2
> +#define PSR_INFO_CAT_SIZE               3

So I need some explanation on the naming here: Are the first three
CAT-independent, but the last one is CAT-dependent? It doesn't
look so (or else it would be odd coincidence for the last one to be
one higher than the biggest of the _IDX ones). And if they're all
in either of the two categories, their names should reflect that
(i.e. either all have _CAT in their names, or none does).

> +
> +
>  struct psr_cmt_l3 {

No double blank lines please (and just in case: comments of this kind
apply to the entire series, i.e. you shouldn't expect them to be
repeated in other patches).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow.
  2017-03-16 11:07 ` [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
@ 2017-03-27  9:23   ` Jan Beulich
  2017-03-27 12:59     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27  9:23 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/domctl.c
> +++ b/xen/arch/x86/domctl.c
> @@ -1455,23 +1455,26 @@ long arch_do_domctl(
>              break;
>  
>          case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
> -            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
> -                                 &domctl->u.psr_cat_op.data,
> -                                 PSR_CBM_TYPE_L3);
> +            domctl->u.psr_cat_op.data = 0;
> +            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
> +                              (uint32_t *)&domctl->u.psr_cat_op.data,

This is exactly why I generally object to casts: The high half of
the field will remain untouched, likely confusing the caller. You
need to decide at what layer you want to do the extension from
the internally used 32-bit type to the public interface induced
64-bit one.

> @@ -504,21 +515,30 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
>      return socket_info + socket;
>  }
>  
> -int psr_get_info(unsigned int socket, enum cbm_type type,
> -                 uint32_t data[], unsigned int array_len)
> +static const struct feat_node * psr_get_feat(unsigned int socket,
> +                                             enum cbm_type type)
>  {
>      const struct psr_socket_info *info = get_socket_info(socket);
>      const struct feat_node *feat;
>      enum psr_feat_type feat_type;
>  
>      if ( IS_ERR(info) )
> -        return PTR_ERR(info);
> +        return NULL;

You're losing the error information here - is that intentional?

> +    feat_type = psr_cbm_type_to_feat_type(type);
> +    feat = info->features[feat_type];
> +    return feat;

No need for at least the intermediate variable "feat". Instead
please add a blank line before the final return statement.

> @@ -528,9 +548,33 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
>      return -EINVAL;
>  }
>  
> -int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> -                   uint64_t *cbm, enum cbm_type type)
> +int psr_get_val(struct domain *d, unsigned int socket,
> +                uint32_t *val, enum cbm_type type)
>  {
> +    const struct feat_node *feat;
> +    unsigned int cos;
> +
> +    if ( !d || !val )
> +        return -EINVAL;

Wouldn't this better be an ASSERT()? -EINVAL should generally
indicate bad hypercall input, not things got wrong internally.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 09/25] x86: refactor psr: L3 CAT: set value: implement framework.
  2017-03-16 11:07 ` [PATCH v9 09/25] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
@ 2017-03-27  9:59   ` Jan Beulich
  2017-03-28  1:21     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27  9:59 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/domctl.c
> +++ b/xen/arch/x86/domctl.c
> @@ -1437,21 +1437,21 @@ long arch_do_domctl(
>          switch ( domctl->u.psr_cat_op.cmd )
>          {
>          case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
> -            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
> -                                 domctl->u.psr_cat_op.data,
> -                                 PSR_CBM_TYPE_L3);
> +            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
> +                              (uint32_t)domctl->u.psr_cat_op.data,

The cast here is pointless, but - along the lines of the comment
on the earlier patch - indicates a problem: You silently ignore the
upper 32-bit the caller handed you. I think for forward
compatibility it would be better if you checked they're zero. And
in such a check you could then use a cast which I would not
grumble about:

    if ( domctl->u.psr_cat_op.data != (uint32_t)domctl->u.psr_cat_op.data )
        return -E...;

> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -578,15 +578,203 @@ int psr_get_val(struct domain *d, unsigned int socket,
>      return 0;
>  }
>  
> -int psr_set_l3_cbm(struct domain *d, unsigned int socket,
> -                   uint64_t cbm, enum cbm_type type)
> +/* Set value functions */
> +static unsigned int get_cos_num(const struct psr_socket_info *info)
>  {
>      return 0;
>  }
>  
> +static int gather_val_array(uint32_t val[],
> +                            uint32_t array_len,

I think I've mentioned this before - please avoid fixed width types
where they're no needed. In the case here the array wants it, but
the length can easily be unsigned int.

> +                            const struct psr_socket_info *info,
> +                            unsigned int old_cos)
> +{
> +    return -EINVAL;
> +}
> +
> +static int insert_new_val_to_array(uint32_t val[],

insert_new_val_into_array()?

(and whether "new" needs to be part of the name is then even
questionable)

> +static int find_cos(const uint32_t val[], uint32_t array_len,
> +                    enum psr_feat_type feat_type,
> +                    const struct psr_socket_info *info)
> +{
> +    ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));

Excuse me, but no, this is another of those really bad casts. I
guess you've added it to deal with info being pointer-to-const.
In such a case you should instead drop the const, unless the
consuming function can be made take a pointer-to-const (which
isn't the case here, as check_lock() wants to write into the
structure).

> +int psr_set_val(struct domain *d, unsigned int socket,
> +                uint32_t val, enum cbm_type type)
> +{
> +    unsigned int old_cos;
> +    int cos, ret;
> +    unsigned int *ref;
> +    uint32_t *val_array;
> +    struct psr_socket_info *info = get_socket_info(socket);
> +    uint32_t array_len;
> +    enum psr_feat_type feat_type;
> +
> +    if ( IS_ERR(info) )
> +        return PTR_ERR(info);
> +
> +    feat_type = psr_cbm_type_to_feat_type(type);
> +    if ( !test_bit(feat_type, &info->feat_mask) )
> +        return -ENOENT;
> +
> +    /*
> +     * Step 0:
> +     * old_cos means the COS ID current domain is using. By default, it is 0.
> +     *
> +     * For every COS ID, there is a reference count to record how many domains
> +     * are using the COS register corresponding to this COS ID.
> +     * - If ref[old_cos] is 0, that means this COS is not used by any domain.
> +     * - If ref[old_cos] is 1, that means this COS is only used by current
> +     *   domain.
> +     * - If ref[old_cos] is more than 1, that mean multiple domains are using
> +     *   this COS.
> +     */
> +    old_cos = d->arch.psr_cos_ids[socket];
> +    ASSERT(old_cos < MAX_COS_REG_CNT && old_cos >= 0);

The right side is always true and should hence be dropped.

> +    ref = info->cos_ref;
> +
> +    /*
> +     * Step 1:
> +     * Gather a value array to store all features cos_reg_val[old_cos].
> +     * And, set the input new val into array according to the feature's
> +     * position in array.
> +     */
> +    array_len = get_cos_num(info);
> +    val_array = xzalloc_array(uint32_t, array_len);
> +    if ( !val_array )
> +        return -ENOMEM;
> +
> +    if ( (ret = gather_val_array(val_array, array_len, info, old_cos)) != 0 )
> +        goto free_array;
> +
> +    if ( (ret = insert_new_val_to_array(val_array, array_len, info,
> +                                        feat_type, type, val)) != 0 )
> +        goto free_array;
> +
> +    spin_lock(&info->ref_lock);

Am I right in understanding that up to here operations are not
risking to race merely because they're inside the domctl lock? If so,
this needs to be spelled out, so we have a chance of noticing the
dependency when we break up that lock (which we have plans for
doing).

> +    /*
> +     * Step 2:
> +     * Try to find if there is already a COS ID on which all features' values

Btw., I think this spelling ("features' values") is the right one - please
go through and make it consistent everywhere, including in the patch
description(s).

> +     * are same as the array. Then, we can reuse this COS ID.
> +     */
> +    cos = find_cos(val_array, array_len, feat_type, info);
> +    if ( cos == old_cos )
> +    {
> +        ret = 0;
> +        goto unlock_free_array;
> +    }
> +    else if ( cos >= 0 )

Pointless "else".

> +        goto cos_found;

I think it would be better not to use goto here, other than for the
error handling parts (where I don't really like it either, but I do
accept it since others think that's the least ugly way).

> +    /*
> +     * Step 3:
> +     * If fail to find, we need pick an available COS ID.

I think you've corrected this somewhat strange wording at the start
of the comment here elsewhere already. I can only repeat that
respective comments given for one location should always be
extended to the entire series.

> +     * In fact, only COS ID which ref is 1 or 0 can be picked for current
> +     * domain. If old_cos is not 0 and its ref==1, that means only current
> +     * domain is using this old_cos ID. So, this old_cos ID certainly can
> +     * be reused by current domain. Ref==0 means there is no any domain
> +     * using this COS ID. So it can be used for current domain too.
> +     */
> +    cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
> +    if ( cos < 0 )
> +    {
> +        ret = cos;
> +        goto unlock_free_array;
> +    }
> +
> +    /*
> +     * Step 4:
> +     * Write all features MSRs according to the COS ID.
> +     */
> +    ret = write_psr_msr(socket, cos, val, type, feat_type);
> +    if ( ret )
> +        goto unlock_free_array;
> +
> +cos_found:

Style (also the other labels further down).

> +    /*
> +     * Step 5:
> +     * Update ref according to COS ID.
> +     */
> +    ref[cos]++;
> +    ASSERT(!cos || ref[cos]);
> +    ASSERT(!old_cos || ref[old_cos]);
> +    ref[old_cos]--;
> +    spin_unlock(&info->ref_lock);
> +
> +    /*
> +     * Step 6:
> +     * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> +     * which COS the domain is using on the socket. One domain can only use
> +     * one COS ID at same time on each socket.
> +     */
> +    d->arch.psr_cos_ids[socket] = cos;
> +    goto free_array;
> +
> +unlock_free_array:
> +    spin_unlock(&info->ref_lock);
> +free_array:
> +    xfree(val_array);
> +    return ret;
> +}

I think overall things would look better if the successful path was the
straight (goto-free) one.

>  /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
>  static void psr_free_cos(struct domain *d)
>  {
> +    unsigned int socket, cos;
> +
> +    if ( !socket_info || !d->arch.psr_cos_ids )
> +        return;

Can d->arch.psr_cos_ids be non-NULL when !socket_info? If not,
check only the former with an if(), and ASSERT() the latter.

> +    /* Domain is destroied so its cos_ref should be decreased. */
> +    for ( socket = 0; socket < nr_sockets; socket++ )
> +    {
> +        struct psr_socket_info *info;
> +
> +        /* cos 0 is default one which does not need be handled. */
> +        cos = d->arch.psr_cos_ids[socket];
> +        if ( cos == 0 )
> +            continue;
> +
> +        /*
> +         * If domain uses other cos ids, all corresponding refs must have been
> +         * increased 1 for this domain. So, we need decrease them.

... by 1 ... need to ... I also question the presence of the word
"must" in here.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-16 11:08 ` [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
@ 2017-03-27 10:17   ` Jan Beulich
  2017-03-28  3:12     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 10:17 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -101,6 +101,28 @@ struct feat_node {
>          /* get_val is used to get feature COS register value. */
>          void (*get_val)(const struct feat_node *feat, unsigned int cos,
>                          enum cbm_type type, uint32_t *val);
> +
> +        /*
> +         * get_old_val and set_new_val are a pair of functions called in order.
> +         * The caller will traverse all features in the array and call
> +         * 'get_old_val' to get old_cos register value of all supported
> +         * features. Then, call 'set_new_val' to set the new value for the
> +         * designated feature.
> +         *
> +         * All the values are set into value array according to the traversal
> +         * order, meaning the same order of feature array members.
> +         *
> +         * The return value meaning of set_new_val:
> +         * 0 - success.
> +         * negative - error.
> +         */
> +        void (*get_old_val)(uint32_t val[],
> +                            const struct feat_node *feat,
> +                            unsigned int old_cos);
> +        int (*set_new_val)(uint32_t val[],
> +                           const struct feat_node *feat,
> +                           enum cbm_type type,
> +                           uint32_t new_val);

Along the lines of an earlier comment - are "old" and "new" really
meaningful here?

> @@ -212,6 +234,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
>  }
>  
>  /* CAT common functions implementation. */
> +static bool psr_check_cbm(unsigned int cbm_len, uint32_t cbm)
> +{
> +    unsigned int first_bit, zero_bit;
> +
> +    /* Set bits should only in the range of [0, cbm_len]. */
> +    if ( cbm & (~0ul << cbm_len) )

Same question as elsewhere about the use of the ul suffix here:
Can cbm_len really be any value in [0,32]? If not, I don't see
why the calculation needs to be done as unsigned long. Otoh ...

> +        return false;
> +
> +    /* At least one bit need to be set. */
> +    if ( cbm == 0 )
> +        return false;
> +
> +    first_bit = find_first_bit((uint64_t *)&cbm, cbm_len);
> +    zero_bit = find_next_zero_bit((uint64_t *)&cbm, cbm_len, first_bit);

... these bogus casts suggest that the function would best have
an "unsigned long" parameter.

> @@ -285,11 +330,35 @@ static void cat_get_val(const struct feat_node *feat, unsigned int cos,
>      *val = feat->cos_reg_val[cos];
>  }
>  
> +/* val[] len checking is done by caller. */
> +static void cat_get_old_val(uint32_t val[],
> +                            const struct feat_node *feat,
> +                            unsigned int old_cos)
> +{
> +    cat_get_val(feat, old_cos, 0, &val[0]);
> +}
> +
> +/* val[] len checking is done by caller. */
> +static int cat_set_new_val(uint32_t val[],
> +                           const struct feat_node *feat,
> +                           enum cbm_type type,
> +                           uint32_t new_val)
> +{
> +    if ( !psr_check_cbm(feat->info.cat_info.cbm_len, new_val) )
> +        return -EINVAL;
> +
> +    val[0] = new_val;
> +
> +    return 0;
> +}
> +
>  /* L3 CAT ops */
>  static const struct feat_ops l3_cat_ops = {
>      .get_cos_max = cat_get_cos_max,
>      .get_feat_info = cat_get_feat_info,
>      .get_val = cat_get_val,
> +    .get_old_val = cat_get_old_val,
> +    .set_new_val = cat_set_new_val,
>  };
>  
>  static void __init parse_psr_bool(char *s, char *value, char *feature,
> @@ -581,7 +650,21 @@ int psr_get_val(struct domain *d, unsigned int socket,
>  /* Set value functions */
>  static unsigned int get_cos_num(const struct psr_socket_info *info)
>  {
> -    return 0;
> +    const struct feat_node *feat;
> +    unsigned int num = 0, i;
> +
> +    /* Get all features total amount. */
> +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> +    {
> +        if ( !info->features[i] )
> +            continue;
> +
> +        feat = info->features[i];
> +
> +        num += feat->cos_num;
> +    }
> +
> +    return num;
>  }
>  
>  static int gather_val_array(uint32_t val[],
> @@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
>                              const struct psr_socket_info *info,
>                              unsigned int old_cos)
>  {
> -    return -EINVAL;
> +    const struct feat_node *feat;
> +    unsigned int i;
> +
> +    if ( !val )
> +        return -EINVAL;
> +
> +    /* Get all features current values according to old_cos. */
> +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> +    {
> +        if ( !info->features[i] )
> +            continue;
> +
> +        feat = info->features[i];
> +
> +        if ( old_cos > feat->ops.get_cos_max(feat) )
> +            old_cos = 0;
> +
> +        /* value getting order is same as feature array */
> +        feat->ops.get_old_val(val, feat, old_cos);
> +
> +        array_len -= feat->cos_num;

So this I should really have asked about on a much earlier patch,
but I've recognize the oddity only now: Why is cos_num
per-feature-node instead of per-feature? This should really be a
field in struct feat_ops (albeit the name "ops" then will be slightly
misleading, but I think that's tolerable if you can't think of a better
name).

> +        if ( array_len < 0 )
> +            return -ENOSPC;

This check needs doing earlier - you need to make sure array_len
>= ops.cos_num prior to calling ops.get_old_val(). (Doing the
check after the subtraction even causes wrapping issues, which
are even more visible in similar code further down.)

> @@ -599,7 +709,43 @@ static int insert_new_val_to_array(uint32_t val[],
>                                     enum cbm_type type,
>                                     uint32_t new_val)
>  {
> -    return -EINVAL;
> +    const struct feat_node *feat;
> +    int ret;
> +    unsigned int i;
> +
> +    ASSERT(feat_type < PSR_SOCKET_MAX_FEAT);
> +
> +    /* Set new value into array according to feature's position in array. */
> +    for ( i = 0; i < feat_type; i++ )
> +    {
> +        if ( !info->features[i] )
> +            continue;
> +
> +        feat = info->features[i];
> +
> +        array_len -= feat->cos_num;
> +        if ( array_len <= 0 )
> +            return -ENOSPC;
> +
> +        val += feat->cos_num;
> +    }
> +
> +    feat = info->features[feat_type];
> +
> +    array_len -= feat->cos_num;
> +    if ( array_len < 0 )
> +        return -ENOSPC;
> +
> +    /*
> +     * Value setting position is same as feature array.
> +     * Different features may have different setting behaviors, e.g. CDP
> +     * has two values (DATA/CODE) which need us to save input value to
> +     * different position in the array according to type, so we have to
> +     * maintain a callback function.
> +     */
> +    ret = feat->ops.set_new_val(val, feat, type, new_val);
> +
> +    return ret;

Again a case of a pointless intermediate variable.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 11/25] x86: refactor psr: L3 CAT: set value: implement cos finding flow.
  2017-03-16 11:08 ` [PATCH v9 11/25] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
@ 2017-03-27 10:28   ` Jan Beulich
  2017-03-28  3:26     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 10:28 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/psr.c
> +++ b/xen/arch/x86/psr.c
> @@ -123,6 +123,19 @@ struct feat_node {
>                             const struct feat_node *feat,
>                             enum cbm_type type,
>                             uint32_t new_val);
> +
> +        /*
> +         * compare_val is used in set value process to compare if the
> +         * input value array can match all the features' COS registers values
> +         * according to input cos id.
> +         *
> +         * The return value is:
> +         * 1 - find the entry in value array.

found ...

> +         * 0 - not find the entry in value array.

didn't find ...

> +static int cat_compare_val(const uint32_t val[],
> +                           const struct feat_node *feat,
> +                           unsigned int cos)
> +{
> +    /*
> +     * Different features' cos_max are different. If cos id of the feature
> +     * being set exceeds other feature's cos_max, the val of other feature
> +     * must be default value. HW supports such case.
> +     */
> +    if ( cos > feat->info.cat_info.cos_max )
> +    {
> +        /* cos_reg_val[0] is the default value. */
> +        if ( val[0] != feat->cos_reg_val[0] )
> +            return -EINVAL;

As you can see, with cos_max moved into the generic portion of the
feature node, this entire check can move into the caller.

> +        /* Find */

Found (also below)

> +        return 1;
> +    }
> +
> +    if ( val[0] == feat->cos_reg_val[cos] )
> +        /* Find */
> +        return 1;
> +
> +    /* Not find */
> +    return 0;
> +}

Or actually, the entire function then becomes feature independent,
as it seems. And I think I did suggest that already during review of
an earlier version.

> @@ -752,7 +793,61 @@ static int find_cos(const uint32_t val[], uint32_t array_len,
>                      enum psr_feat_type feat_type,
>                      const struct psr_socket_info *info)
>  {
> +    unsigned int cos, i;
> +    const unsigned int *ref = info->cos_ref;
> +    const struct feat_node *feat;
> +    const uint32_t *val_array = val;

The name doesn't match the purpose - as you increment the pointer,
its name should rather be "val_ptr" or some such.

> +    int find = 0;

"found" again, or even simply "rc"? Also I think this would better
move into the outer for() scope.

> +    unsigned int cos_max;
> +
>      ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
> +
> +    /* cos_max is the one of the feature which is being set. */
> +    feat = info->features[feat_type];
> +    if ( !feat )
> +        return -ENOENT;
> +
> +    cos_max = feat->ops.get_cos_max(feat);
> +
> +    for ( cos = 0; cos <= cos_max; cos++ )
> +    {
> +        if ( cos && !ref[cos] )
> +            continue;
> +
> +        /*
> +         * If fail to find cos in below loop, need find whole feature array
> +         * again from beginning.
> +         */
> +        val_array = val;

You wouldn't need to re-do this here if you moved the variable
declaration (with initializer) into this scope. This then also
eliminates the need for the comment, which otherwise would
need its wording corrected.

> +        for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> +        {
> +            if ( !info->features[i] )
> +                continue;
> +
> +            feat = info->features[i];

Please swap if() and assignment, utilizing the local variable in the
if().

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-16 11:08 ` [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
@ 2017-03-27 10:37   ` Jan Beulich
  2017-03-28  4:58     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 10:37 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> +static bool cat_fits_cos_max(const uint32_t val[],
> +                             const struct feat_node *feat,
> +                             unsigned int cos)
> +{
> +    if ( cos > feat->info.cat_info.cos_max &&
> +         val[0] != feat->cos_reg_val[0] )
> +            /*
> +             * Exceed cos_max and value to set is not default,
> +             * return error.
> +             */
> +            return false;
> +
> +    return true;
> +}

Same here - with cos_max moved out, the hook would seem to
become unnecessary.

>  static int pick_avail_cos(const struct psr_socket_info *info,
>                            const uint32_t val[], uint32_t array_len,
>                            unsigned int old_cos,
>                            enum psr_feat_type feat_type)
>  {
> +    unsigned int cos;
> +    unsigned int cos_max = 0;
> +    const struct feat_node *feat;
> +    const unsigned int *ref = info->cos_ref;
> +
>      ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
> -    return -ENOENT;
> +
> +    /* cos_max is the one of the feature which is being set. */
> +    feat = info->features[feat_type];
> +    if ( !feat )
> +        return -ENOENT;
> +
> +    cos_max = feat->ops.get_cos_max(feat);
> +    if ( !cos_max )
> +        return -ENOENT;
> +
> +    /*
> +     * If old cos is referred only by the domain, then use it. And, we cannot

"the domain" here is lacking context - there's no domain involved
in the function - "... the domain currently using it, then re-use it"?

> +     * use id 0 because it stores the default values.
> +     */
> +    if ( old_cos && ref[old_cos] == 1 &&
> +         fits_cos_max(val, array_len, info, old_cos) )
> +            return old_cos;
> +
> +    /* Find an unused one other than cos0. */
> +    for ( cos = 1; cos <= cos_max; cos++ )
> +    {
> +        /*
> +         * ref is 0 means this COS is not used by other domain and
> +         * can be used for current setting.
> +         */
> +        if ( !ref[cos] )
> +        {
> +            if ( !fits_cos_max(val, array_len, info, cos) )
> +                return -EOVERFLOW;

Perhaps better use "break;" here.

> +            return cos;
> +        }
> +    }
> +
> +    return -EOVERFLOW;
>  }

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 13/25] x86: refactor psr: L3 CAT: set value: implement write msr flow.
  2017-03-16 11:08 ` [PATCH v9 13/25] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
@ 2017-03-27 10:46   ` Jan Beulich
  2017-03-28  5:06     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 10:46 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> @@ -421,6 +425,18 @@ static bool cat_fits_cos_max(const uint32_t val[],
>  }
>  
>  /* L3 CAT ops */
> +static void l3_cat_write_msr(unsigned int cos, uint32_t val,
> +                             enum cbm_type type, struct feat_node *feat)

"type" is an unused parameter. Please remove it from the hook
and this function.

> +{
> +    if ( feat->cos_reg_val[cos] != val )
> +    {
> +        feat->cos_reg_val[cos] = val;
> +        wrmsrl(MSR_IA32_PSR_L3_MASK(cos), (uint64_t)val);

I don't see the need for the cast.

> +    }
> +
> +    return;
> +}

Stray "return".

> +struct cos_write_info
> +{
> +    unsigned int cos;
> +    struct feat_node *feature;
> +    uint32_t val;
> +    enum cbm_type type;

With the "type" parameter removed above, this looks to be an
unused field then too. After the removal of it, please re-order
fields to leave no holes.

> +static void do_write_psr_msr(void *data)
> +{
> +    struct cos_write_info *info = (struct cos_write_info *)data;

Unnecessary cast again.

> +    unsigned int cos            = info->cos;
> +    struct feat_node *feat      = info->feature;
> +
> +    if ( !feat )
> +        return;

You shouldn't even call this function when !feat.

> +    if ( cos > feat->ops.get_cos_max(feat) )
> +        return;
> +
> +    feat->ops.write_msr(cos, info->val, info->type, feat);
> +}
> +
>  static int write_psr_msr(unsigned int socket, unsigned int cos,
>                           uint32_t val, enum cbm_type type,

"type", according to the above comments, should then go away
here too as it seems.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow.
  2017-03-27  9:07   ` Jan Beulich
@ 2017-03-27 12:24     ` Yi Sun
  2017-03-27 12:51       ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-27 12:24 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 03:07:37, Jan Beulich wrote:
> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> > +static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> > +{
> > +    enum psr_feat_type feat_type;
> > +
> > +    switch ( type )
> > +    {
> > +    case PSR_CBM_TYPE_L3:
> > +        feat_type = PSR_SOCKET_L3_CAT;
> > +        break;
> > +    default:
> > +        feat_type = PSR_SOCKET_UNKNOWN;
> > +        break;
> 
> Is this actually reachable, if there are no bugs in the code? If not,
> you will want to add ASSERT_UNREACHABLE().
> 
If there is no bug, we should not reach here. Will use ASSERT_UNREACHABLE().

> > +int psr_get_info(unsigned int socket, enum cbm_type type,
> > +                 uint32_t data[], unsigned int array_len)
> > +{
> > +    const struct psr_socket_info *info = get_socket_info(socket);
> > +    const struct feat_node *feat;
> > +    enum psr_feat_type feat_type;
> > +
> > +    if ( IS_ERR(info) )
> > +        return PTR_ERR(info);
> > +
> > +    if ( !data )
> > +        return -EINVAL;
> > +
> > +    feat_type = psr_cbm_type_to_feat_type(type);
> > +    feat = info->features[feat_type];
> 
> You can't blindly use the return value here as array index, as (at
> least in theory, see above) the function may return
> PSR_SOCKET_UNKNOWN. IOW you need to check against
> ARRAY_SIZE(info->features) first.
> 
If I use 'ASSERT_UNREACHABLE()' above, I don't need check against
ARRAY_SIZE(info->features) here, right?

> > +/* Used by psr_get_info() */
> > +#define PSR_INFO_IDX_CBM_LEN            0
CAT/CDP specific.

> > +#define PSR_INFO_IDX_COS_MAX            1
Common so far.

> > +#define PSR_INFO_IDX_FLAG               2
CAT/CDP specific so far.

> > +#define PSR_INFO_CAT_SIZE               3
Array size which can be used for all features but may not be appropriate for
future features. So I defined it as CAT specific.

> 
> So I need some explanation on the naming here: Are the first three
> CAT-independent, but the last one is CAT-dependent? It doesn't
> look so (or else it would be odd coincidence for the last one to be
> one higher than the biggest of the _IDX ones). And if they're all
> in either of the two categories, their names should reflect that
> (i.e. either all have _CAT in their names, or none does).
> 
Please check above comments. Maybe below definitions are better?
PSR_INFO_IDX_CAT_CBM_LEN
PSR_INFO_IDX_COS_MAX
PSR_INFO_IDX_CAT_FLAG
PSR_INFO_ARRAY_SIZE

> > +
> > +
> >  struct psr_cmt_l3 {
> 
> No double blank lines please (and just in case: comments of this kind
> apply to the entire series, i.e. you shouldn't expect them to be
> repeated in other patches).
> 
Sorry for this. Do we have code style check tool in xen?

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow.
  2017-03-27 12:24     ` Yi Sun
@ 2017-03-27 12:51       ` Jan Beulich
  2017-03-27 13:19         ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 12:51 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 27.03.17 at 14:24, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 03:07:37, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
>> > +static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
>> > +{
>> > +    enum psr_feat_type feat_type;
>> > +
>> > +    switch ( type )
>> > +    {
>> > +    case PSR_CBM_TYPE_L3:
>> > +        feat_type = PSR_SOCKET_L3_CAT;
>> > +        break;
>> > +    default:
>> > +        feat_type = PSR_SOCKET_UNKNOWN;
>> > +        break;
>> 
>> Is this actually reachable, if there are no bugs in the code? If not,
>> you will want to add ASSERT_UNREACHABLE().
>> 
> If there is no bug, we should not reach here. Will use ASSERT_UNREACHABLE().
> 
>> > +int psr_get_info(unsigned int socket, enum cbm_type type,
>> > +                 uint32_t data[], unsigned int array_len)
>> > +{
>> > +    const struct psr_socket_info *info = get_socket_info(socket);
>> > +    const struct feat_node *feat;
>> > +    enum psr_feat_type feat_type;
>> > +
>> > +    if ( IS_ERR(info) )
>> > +        return PTR_ERR(info);
>> > +
>> > +    if ( !data )
>> > +        return -EINVAL;
>> > +
>> > +    feat_type = psr_cbm_type_to_feat_type(type);
>> > +    feat = info->features[feat_type];
>> 
>> You can't blindly use the return value here as array index, as (at
>> least in theory, see above) the function may return
>> PSR_SOCKET_UNKNOWN. IOW you need to check against
>> ARRAY_SIZE(info->features) first.
>> 
> If I use 'ASSERT_UNREACHABLE()' above, I don't need check against
> ARRAY_SIZE(info->features) here, right?

That's a slightly difficult call: The assertion will expand to nothing
in production builds, so to be on the safe side I think you better
check function return values _everywhere_.

>> > +/* Used by psr_get_info() */
>> > +#define PSR_INFO_IDX_CBM_LEN            0
> CAT/CDP specific.
> 
>> > +#define PSR_INFO_IDX_COS_MAX            1
> Common so far.
> 
>> > +#define PSR_INFO_IDX_FLAG               2
> CAT/CDP specific so far.
> 
>> > +#define PSR_INFO_CAT_SIZE               3
> Array size which can be used for all features but may not be appropriate for
> future features. So I defined it as CAT specific.
> 
>> 
>> So I need some explanation on the naming here: Are the first three
>> CAT-independent, but the last one is CAT-dependent? It doesn't
>> look so (or else it would be odd coincidence for the last one to be
>> one higher than the biggest of the _IDX ones). And if they're all
>> in either of the two categories, their names should reflect that
>> (i.e. either all have _CAT in their names, or none does).
>> 
> Please check above comments. Maybe below definitions are better?
> PSR_INFO_IDX_CAT_CBM_LEN
> PSR_INFO_IDX_COS_MAX
> PSR_INFO_IDX_CAT_FLAG
> PSR_INFO_ARRAY_SIZE

But why would the array size be 3 for some feature only having
COS_MAX (for example)? I think you should
- put common indexes first
- have PSR_INFO_CAT_ARRAY_SIZE (or PSR_INFO_NUM_IDX_CAT
or whatever, but with CAT in it).

>> > +
>> > +
>> >  struct psr_cmt_l3 {
>> 
>> No double blank lines please (and just in case: comments of this kind
>> apply to the entire series, i.e. you shouldn't expect them to be
>> repeated in other patches).
>> 
> Sorry for this. Do we have code style check tool in xen?

No, no-one so far had the time to put one together.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow.
  2017-03-27  9:23   ` Jan Beulich
@ 2017-03-27 12:59     ` Yi Sun
  2017-03-27 13:34       ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-27 12:59 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 03:23:08, Jan Beulich wrote:
> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/domctl.c
> > +++ b/xen/arch/x86/domctl.c
> > @@ -1455,23 +1455,26 @@ long arch_do_domctl(
> >              break;
> >  
> >          case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
> > -            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
> > -                                 &domctl->u.psr_cat_op.data,
> > -                                 PSR_CBM_TYPE_L3);
> > +            domctl->u.psr_cat_op.data = 0;
> > +            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
> > +                              (uint32_t *)&domctl->u.psr_cat_op.data,
> 
> This is exactly why I generally object to casts: The high half of
> the field will remain untouched, likely confusing the caller. You
> need to decide at what layer you want to do the extension from
> the internally used 32-bit type to the public interface induced
> 64-bit one.
> 
'psr_cat_op.data' is used as interface between tools/ and hyperviosr. We
defined it as 'uint64_t' to fulfill future requests because MSRs registers
are 64bit although the upper 32bit are not used yet.

Per your suggetion to use 'uint32_t' internally for CBM, I changed
psr_get_val/psr_set_val parameters type from 'uint64_t' to 'uint32_t'. That
is the reason to do cast here. Is this an appropriate choice?

> > @@ -504,21 +515,30 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
> >      return socket_info + socket;
> >  }
> >  
> > -int psr_get_info(unsigned int socket, enum cbm_type type,
> > -                 uint32_t data[], unsigned int array_len)
> > +static const struct feat_node * psr_get_feat(unsigned int socket,
> > +                                             enum cbm_type type)
> >  {
> >      const struct psr_socket_info *info = get_socket_info(socket);
> >      const struct feat_node *feat;
> >      enum psr_feat_type feat_type;
> >  
> >      if ( IS_ERR(info) )
> > -        return PTR_ERR(info);
> > +        return NULL;
> 
> You're losing the error information here - is that intentional?
> 
This function returns the 'struct feat_node *' object. If error happens, it
returns NULL. The caller handles the error.

> > +    feat_type = psr_cbm_type_to_feat_type(type);
> > +    feat = info->features[feat_type];
> > +    return feat;
> 
> No need for at least the intermediate variable "feat". Instead
> please add a blank line before the final return statement.
> 
Sure. Thanks!

> > @@ -528,9 +548,33 @@ int psr_get_info(unsigned int socket, enum cbm_type type,
> >      return -EINVAL;
> >  }
> >  
> > -int psr_get_l3_cbm(struct domain *d, unsigned int socket,
> > -                   uint64_t *cbm, enum cbm_type type)
> > +int psr_get_val(struct domain *d, unsigned int socket,
> > +                uint32_t *val, enum cbm_type type)
> >  {
> > +    const struct feat_node *feat;
> > +    unsigned int cos;
> > +
> > +    if ( !d || !val )
> > +        return -EINVAL;
> 
> Wouldn't this better be an ASSERT()? -EINVAL should generally
> indicate bad hypercall input, not things got wrong internally.
> 
Ok, thanks!

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow.
  2017-03-27 12:51       ` Jan Beulich
@ 2017-03-27 13:19         ` Yi Sun
  2017-03-27 13:32           ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-27 13:19 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 06:51:07, Jan Beulich wrote:
> >>> On 27.03.17 at 14:24, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-27 03:07:37, Jan Beulich wrote:
> >> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> >> > +static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> >> > +{
> >> > +    enum psr_feat_type feat_type;
> >> > +
> >> > +    switch ( type )
> >> > +    {
> >> > +    case PSR_CBM_TYPE_L3:
> >> > +        feat_type = PSR_SOCKET_L3_CAT;
> >> > +        break;
> >> > +    default:
> >> > +        feat_type = PSR_SOCKET_UNKNOWN;
> >> > +        break;
> >> 
> >> Is this actually reachable, if there are no bugs in the code? If not,
> >> you will want to add ASSERT_UNREACHABLE().
> >> 
> > If there is no bug, we should not reach here. Will use ASSERT_UNREACHABLE().
> > 
> >> > +int psr_get_info(unsigned int socket, enum cbm_type type,
> >> > +                 uint32_t data[], unsigned int array_len)
> >> > +{
> >> > +    const struct psr_socket_info *info = get_socket_info(socket);
> >> > +    const struct feat_node *feat;
> >> > +    enum psr_feat_type feat_type;
> >> > +
> >> > +    if ( IS_ERR(info) )
> >> > +        return PTR_ERR(info);
> >> > +
> >> > +    if ( !data )
> >> > +        return -EINVAL;
> >> > +
> >> > +    feat_type = psr_cbm_type_to_feat_type(type);
> >> > +    feat = info->features[feat_type];
> >> 
> >> You can't blindly use the return value here as array index, as (at
> >> least in theory, see above) the function may return
> >> PSR_SOCKET_UNKNOWN. IOW you need to check against
> >> ARRAY_SIZE(info->features) first.
> >> 
> > If I use 'ASSERT_UNREACHABLE()' above, I don't need check against
> > ARRAY_SIZE(info->features) here, right?
> 
> That's a slightly difficult call: The assertion will expand to nothing
> in production builds, so to be on the safe side I think you better
> check function return values _everywhere_.
> 
Got it, will check feat_type.

> >> > +/* Used by psr_get_info() */
> >> > +#define PSR_INFO_IDX_CBM_LEN            0
> > CAT/CDP specific.
> > 
> >> > +#define PSR_INFO_IDX_COS_MAX            1
> > Common so far.
> > 
> >> > +#define PSR_INFO_IDX_FLAG               2
> > CAT/CDP specific so far.
> > 
> >> > +#define PSR_INFO_CAT_SIZE               3
> > Array size which can be used for all features but may not be appropriate for
> > future features. So I defined it as CAT specific.
> > 
> >> 
> >> So I need some explanation on the naming here: Are the first three
> >> CAT-independent, but the last one is CAT-dependent? It doesn't
> >> look so (or else it would be odd coincidence for the last one to be
> >> one higher than the biggest of the _IDX ones). And if they're all
> >> in either of the two categories, their names should reflect that
> >> (i.e. either all have _CAT in their names, or none does).
> >> 
> > Please check above comments. Maybe below definitions are better?
> > PSR_INFO_IDX_CAT_CBM_LEN
> > PSR_INFO_IDX_COS_MAX
> > PSR_INFO_IDX_CAT_FLAG
> > PSR_INFO_ARRAY_SIZE
> 
> But why would the array size be 3 for some feature only having
> COS_MAX (for example)? I think you should
> - put common indexes first
> - have PSR_INFO_CAT_ARRAY_SIZE (or PSR_INFO_NUM_IDX_CAT
> or whatever, but with CAT in it).
> 
Per current known features, all of them (CAT/MBA) have three members in
HW info. So, I think we can define ARRAY_SIZE to 3 and define its name to
a common name so far.

BRs,
Sun Yi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow.
  2017-03-27 13:19         ` Yi Sun
@ 2017-03-27 13:32           ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 13:32 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 27.03.17 at 15:19, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 06:51:07, Jan Beulich wrote:
>> >>> On 27.03.17 at 14:24, <yi.y.sun@linux.intel.com> wrote:
>> > Please check above comments. Maybe below definitions are better?
>> > PSR_INFO_IDX_CAT_CBM_LEN
>> > PSR_INFO_IDX_COS_MAX
>> > PSR_INFO_IDX_CAT_FLAG
>> > PSR_INFO_ARRAY_SIZE
>> 
>> But why would the array size be 3 for some feature only having
>> COS_MAX (for example)? I think you should
>> - put common indexes first
>> - have PSR_INFO_CAT_ARRAY_SIZE (or PSR_INFO_NUM_IDX_CAT
>> or whatever, but with CAT in it).
>> 
> Per current known features, all of them (CAT/MBA) have three members in
> HW info. So, I think we can define ARRAY_SIZE to 3 and define its name to
> a common name so far.

Okay then.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow.
  2017-03-27 12:59     ` Yi Sun
@ 2017-03-27 13:34       ` Jan Beulich
  2017-03-28  2:13         ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 13:34 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 27.03.17 at 14:59, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 03:23:08, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
>> > --- a/xen/arch/x86/domctl.c
>> > +++ b/xen/arch/x86/domctl.c
>> > @@ -1455,23 +1455,26 @@ long arch_do_domctl(
>> >              break;
>> >  
>> >          case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
>> > -            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
>> > -                                 &domctl->u.psr_cat_op.data,
>> > -                                 PSR_CBM_TYPE_L3);
>> > +            domctl->u.psr_cat_op.data = 0;
>> > +            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
>> > +                              (uint32_t *)&domctl->u.psr_cat_op.data,
>> 
>> This is exactly why I generally object to casts: The high half of
>> the field will remain untouched, likely confusing the caller. You
>> need to decide at what layer you want to do the extension from
>> the internally used 32-bit type to the public interface induced
>> 64-bit one.
>> 
> 'psr_cat_op.data' is used as interface between tools/ and hyperviosr. We
> defined it as 'uint64_t' to fulfill future requests because MSRs registers
> are 64bit although the upper 32bit are not used yet.
> 
> Per your suggetion to use 'uint32_t' internally for CBM, I changed
> psr_get_val/psr_set_val parameters type from 'uint64_t' to 'uint32_t'. That
> is the reason to do cast here. Is this an appropriate choice?

No, as said, it is not. The choice of types is fine, but you need to
make this work without casts (i.e. presumably with some
intermediate variable at one of the layers).

>> > @@ -504,21 +515,30 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
>> >      return socket_info + socket;
>> >  }
>> >  
>> > -int psr_get_info(unsigned int socket, enum cbm_type type,
>> > -                 uint32_t data[], unsigned int array_len)
>> > +static const struct feat_node * psr_get_feat(unsigned int socket,
>> > +                                             enum cbm_type type)
>> >  {
>> >      const struct psr_socket_info *info = get_socket_info(socket);
>> >      const struct feat_node *feat;
>> >      enum psr_feat_type feat_type;
>> >  
>> >      if ( IS_ERR(info) )
>> > -        return PTR_ERR(info);
>> > +        return NULL;
>> 
>> You're losing the error information here - is that intentional?
>> 
> This function returns the 'struct feat_node *' object. If error happens, it
> returns NULL. The caller handles the error.

You didn't understand: Your callee has handed you error
information (a -E... value encoded as a pointer), which you
discard.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 14/25] x86: refactor psr: CDP: implement CPU init and free flow.
  2017-03-16 11:08 ` [PATCH v9 14/25] x86: refactor psr: CDP: implement CPU init and free flow Yi Sun
@ 2017-03-27 13:58   ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 13:58 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> @@ -209,12 +212,29 @@ static DEFINE_PER_CPU(struct psr_assoc, psr_assoc);
>   * array creation. It is used to transiently store a spare node.
>   */
>  static struct feat_node *feat_l3_cat;
> +static struct feat_node *feat_l3_cdp;
>  
>  /* Common functions */
>  #define cat_default_val(len)                 \
>              ( (uint32_t)((1ul << len) - 1) )
>  
>  /*
> + * get_data - get DATA COS register value from input COS ID.
> + * @feat:        the feature node.
> + * @cos:         the COS ID.
> + */
> +#define get_cdp_data(feat, cos)              \
> +            ( (feat)->cos_reg_val[(cos) * 2] )

The comment above should have the correct macro name.

> @@ -303,20 +323,56 @@ static void cat_init_feature(struct cpuid_leaf regs,
>      cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
>      cat.cos_max = min(opt_cos_max, regs.d & CAT_COS_MAX_MASK);
>  
> -    /* cos=0 is reserved as default cbm(all bits within cbm_len are 1). */
> -    feat->cos_reg_val[0] = cat_default_val(cat.cbm_len);
> -    /*
> -     * To handle cpu offline and then online case, we need read MSRs back to
> -     * save values into cos_reg_val array.
> -     */
> -    for ( i = 1; i <= cat.cos_max; i++ )
> +    if ( type == PSR_SOCKET_L3_CDP )
> +    {

Considering this change, I think you'd better make this a switch
statement right in the earlier patch, which will avoid the need to
re-indent things here.

Apart from these many comments given earlier apply here too.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 15/25] x86: refactor psr: CDP: implement get hw info flow.
  2017-03-16 11:08 ` [PATCH v9 15/25] x86: refactor psr: CDP: implement get hw info flow Yi Sun
@ 2017-03-27 14:08   ` Jan Beulich
  2017-03-28  5:13     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 14:08 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/arch/x86/sysctl.c
> +++ b/xen/arch/x86/sysctl.c
> @@ -180,10 +180,36 @@ long arch_do_sysctl(
>  
>              ret = psr_get_info(sysctl->u.psr_cat_op.target,
>                                 PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
> -
> -            sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[PSR_INFO_IDX_CBM_LEN];
> -            sysctl->u.psr_cat_op.u.l3_info.cos_max = data[PSR_INFO_IDX_COS_MAX];
> -            sysctl->u.psr_cat_op.u.l3_info.flags   = data[PSR_INFO_IDX_FLAG];
> +            if ( !ret )
> +            {
> +                sysctl->u.psr_cat_op.u.l3_info.cbm_len =
> +                                               data[PSR_INFO_IDX_CBM_LEN];
> +                sysctl->u.psr_cat_op.u.l3_info.cos_max =
> +                                               data[PSR_INFO_IDX_COS_MAX];
> +                sysctl->u.psr_cat_op.u.l3_info.flags =
> +                                               data[PSR_INFO_IDX_FLAG];
> +            }
> +            else
> +            {
> +                /*
> +                 * Check if CDP is enabled.
> +                 *
> +                 * Per spec, L3 CAT and CDP cannot co-exist. So, we need replace
> +                 * output values to CDP's if it is enabled.
> +                 */
> +                ret = psr_get_info(sysctl->u.psr_cat_op.target,
> +                                   PSR_CBM_TYPE_L3_CODE, data,
> +                                   ARRAY_SIZE(data));

I think this could/should be done without such a strange retry
mechanism. You can find out which of the features is available,
can't you?

> +                if ( !ret )
> +                {
> +                    sysctl->u.psr_cat_op.u.l3_info.cbm_len =
> +                                               data[PSR_INFO_IDX_CBM_LEN];
> +                    sysctl->u.psr_cat_op.u.l3_info.cos_max =
> +                                               data[PSR_INFO_IDX_COS_MAX];
> +                    sysctl->u.psr_cat_op.u.l3_info.flags =
> +                                               data[PSR_INFO_IDX_FLAG];
> +                }
> +            }
>  
>              if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
>                  ret = -EFAULT;

So where is PSR_CBM_TYPE_L3_DATA being handled?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 17/25] x86: refactor psr: CDP: implement set value callback functions.
  2017-03-16 11:08 ` [PATCH v9 17/25] x86: refactor psr: CDP: implement set value callback functions Yi Sun
@ 2017-03-27 14:17   ` Jan Beulich
  2017-03-28  5:14     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 14:17 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> +static bool l3_cdp_fits_cos_max(const uint32_t val[],
> +                                const struct feat_node *feat,
> +                                unsigned int cos)
> +{
> +    if ( cos > feat->info.cat_info.cos_max &&
> +         (val[0] != get_cdp_data(feat, 0) || val[1] != get_cdp_code(feat, 0)) )
> +            /*
> +             * Exceed cos_max and value to set is not default,
> +             * return error.
> +             */
> +            return false;

Indentation. But presumably this function will go away anyway.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 19/25] x86: L2 CAT: implement get hw info flow.
  2017-03-16 11:08 ` [PATCH v9 19/25] x86: L2 CAT: implement get hw info flow Yi Sun
@ 2017-03-27 14:38   ` Jan Beulich
  2017-03-28  5:16     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 14:38 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/include/public/sysctl.h
> +++ b/xen/include/public/sysctl.h
> @@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
>  DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
>  
>  #define XEN_SYSCTL_PSR_CAT_get_l3_info               0
> +#define XEN_SYSCTL_PSR_CAT_get_l2_info               1

So is the lack of CDP here the reason why the earlier patch didn't
do anything for the _DATA part of it?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 20/25] x86: L2 CAT: implement get value flow.
  2017-03-16 11:08 ` [PATCH v9 20/25] x86: L2 CAT: implement get value flow Yi Sun
@ 2017-03-27 14:39   ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 14:39 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/include/public/domctl.h
> +++ b/xen/include/public/domctl.h
> @@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
>  #define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA    3
>  #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE    4
>  #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA    5
> +#define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM     7

Any reason you skip 6 here?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 21/25] x86: L2 CAT: implement set value flow.
  2017-03-16 11:08 ` [PATCH v9 21/25] x86: L2 CAT: implement set " Yi Sun
@ 2017-03-27 14:40   ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-27 14:40 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> --- a/xen/include/public/domctl.h
> +++ b/xen/include/public/domctl.h
> @@ -1138,6 +1138,7 @@ struct xen_domctl_psr_cat_op {
>  #define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA    3
>  #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE    4
>  #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA    5
> +#define XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM     6
>  #define XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM     7

Oh, I see now.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 09/25] x86: refactor psr: L3 CAT: set value: implement framework.
  2017-03-27  9:59   ` Jan Beulich
@ 2017-03-28  1:21     ` Yi Sun
  2017-03-28  8:21       ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28  1:21 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 03:59:32, Jan Beulich wrote:
> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/domctl.c
> > +++ b/xen/arch/x86/domctl.c
> > @@ -1437,21 +1437,21 @@ long arch_do_domctl(
> >          switch ( domctl->u.psr_cat_op.cmd )
> >          {
> >          case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
> > -            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
> > -                                 domctl->u.psr_cat_op.data,
> > -                                 PSR_CBM_TYPE_L3);
> > +            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
> > +                              (uint32_t)domctl->u.psr_cat_op.data,
> 
> The cast here is pointless, but - along the lines of the comment
> on the earlier patch - indicates a problem: You silently ignore the
> upper 32-bit the caller handed you. I think for forward
> compatibility it would be better if you checked they're zero. And
> in such a check you could then use a cast which I would not
> grumble about:
> 
>     if ( domctl->u.psr_cat_op.data != (uint32_t)domctl->u.psr_cat_op.data )
>         return -E...;
> 
Thanks for the suggestion! I will check the 'data' for both get_val/set_val.

> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
[...]

> > +static int insert_new_val_to_array(uint32_t val[],
> 
> insert_new_val_into_array()?
>
> (and whether "new" needs to be part of the name is then even
> questionable)
> 
Hmm, will remove 'new'.
 
> > +static int find_cos(const uint32_t val[], uint32_t array_len,
> > +                    enum psr_feat_type feat_type,
> > +                    const struct psr_socket_info *info)
> > +{
> > +    ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
> 
> Excuse me, but no, this is another of those really bad casts. I
> guess you've added it to deal with info being pointer-to-const.
> In such a case you should instead drop the const, unless the
> consuming function can be made take a pointer-to-const (which
> isn't the case here, as check_lock() wants to write into the
> structure).
> 
Sorry for this. Will pass '&info->ref_lock' as a parameter into the function
to check it.

[...]
> > +    ref = info->cos_ref;
> > +
> > +    /*
> > +     * Step 1:
> > +     * Gather a value array to store all features cos_reg_val[old_cos].
> > +     * And, set the input new val into array according to the feature's
> > +     * position in array.
> > +     */
> > +    array_len = get_cos_num(info);
> > +    val_array = xzalloc_array(uint32_t, array_len);
> > +    if ( !val_array )
> > +        return -ENOMEM;
> > +
> > +    if ( (ret = gather_val_array(val_array, array_len, info, old_cos)) != 0 )
> > +        goto free_array;
> > +
> > +    if ( (ret = insert_new_val_to_array(val_array, array_len, info,
> > +                                        feat_type, type, val)) != 0 )
> > +        goto free_array;
> > +
> > +    spin_lock(&info->ref_lock);
> 
> Am I right in understanding that up to here operations are not
> risking to race merely because they're inside the domctl lock? If so,
> this needs to be spelled out, so we have a chance of noticing the
> dependency when we break up that lock (which we have plans for
> doing).
> 
Yes, you are right. I will add comments.

> > +    /*
> > +     * Step 2:
> > +     * Try to find if there is already a COS ID on which all features' values
> 
> Btw., I think this spelling ("features' values") is the right one - please
> go through and make it consistent everywhere, including in the patch
> description(s).
> 
Sure, I will do it.

> > +     * are same as the array. Then, we can reuse this COS ID.
> > +     */
> > +    cos = find_cos(val_array, array_len, feat_type, info);
> > +    if ( cos == old_cos )
> > +    {
> > +        ret = 0;
> > +        goto unlock_free_array;
> > +    }
> > +    else if ( cos >= 0 )
> 
> Pointless "else".
> 
Ok.

> > +        goto cos_found;
> 
> I think it would be better not to use goto here, other than for the
> error handling parts (where I don't really like it either, but I do
> accept it since others think that's the least ugly way).
> 
Sorry, I don't understand your meaning here. DYM I should goto error handling
route? 'cos >= 0' means a valid cos id has been found. Then, we should do the
'cos_found' process to handle ref and set cos id into domain.

> > +    /*
> > +     * Step 3:
> > +     * If fail to find, we need pick an available COS ID.
> 
> I think you've corrected this somewhat strange wording at the start
> of the comment here elsewhere already. I can only repeat that
> respective comments given for one location should always be
> extended to the entire series.
> 
I will try best to check entire series to keep it consistent.

> > +     * In fact, only COS ID which ref is 1 or 0 can be picked for current
> > +     * domain. If old_cos is not 0 and its ref==1, that means only current
> > +     * domain is using this old_cos ID. So, this old_cos ID certainly can
> > +     * be reused by current domain. Ref==0 means there is no any domain
> > +     * using this COS ID. So it can be used for current domain too.
> > +     */
> > +    cos = pick_avail_cos(info, val_array, array_len, old_cos, feat_type);
> > +    if ( cos < 0 )
> > +    {
> > +        ret = cos;
> > +        goto unlock_free_array;
> > +    }
> > +
> > +    /*
> > +     * Step 4:
> > +     * Write all features MSRs according to the COS ID.
> > +     */
> > +    ret = write_psr_msr(socket, cos, val, type, feat_type);
> > +    if ( ret )
> > +        goto unlock_free_array;
> > +
> > +cos_found:
> 
> Style (also the other labels further down).
> 
Will correct all of them.

> > +    /*
> > +     * Step 5:
> > +     * Update ref according to COS ID.
> > +     */
> > +    ref[cos]++;
> > +    ASSERT(!cos || ref[cos]);
> > +    ASSERT(!old_cos || ref[old_cos]);
> > +    ref[old_cos]--;
> > +    spin_unlock(&info->ref_lock);
> > +
> > +    /*
> > +     * Step 6:
> > +     * Save the COS ID into current domain's psr_cos_ids[] so that we can know
> > +     * which COS the domain is using on the socket. One domain can only use
> > +     * one COS ID at same time on each socket.
> > +     */
> > +    d->arch.psr_cos_ids[socket] = cos;
> > +    goto free_array;
> > +
> > +unlock_free_array:
> > +    spin_unlock(&info->ref_lock);
> > +free_array:
> > +    xfree(val_array);
> > +    return ret;
> > +}
> 
> I think overall things would look better if the successful path was the
> straight (goto-free) one.
> 
DYM change 'free_array' to 'free'?

> >  /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
> >  static void psr_free_cos(struct domain *d)
> >  {
> > +    unsigned int socket, cos;
> > +
> > +    if ( !socket_info || !d->arch.psr_cos_ids )
> > +        return;
> 
> Can d->arch.psr_cos_ids be non-NULL when !socket_info? If not,
> check only the former with an if(), and ASSERT() the latter.
> 
In normal case, 'psr_cos_ids' should not be NULL when entering this function.
So, I think ASSERT() is a good option. Thanks!

> > +    /* Domain is destroied so its cos_ref should be decreased. */
> > +    for ( socket = 0; socket < nr_sockets; socket++ )
> > +    {
> > +        struct psr_socket_info *info;
> > +
> > +        /* cos 0 is default one which does not need be handled. */
> > +        cos = d->arch.psr_cos_ids[socket];
> > +        if ( cos == 0 )
> > +            continue;
> > +
> > +        /*
> > +         * If domain uses other cos ids, all corresponding refs must have been
> > +         * increased 1 for this domain. So, we need decrease them.
> 
> ... by 1 ... need to ... I also question the presence of the word
> "must" in here.
> 
After considering this comments again, I think the purpose of this piece of
codes has been explained outer the circle. So, I think this comment can be
removed. Is that OK for you?

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow.
  2017-03-27 13:34       ` Jan Beulich
@ 2017-03-28  2:13         ` Yi Sun
  2017-03-28  8:10           ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28  2:13 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 07:34:43, Jan Beulich wrote:
> >>> On 27.03.17 at 14:59, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-27 03:23:08, Jan Beulich wrote:
> >> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
> >> > --- a/xen/arch/x86/domctl.c
> >> > +++ b/xen/arch/x86/domctl.c
> >> > @@ -1455,23 +1455,26 @@ long arch_do_domctl(
> >> >              break;
> >> >  
> >> >          case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
> >> > -            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
> >> > -                                 &domctl->u.psr_cat_op.data,
> >> > -                                 PSR_CBM_TYPE_L3);
> >> > +            domctl->u.psr_cat_op.data = 0;
> >> > +            ret = psr_get_val(d, domctl->u.psr_cat_op.target,
> >> > +                              (uint32_t *)&domctl->u.psr_cat_op.data,
> >> 
> >> This is exactly why I generally object to casts: The high half of
> >> the field will remain untouched, likely confusing the caller. You
> >> need to decide at what layer you want to do the extension from
> >> the internally used 32-bit type to the public interface induced
> >> 64-bit one.
> >> 
> > 'psr_cat_op.data' is used as interface between tools/ and hyperviosr. We
> > defined it as 'uint64_t' to fulfill future requests because MSRs registers
> > are 64bit although the upper 32bit are not used yet.
> > 
> > Per your suggetion to use 'uint32_t' internally for CBM, I changed
> > psr_get_val/psr_set_val parameters type from 'uint64_t' to 'uint32_t'. That
> > is the reason to do cast here. Is this an appropriate choice?
> 
> No, as said, it is not. The choice of types is fine, but you need to
> make this work without casts (i.e. presumably with some
> intermediate variable at one of the layers).
> 
Then, I have to use a local variable to do this.

> >> > @@ -504,21 +515,30 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
> >> >      return socket_info + socket;
> >> >  }
> >> >  
> >> > -int psr_get_info(unsigned int socket, enum cbm_type type,
> >> > -                 uint32_t data[], unsigned int array_len)
> >> > +static const struct feat_node * psr_get_feat(unsigned int socket,
> >> > +                                             enum cbm_type type)
> >> >  {
> >> >      const struct psr_socket_info *info = get_socket_info(socket);
> >> >      const struct feat_node *feat;
> >> >      enum psr_feat_type feat_type;
> >> >  
> >> >      if ( IS_ERR(info) )
> >> > -        return PTR_ERR(info);
> >> > +        return NULL;
> >> 
> >> You're losing the error information here - is that intentional?
> >> 
> > This function returns the 'struct feat_node *' object. If error happens, it
> > returns NULL. The caller handles the error.
> 
> You didn't understand: Your callee has handed you error
> information (a -E... value encoded as a pointer), which you
> discard.
> 
Thanks for explanation! I will modify the type of 'psr_get_feat' to be able to
return such error back.

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-27 10:17   ` Jan Beulich
@ 2017-03-28  3:12     ` Yi Sun
  2017-03-28  8:05       ` Yi Sun
  2017-03-28  8:34       ` Jan Beulich
  0 siblings, 2 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-28  3:12 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 04:17:28, Jan Beulich wrote:
> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
> > @@ -101,6 +101,28 @@ struct feat_node {
> >          /* get_val is used to get feature COS register value. */
> >          void (*get_val)(const struct feat_node *feat, unsigned int cos,
> >                          enum cbm_type type, uint32_t *val);
> > +
> > +        /*
> > +         * get_old_val and set_new_val are a pair of functions called in order.
> > +         * The caller will traverse all features in the array and call
> > +         * 'get_old_val' to get old_cos register value of all supported
> > +         * features. Then, call 'set_new_val' to set the new value for the
> > +         * designated feature.
> > +         *
> > +         * All the values are set into value array according to the traversal
> > +         * order, meaning the same order of feature array members.
> > +         *
> > +         * The return value meaning of set_new_val:
> > +         * 0 - success.
> > +         * negative - error.
> > +         */
> > +        void (*get_old_val)(uint32_t val[],
> > +                            const struct feat_node *feat,
> > +                            unsigned int old_cos);
> > +        int (*set_new_val)(uint32_t val[],
> > +                           const struct feat_node *feat,
> > +                           enum cbm_type type,
> > +                           uint32_t new_val);
> 
> Along the lines of an earlier comment - are "old" and "new" really
> meaningful here?
> 
Maybe 'old' is not accurate. How about 'current'? In fact, we use this
function to get domain's current CBM value. Furthermore, this is to distinguish
'get_val' which is declared above.

I think 'new' is meaningful to express we are setting the newly input value.

How do you think? Thanks!

> > @@ -212,6 +234,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
> >  }
> >  
> >  /* CAT common functions implementation. */
> > +static bool psr_check_cbm(unsigned int cbm_len, uint32_t cbm)
> > +{
> > +    unsigned int first_bit, zero_bit;
> > +
> > +    /* Set bits should only in the range of [0, cbm_len]. */
> > +    if ( cbm & (~0ul << cbm_len) )
> 
> Same question as elsewhere about the use of the ul suffix here:
> Can cbm_len really be any value in [0,32]? If not, I don't see
> why the calculation needs to be done as unsigned long. Otoh ...
> 
cbm_len is got as below:
#define CAT_CBM_LEN_MASK 0x1f
cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;

So its max value is 32.

> > +        return false;
> > +
> > +    /* At least one bit need to be set. */
> > +    if ( cbm == 0 )
> > +        return false;
> > +
> > +    first_bit = find_first_bit((uint64_t *)&cbm, cbm_len);
> > +    zero_bit = find_next_zero_bit((uint64_t *)&cbm, cbm_len, first_bit);
> 
> ... these bogus casts suggest that the function would best have
> an "unsigned long" parameter.
> 
I would like to modify 'cbm' type to 'uint64_t'. Use a local variable in caller
to do the type conversion. 

> > @@ -285,11 +330,35 @@ static void cat_get_val(const struct feat_node *feat, unsigned int cos,
> >      *val = feat->cos_reg_val[cos];
> >  }
> >  

[...]
> >  static int gather_val_array(uint32_t val[],
> > @@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
> >                              const struct psr_socket_info *info,
> >                              unsigned int old_cos)
> >  {
> > -    return -EINVAL;
> > +    const struct feat_node *feat;
> > +    unsigned int i;
> > +
> > +    if ( !val )
> > +        return -EINVAL;
> > +
> > +    /* Get all features current values according to old_cos. */
> > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> > +    {
> > +        if ( !info->features[i] )
> > +            continue;
> > +
> > +        feat = info->features[i];
> > +
> > +        if ( old_cos > feat->ops.get_cos_max(feat) )
> > +            old_cos = 0;
> > +
> > +        /* value getting order is same as feature array */
> > +        feat->ops.get_old_val(val, feat, old_cos);
> > +
> > +        array_len -= feat->cos_num;
> 
> So this I should really have asked about on a much earlier patch,
> but I've recognize the oddity only now: Why is cos_num
> per-feature-node instead of per-feature? This should really be a
> field in struct feat_ops (albeit the name "ops" then will be slightly
> misleading, but I think that's tolerable if you can't think of a better
> name).
> 
Ok, I got your meaning. How about 'feat_props'? No matter operations or
variables are all properties of the feature.

> > +        if ( array_len < 0 )
> > +            return -ENOSPC;
> 
> This check needs doing earlier - you need to make sure array_len
> >= ops.cos_num prior to calling ops.get_old_val(). (Doing the
> check after the subtraction even causes wrapping issues, which
> are even more visible in similar code further down.)
> 
Thanks for the suggestion! Will move 'array_len >= cos_num' check prior to
call 'get_old_val'.

> > @@ -599,7 +709,43 @@ static int insert_new_val_to_array(uint32_t val[],
> >                                     enum cbm_type type,
> >                                     uint32_t new_val)
> >  {
> > -    return -EINVAL;
> > +    const struct feat_node *feat;
> > +    int ret;
> > +    unsigned int i;
> > +
> > +    ASSERT(feat_type < PSR_SOCKET_MAX_FEAT);
> > +
> > +    /* Set new value into array according to feature's position in array. */
> > +    for ( i = 0; i < feat_type; i++ )
> > +    {
> > +        if ( !info->features[i] )
> > +            continue;
> > +
> > +        feat = info->features[i];
> > +
> > +        array_len -= feat->cos_num;
> > +        if ( array_len <= 0 )
> > +            return -ENOSPC;
> > +
> > +        val += feat->cos_num;
> > +    }
> > +
> > +    feat = info->features[feat_type];
> > +
> > +    array_len -= feat->cos_num;
> > +    if ( array_len < 0 )
> > +        return -ENOSPC;
> > +
> > +    /*
> > +     * Value setting position is same as feature array.
> > +     * Different features may have different setting behaviors, e.g. CDP
> > +     * has two values (DATA/CODE) which need us to save input value to
> > +     * different position in the array according to type, so we have to
> > +     * maintain a callback function.
> > +     */
> > +    ret = feat->ops.set_new_val(val, feat, type, new_val);
> > +
> > +    return ret;
> 
> Again a case of a pointless intermediate variable.
> 
Will remove it.

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 11/25] x86: refactor psr: L3 CAT: set value: implement cos finding flow.
  2017-03-27 10:28   ` Jan Beulich
@ 2017-03-28  3:26     ` Yi Sun
  2017-03-28  8:41       ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28  3:26 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 04:28:23, Jan Beulich wrote:
> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/psr.c
> > +++ b/xen/arch/x86/psr.c
[...]
 
> > +static int cat_compare_val(const uint32_t val[],
> > +                           const struct feat_node *feat,
> > +                           unsigned int cos)
> > +{
> > +    /*
> > +     * Different features' cos_max are different. If cos id of the feature
> > +     * being set exceeds other feature's cos_max, the val of other feature
> > +     * must be default value. HW supports such case.
> > +     */
> > +    if ( cos > feat->info.cat_info.cos_max )
> > +    {
> > +        /* cos_reg_val[0] is the default value. */
> > +        if ( val[0] != feat->cos_reg_val[0] )
> > +            return -EINVAL;
> 
> As you can see, with cos_max moved into the generic portion of the
> feature node, this entire check can move into the caller.
> 
CDP has different behavior in this callback function. We need to check val[0]
and val[1] like below:
static int l3_cdp_compare_val(...)
{
    if ( cos > feat->info.cat_info.cos_max )
    {
        if ( val[0] != get_cdp_data(feat, 0) ||
             val[1] != get_cdp_code(feat, 0) )
            return -EINVAL;

        /* Find */
        return 1;
    }

    if ( val[0] == get_cdp_data(feat, cos) &&
         val[1] == get_cdp_code(feat, cos) )
        /* Find */
        return 1;
......
}

> > +        /* Find */
> 
> Found (also below)
> 
> > +        return 1;
> > +    }
> > +
> > +    if ( val[0] == feat->cos_reg_val[cos] )
> > +        /* Find */
> > +        return 1;
> > +
> > +    /* Not find */
> > +    return 0;
> > +}
> 
> Or actually, the entire function then becomes feature independent,
> as it seems. And I think I did suggest that already during review of
> an earlier version.
> 
Per above explanation, I think we have to keep this callback function.

> > @@ -752,7 +793,61 @@ static int find_cos(const uint32_t val[], uint32_t array_len,
> >                      enum psr_feat_type feat_type,
> >                      const struct psr_socket_info *info)
> >  {
> > +    unsigned int cos, i;
> > +    const unsigned int *ref = info->cos_ref;
> > +    const struct feat_node *feat;
> > +    const uint32_t *val_array = val;
> 
> The name doesn't match the purpose - as you increment the pointer,
> its name should rather be "val_ptr" or some such.
> 
Got it, thanks!

> > +    int find = 0;
> 
> "found" again, or even simply "rc"? Also I think this would better
> move into the outer for() scope.
> 
Ok, will use 'found' and move it.

> > +    unsigned int cos_max;
> > +
> >      ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
> > +
> > +    /* cos_max is the one of the feature which is being set. */
> > +    feat = info->features[feat_type];
> > +    if ( !feat )
> > +        return -ENOENT;
> > +
> > +    cos_max = feat->ops.get_cos_max(feat);
> > +
> > +    for ( cos = 0; cos <= cos_max; cos++ )
> > +    {
> > +        if ( cos && !ref[cos] )
> > +            continue;
> > +
> > +        /*
> > +         * If fail to find cos in below loop, need find whole feature array
> > +         * again from beginning.
> > +         */
> > +        val_array = val;
> 
> You wouldn't need to re-do this here if you moved the variable
> declaration (with initializer) into this scope. This then also
> eliminates the need for the comment, which otherwise would
> need its wording corrected.
> 
Ok, thanks!

> > +        for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> > +        {
> > +            if ( !info->features[i] )
> > +                continue;
> > +
> > +            feat = info->features[i];
> 
> Please swap if() and assignment, utilizing the local variable in the
> if().
> 
Ok, thanks!

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-27 10:37   ` Jan Beulich
@ 2017-03-28  4:58     ` Yi Sun
  2017-03-28  8:45       ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28  4:58 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 04:37:37, Jan Beulich wrote:
> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> > +static bool cat_fits_cos_max(const uint32_t val[],
> > +                             const struct feat_node *feat,
> > +                             unsigned int cos)
> > +{
> > +    if ( cos > feat->info.cat_info.cos_max &&
> > +         val[0] != feat->cos_reg_val[0] )
> > +            /*
> > +             * Exceed cos_max and value to set is not default,
> > +             * return error.
> > +             */
> > +            return false;
> > +
> > +    return true;
> > +}
> 
> Same here - with cos_max moved out, the hook would seem to
> become unnecessary.
> 
As explanation in previous patch, CDP has different behavior.
static bool l3_cdp_fits_cos_max(...)
{
    if ( cos > feat->info.cat_info.cos_max &&
         (val[0] != get_cdp_data(feat, 0) || val[1] != get_cdp_code(feat, 0)) )
            /*
             * Exceed cos_max and value to set is not default,
             * return error.
             */
            return false;

    return true;

}

> >  static int pick_avail_cos(const struct psr_socket_info *info,
> >                            const uint32_t val[], uint32_t array_len,
> >                            unsigned int old_cos,
> >                            enum psr_feat_type feat_type)
> >  {
> > +    unsigned int cos;
> > +    unsigned int cos_max = 0;
> > +    const struct feat_node *feat;
> > +    const unsigned int *ref = info->cos_ref;
> > +
> >      ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
> > -    return -ENOENT;
> > +
> > +    /* cos_max is the one of the feature which is being set. */
> > +    feat = info->features[feat_type];
> > +    if ( !feat )
> > +        return -ENOENT;
> > +
> > +    cos_max = feat->ops.get_cos_max(feat);
> > +    if ( !cos_max )
> > +        return -ENOENT;
> > +
> > +    /*
> > +     * If old cos is referred only by the domain, then use it. And, we cannot
> 
> "the domain" here is lacking context - there's no domain involved

How about "the domain input through 'psr_set_val'"?

> in the function - "... the domain currently using it, then re-use it"?
> 
Ok, 're-use it' is better.

> > +     * use id 0 because it stores the default values.
> > +     */
> > +    if ( old_cos && ref[old_cos] == 1 &&
> > +         fits_cos_max(val, array_len, info, old_cos) )
> > +            return old_cos;
> > +
> > +    /* Find an unused one other than cos0. */
> > +    for ( cos = 1; cos <= cos_max; cos++ )
> > +    {
> > +        /*
> > +         * ref is 0 means this COS is not used by other domain and
> > +         * can be used for current setting.
> > +         */
> > +        if ( !ref[cos] )
> > +        {
> > +            if ( !fits_cos_max(val, array_len, info, cos) )
> > +                return -EOVERFLOW;
> 
> Perhaps better use "break;" here.
> 
Oh yes, thanks!

> > +            return cos;
> > +        }
> > +    }
> > +
> > +    return -EOVERFLOW;
> >  }
> 
> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 13/25] x86: refactor psr: L3 CAT: set value: implement write msr flow.
  2017-03-27 10:46   ` Jan Beulich
@ 2017-03-28  5:06     ` Yi Sun
  2017-03-28  8:48       ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28  5:06 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 04:46:01, Jan Beulich wrote:
> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> > @@ -421,6 +425,18 @@ static bool cat_fits_cos_max(const uint32_t val[],
> >  }
> >  
> >  /* L3 CAT ops */
> > +static void l3_cat_write_msr(unsigned int cos, uint32_t val,
> > +                             enum cbm_type type, struct feat_node *feat)
> 
> "type" is an unused parameter. Please remove it from the hook
> and this function.
> 
This is used by CDP callback function. We need it to decide whether CODE
or DATA is written.

[...]
> > +struct cos_write_info
> > +{
> > +    unsigned int cos;
> > +    struct feat_node *feature;
> > +    uint32_t val;
> > +    enum cbm_type type;
> 
> With the "type" parameter removed above, this looks to be an
> unused field then too. After the removal of it, please re-order
> fields to leave no holes.
> 
Per above comment, 'type' is needed.

> > +static void do_write_psr_msr(void *data)
> > +{
> > +    struct cos_write_info *info = (struct cos_write_info *)data;
> 
> Unnecessary cast again.
> 
Thanks!

> > +    unsigned int cos            = info->cos;
> > +    struct feat_node *feat      = info->feature;
> > +
> > +    if ( !feat )
> > +        return;
> 
> You shouldn't even call this function when !feat.
> 
Ok, may I move the check to the 'write_psr_msr'?

> > +    if ( cos > feat->ops.get_cos_max(feat) )
> > +        return;
> > +
> > +    feat->ops.write_msr(cos, info->val, info->type, feat);
> > +}
> > +
> >  static int write_psr_msr(unsigned int socket, unsigned int cos,
> >                           uint32_t val, enum cbm_type type,
> 
> "type", according to the above comments, should then go away
> here too as it seems.
> 
Please check above comments.

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 15/25] x86: refactor psr: CDP: implement get hw info flow.
  2017-03-27 14:08   ` Jan Beulich
@ 2017-03-28  5:13     ` Yi Sun
  0 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-28  5:13 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 08:08:13, Jan Beulich wrote:
> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/arch/x86/sysctl.c
> > +++ b/xen/arch/x86/sysctl.c
> > @@ -180,10 +180,36 @@ long arch_do_sysctl(
> >  
> >              ret = psr_get_info(sysctl->u.psr_cat_op.target,
> >                                 PSR_CBM_TYPE_L3, data, ARRAY_SIZE(data));
> > -
> > -            sysctl->u.psr_cat_op.u.l3_info.cbm_len = data[PSR_INFO_IDX_CBM_LEN];
> > -            sysctl->u.psr_cat_op.u.l3_info.cos_max = data[PSR_INFO_IDX_COS_MAX];
> > -            sysctl->u.psr_cat_op.u.l3_info.flags   = data[PSR_INFO_IDX_FLAG];
> > +            if ( !ret )
> > +            {
> > +                sysctl->u.psr_cat_op.u.l3_info.cbm_len =
> > +                                               data[PSR_INFO_IDX_CBM_LEN];
> > +                sysctl->u.psr_cat_op.u.l3_info.cos_max =
> > +                                               data[PSR_INFO_IDX_COS_MAX];
> > +                sysctl->u.psr_cat_op.u.l3_info.flags =
> > +                                               data[PSR_INFO_IDX_FLAG];
> > +            }
> > +            else
> > +            {
> > +                /*
> > +                 * Check if CDP is enabled.
> > +                 *
> > +                 * Per spec, L3 CAT and CDP cannot co-exist. So, we need replace
> > +                 * output values to CDP's if it is enabled.
> > +                 */
> > +                ret = psr_get_info(sysctl->u.psr_cat_op.target,
> > +                                   PSR_CBM_TYPE_L3_CODE, data,
> > +                                   ARRAY_SIZE(data));
> 
> I think this could/should be done without such a strange retry
> mechanism. You can find out which of the features is available,
> can't you?
> 
Ok, I can handle it inside psr_get_info. If L3 CAT feature node is NULL, will
use CDP feature node.

> > +                if ( !ret )
> > +                {
> > +                    sysctl->u.psr_cat_op.u.l3_info.cbm_len =
> > +                                               data[PSR_INFO_IDX_CBM_LEN];
> > +                    sysctl->u.psr_cat_op.u.l3_info.cos_max =
> > +                                               data[PSR_INFO_IDX_COS_MAX];
> > +                    sysctl->u.psr_cat_op.u.l3_info.flags =
> > +                                               data[PSR_INFO_IDX_FLAG];
> > +                }
> > +            }
> >  
> >              if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
> >                  ret = -EFAULT;
> 
> So where is PSR_CBM_TYPE_L3_DATA being handled?
> 
This interface is to get L3 CAT/CDP HW info. So, either PSR_CBM_TYPE_L3_DATA or
PSR_CBM_TYPE_L3_CODE can be used to get CDP's HW info. Either of them will be
converted to feature type 'PSR_SOCKET_L3_CDP'.

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 17/25] x86: refactor psr: CDP: implement set value callback functions.
  2017-03-27 14:17   ` Jan Beulich
@ 2017-03-28  5:14     ` Yi Sun
  0 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-28  5:14 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-27 08:17:04, Jan Beulich wrote:
> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> > +static bool l3_cdp_fits_cos_max(const uint32_t val[],
> > +                                const struct feat_node *feat,
> > +                                unsigned int cos)
> > +{
> > +    if ( cos > feat->info.cat_info.cos_max &&
> > +         (val[0] != get_cdp_data(feat, 0) || val[1] != get_cdp_code(feat, 0)) )
> > +            /*
> > +             * Exceed cos_max and value to set is not default,
> > +             * return error.
> > +             */
> > +            return false;
> 
> Indentation. But presumably this function will go away anyway.
> 
As explanation in previous patch, this CDP callback function is different
with CAT. So, I am afraid we have to keep it.

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 19/25] x86: L2 CAT: implement get hw info flow.
  2017-03-27 14:38   ` Jan Beulich
@ 2017-03-28  5:16     ` Yi Sun
  0 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-28  5:16 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau

On 17-03-27 08:38:39, Jan Beulich wrote:
> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> > --- a/xen/include/public/sysctl.h
> > +++ b/xen/include/public/sysctl.h
> > @@ -744,6 +744,7 @@ typedef struct xen_sysctl_pcitopoinfo xen_sysctl_pcitopoinfo_t;
> >  DEFINE_XEN_GUEST_HANDLE(xen_sysctl_pcitopoinfo_t);
> >  
> >  #define XEN_SYSCTL_PSR_CAT_get_l3_info               0
> > +#define XEN_SYSCTL_PSR_CAT_get_l2_info               1
> 
> So is the lack of CDP here the reason why the earlier patch didn't
> do anything for the _DATA part of it?
> 
As explanation in previous patch, either _DATA or _CODE can be used to get
CDP HW info.

> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-28  3:12     ` Yi Sun
@ 2017-03-28  8:05       ` Yi Sun
  2017-03-28  8:36         ` Jan Beulich
  2017-03-28  8:34       ` Jan Beulich
  1 sibling, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28  8:05 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau

On 17-03-28 11:12:43, Yi Sun wrote:
> On 17-03-27 04:17:28, Jan Beulich wrote:
> > >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> > > --- a/xen/arch/x86/psr.c
> > > +++ b/xen/arch/x86/psr.c
[...]

> > >  static int gather_val_array(uint32_t val[],
> > > @@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
> > >                              const struct psr_socket_info *info,
> > >                              unsigned int old_cos)
> > >  {
> > > -    return -EINVAL;
> > > +    const struct feat_node *feat;
> > > +    unsigned int i;
> > > +
> > > +    if ( !val )
> > > +        return -EINVAL;
> > > +
> > > +    /* Get all features current values according to old_cos. */
> > > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> > > +    {
> > > +        if ( !info->features[i] )
> > > +            continue;
> > > +
> > > +        feat = info->features[i];
> > > +
> > > +        if ( old_cos > feat->ops.get_cos_max(feat) )
> > > +            old_cos = 0;
> > > +
> > > +        /* value getting order is same as feature array */
> > > +        feat->ops.get_old_val(val, feat, old_cos);
> > > +
> > > +        array_len -= feat->cos_num;
> > 
> > So this I should really have asked about on a much earlier patch,
> > but I've recognize the oddity only now: Why is cos_num
> > per-feature-node instead of per-feature? This should really be a
> > field in struct feat_ops (albeit the name "ops" then will be slightly
> > misleading, but I think that's tolerable if you can't think of a better
> > name).
> > 
> Ok, I got your meaning. How about 'feat_props'? No matter operations or
> variables are all properties of the feature.
> 
One more thing here. If we move 'cos_max' into 'feat_ops', we cannot declare
'feat_ops' as const. Because we have to assign value to 'cos_max' in
cat_init_feature().

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow.
  2017-03-28  2:13         ` Yi Sun
@ 2017-03-28  8:10           ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-28  8:10 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 04:13, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 07:34:43, Jan Beulich wrote:
>> >>> On 27.03.17 at 14:59, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-27 03:23:08, Jan Beulich wrote:
>> >> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
>> >> > @@ -504,21 +515,30 @@ static struct psr_socket_info *get_socket_info(unsigned int socket)
>> >> >      return socket_info + socket;
>> >> >  }
>> >> >  
>> >> > -int psr_get_info(unsigned int socket, enum cbm_type type,
>> >> > -                 uint32_t data[], unsigned int array_len)
>> >> > +static const struct feat_node * psr_get_feat(unsigned int socket,
>> >> > +                                             enum cbm_type type)
>> >> >  {
>> >> >      const struct psr_socket_info *info = get_socket_info(socket);
>> >> >      const struct feat_node *feat;
>> >> >      enum psr_feat_type feat_type;
>> >> >  
>> >> >      if ( IS_ERR(info) )
>> >> > -        return PTR_ERR(info);
>> >> > +        return NULL;
>> >> 
>> >> You're losing the error information here - is that intentional?
>> >> 
>> > This function returns the 'struct feat_node *' object. If error happens, it
>> > returns NULL. The caller handles the error.
>> 
>> You didn't understand: Your callee has handed you error
>> information (a -E... value encoded as a pointer), which you
>> discard.
>> 
> Thanks for explanation! I will modify the type of 'psr_get_feat' to be able to
> return such error back.

Why would you need to modify its type? Just pass on the error
indicator, instead of returning NULL. It's the caller(s) who need(s)
to change.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 09/25] x86: refactor psr: L3 CAT: set value: implement framework.
  2017-03-28  1:21     ` Yi Sun
@ 2017-03-28  8:21       ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-28  8:21 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 03:21, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 03:59:32, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:07, <yi.y.sun@linux.intel.com> wrote:
>> > --- a/xen/arch/x86/domctl.c
>> > +++ b/xen/arch/x86/domctl.c
>> > @@ -1437,21 +1437,21 @@ long arch_do_domctl(
>> >          switch ( domctl->u.psr_cat_op.cmd )
>> >          {
>> >          case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
>> > -            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
>> > -                                 domctl->u.psr_cat_op.data,
>> > -                                 PSR_CBM_TYPE_L3);
>> > +            ret = psr_set_val(d, domctl->u.psr_cat_op.target,
>> > +                              (uint32_t)domctl->u.psr_cat_op.data,
>> 
>> The cast here is pointless, but - along the lines of the comment
>> on the earlier patch - indicates a problem: You silently ignore the
>> upper 32-bit the caller handed you. I think for forward
>> compatibility it would be better if you checked they're zero. And
>> in such a check you could then use a cast which I would not
>> grumble about:
>> 
>>     if ( domctl->u.psr_cat_op.data != (uint32_t)domctl->u.psr_cat_op.data )
>>         return -E...;
>> 
> Thanks for the suggestion! I will check the 'data' for both get_val/set_val.

There's no data to check on the "get" path - you return data there.

>> > +static int find_cos(const uint32_t val[], uint32_t array_len,
>> > +                    enum psr_feat_type feat_type,
>> > +                    const struct psr_socket_info *info)
>> > +{
>> > +    ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
>> 
>> Excuse me, but no, this is another of those really bad casts. I
>> guess you've added it to deal with info being pointer-to-const.
>> In such a case you should instead drop the const, unless the
>> consuming function can be made take a pointer-to-const (which
>> isn't the case here, as check_lock() wants to write into the
>> structure).
>> 
> Sorry for this. Will pass '&info->ref_lock' as a parameter into the function
> to check it.

That's suitable if you don't need "info" for anything else. Otherwise
- as said - you should drop const from info.

>> > +     * are same as the array. Then, we can reuse this COS ID.
>> > +     */
>> > +    cos = find_cos(val_array, array_len, feat_type, info);
>> > +    if ( cos == old_cos )
>> > +    {
>> > +        ret = 0;
>> > +        goto unlock_free_array;
>> > +    }
>> > +    else if ( cos >= 0 )
>> 
>> Pointless "else".
>> 
> Ok.
> 
>> > +        goto cos_found;
>> 
>> I think it would be better not to use goto here, other than for the
>> error handling parts (where I don't really like it either, but I do
>> accept it since others think that's the least ugly way).
>> 
> Sorry, I don't understand your meaning here. DYM I should goto error handling
> route? 'cos >= 0' means a valid cos id has been found. Then, we should do the
> 'cos_found' process to handle ref and set cos id into domain.

Using goto for error handling is acceptable, as said. But please use

    if ( cos < 0 )
    {
    }

followed by the code that's now after the cos_found label.

>> > +    /*
>> > +     * Step 5:
>> > +     * Update ref according to COS ID.
>> > +     */
>> > +    ref[cos]++;
>> > +    ASSERT(!cos || ref[cos]);
>> > +    ASSERT(!old_cos || ref[old_cos]);
>> > +    ref[old_cos]--;
>> > +    spin_unlock(&info->ref_lock);
>> > +
>> > +    /*
>> > +     * Step 6:
>> > +     * Save the COS ID into current domain's psr_cos_ids[] so that we can know
>> > +     * which COS the domain is using on the socket. One domain can only use
>> > +     * one COS ID at same time on each socket.
>> > +     */
>> > +    d->arch.psr_cos_ids[socket] = cos;
>> > +    goto free_array;
>> > +
>> > +unlock_free_array:
>> > +    spin_unlock(&info->ref_lock);
>> > +free_array:
>> > +    xfree(val_array);
>> > +    return ret;
>> > +}
>> 
>> I think overall things would look better if the successful path was the
>> straight (goto-free) one.
>> 
> DYM change 'free_array' to 'free'?

My remark was not about label names, but about code flow: Please
use goto only for error paths, unless not using it on normal paths
means unacceptable resulting code (use your own judgment).

>> >  /* Called with domain lock held, no extra lock needed for 'psr_cos_ids' */
>> >  static void psr_free_cos(struct domain *d)
>> >  {
>> > +    unsigned int socket, cos;
>> > +
>> > +    if ( !socket_info || !d->arch.psr_cos_ids )
>> > +        return;
>> 
>> Can d->arch.psr_cos_ids be non-NULL when !socket_info? If not,
>> check only the former with an if(), and ASSERT() the latter.
>> 
> In normal case, 'psr_cos_ids' should not be NULL when entering this function.
> So, I think ASSERT() is a good option. Thanks!

What does "normal" mean to you here? If, e.g. due to an error
during domain creation, we can come here with the pointer
being NULL, you must not ASSERT() but use if() instead. Please
note that my earlier comment did _not_ suggest to replace the
checking of d->arch.psr_cos_ids, but that of socket_info (as
long as the answer to the question is "no").

>> > +    /* Domain is destroied so its cos_ref should be decreased. */
>> > +    for ( socket = 0; socket < nr_sockets; socket++ )
>> > +    {
>> > +        struct psr_socket_info *info;
>> > +
>> > +        /* cos 0 is default one which does not need be handled. */
>> > +        cos = d->arch.psr_cos_ids[socket];
>> > +        if ( cos == 0 )
>> > +            continue;
>> > +
>> > +        /*
>> > +         * If domain uses other cos ids, all corresponding refs must have been
>> > +         * increased 1 for this domain. So, we need decrease them.
>> 
>> ... by 1 ... need to ... I also question the presence of the word
>> "must" in here.
>> 
> After considering this comments again, I think the purpose of this piece of
> codes has been explained outer the circle. So, I think this comment can be
> removed. Is that OK for you?

If "outer the circle" means "outside of the loop", then yes, that earlier
comment should be sufficient on its own.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-28  3:12     ` Yi Sun
  2017-03-28  8:05       ` Yi Sun
@ 2017-03-28  8:34       ` Jan Beulich
  2017-03-28 10:12         ` Yi Sun
  1 sibling, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-28  8:34 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 05:12, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 04:17:28, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
>> > --- a/xen/arch/x86/psr.c
>> > +++ b/xen/arch/x86/psr.c
>> > @@ -101,6 +101,28 @@ struct feat_node {
>> >          /* get_val is used to get feature COS register value. */
>> >          void (*get_val)(const struct feat_node *feat, unsigned int cos,
>> >                          enum cbm_type type, uint32_t *val);
>> > +
>> > +        /*
>> > +         * get_old_val and set_new_val are a pair of functions called in order.
>> > +         * The caller will traverse all features in the array and call
>> > +         * 'get_old_val' to get old_cos register value of all supported
>> > +         * features. Then, call 'set_new_val' to set the new value for the
>> > +         * designated feature.
>> > +         *
>> > +         * All the values are set into value array according to the traversal
>> > +         * order, meaning the same order of feature array members.
>> > +         *
>> > +         * The return value meaning of set_new_val:
>> > +         * 0 - success.
>> > +         * negative - error.
>> > +         */
>> > +        void (*get_old_val)(uint32_t val[],
>> > +                            const struct feat_node *feat,
>> > +                            unsigned int old_cos);
>> > +        int (*set_new_val)(uint32_t val[],
>> > +                           const struct feat_node *feat,
>> > +                           enum cbm_type type,
>> > +                           uint32_t new_val);
>> 
>> Along the lines of an earlier comment - are "old" and "new" really
>> meaningful here?
>> 
> Maybe 'old' is not accurate. How about 'current'? In fact, we use this
> function to get domain's current CBM value. Furthermore, this is to distinguish
> 'get_val' which is declared above.

I'm fine with "current", but the name collision - would "current" be
omitted still bothers me. The fact that cat_get_old_val() calls
cat_get_val(), however, strongly suggests that the hook here is
redundant anyway. Even in the CDP case I think you can get
away without it, but if this turns out really impossible or clumsy,
then the hook could be introduced there (with a better name)
and be an optional one (with the caller using ->get_val() if the
one here is NULL).

> I think 'new' is meaningful to express we are setting the newly input value.

Well, this is the meaning to its caller. The function itself doesn't
care whether the value is a new one, or just some other value
coming from an unnamed source.

>> > @@ -212,6 +234,29 @@ static enum psr_feat_type psr_cbm_type_to_feat_type(enum cbm_type type)
>> >  }
>> >  
>> >  /* CAT common functions implementation. */
>> > +static bool psr_check_cbm(unsigned int cbm_len, uint32_t cbm)
>> > +{
>> > +    unsigned int first_bit, zero_bit;
>> > +
>> > +    /* Set bits should only in the range of [0, cbm_len]. */
>> > +    if ( cbm & (~0ul << cbm_len) )
>> 
>> Same question as elsewhere about the use of the ul suffix here:
>> Can cbm_len really be any value in [0,32]? If not, I don't see
>> why the calculation needs to be done as unsigned long. Otoh ...
>> 
> cbm_len is got as below:
> #define CAT_CBM_LEN_MASK 0x1f
> cat.cbm_len = (regs.a & CAT_CBM_LEN_MASK) + 1;
> 
> So its max value is 32.

And its min one is 1. I.e. no need for an unsigned long calculation.

>> > +        return false;
>> > +
>> > +    /* At least one bit need to be set. */
>> > +    if ( cbm == 0 )
>> > +        return false;
>> > +
>> > +    first_bit = find_first_bit((uint64_t *)&cbm, cbm_len);
>> > +    zero_bit = find_next_zero_bit((uint64_t *)&cbm, cbm_len, first_bit);
>> 
>> ... these bogus casts suggest that the function would best have
>> an "unsigned long" parameter.
>> 
> I would like to modify 'cbm' type to 'uint64_t'. Use a local variable in caller
> to do the type conversion. 

Why uint64_t? As said elsewhere, please stay away from fixed width
types if you don't really need them.

>> > @@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
>> >                              const struct psr_socket_info *info,
>> >                              unsigned int old_cos)
>> >  {
>> > -    return -EINVAL;
>> > +    const struct feat_node *feat;
>> > +    unsigned int i;
>> > +
>> > +    if ( !val )
>> > +        return -EINVAL;
>> > +
>> > +    /* Get all features current values according to old_cos. */
>> > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
>> > +    {
>> > +        if ( !info->features[i] )
>> > +            continue;
>> > +
>> > +        feat = info->features[i];
>> > +
>> > +        if ( old_cos > feat->ops.get_cos_max(feat) )
>> > +            old_cos = 0;
>> > +
>> > +        /* value getting order is same as feature array */
>> > +        feat->ops.get_old_val(val, feat, old_cos);
>> > +
>> > +        array_len -= feat->cos_num;
>> 
>> So this I should really have asked about on a much earlier patch,
>> but I've recognize the oddity only now: Why is cos_num
>> per-feature-node instead of per-feature? This should really be a
>> field in struct feat_ops (albeit the name "ops" then will be slightly
>> misleading, but I think that's tolerable if you can't think of a better
>> name).
>> 
> Ok, I got your meaning. How about 'feat_props'? No matter operations or
> variables are all properties of the feature.

"props" sounds fine to me.

>> > +        if ( array_len < 0 )
>> > +            return -ENOSPC;
>> 
>> This check needs doing earlier - you need to make sure array_len
>> >= ops.cos_num prior to calling ops.get_old_val(). (Doing the
>> check after the subtraction even causes wrapping issues, which
>> are even more visible in similar code further down.)
>> 
> Thanks for the suggestion! Will move 'array_len >= cos_num' check prior to
> call 'get_old_val'.

And don't forget to do this everywhere in the series - there were
quite a few similar constructs in later patches, iirc.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-28  8:05       ` Yi Sun
@ 2017-03-28  8:36         ` Jan Beulich
  2017-03-28  9:11           ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-28  8:36 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 10:05, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-28 11:12:43, Yi Sun wrote:
>> On 17-03-27 04:17:28, Jan Beulich wrote:
>> > >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
>> > > --- a/xen/arch/x86/psr.c
>> > > +++ b/xen/arch/x86/psr.c
> [...]
> 
>> > >  static int gather_val_array(uint32_t val[],
>> > > @@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
>> > >                              const struct psr_socket_info *info,
>> > >                              unsigned int old_cos)
>> > >  {
>> > > -    return -EINVAL;
>> > > +    const struct feat_node *feat;
>> > > +    unsigned int i;
>> > > +
>> > > +    if ( !val )
>> > > +        return -EINVAL;
>> > > +
>> > > +    /* Get all features current values according to old_cos. */
>> > > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
>> > > +    {
>> > > +        if ( !info->features[i] )
>> > > +            continue;
>> > > +
>> > > +        feat = info->features[i];
>> > > +
>> > > +        if ( old_cos > feat->ops.get_cos_max(feat) )
>> > > +            old_cos = 0;
>> > > +
>> > > +        /* value getting order is same as feature array */
>> > > +        feat->ops.get_old_val(val, feat, old_cos);
>> > > +
>> > > +        array_len -= feat->cos_num;
>> > 
>> > So this I should really have asked about on a much earlier patch,
>> > but I've recognize the oddity only now: Why is cos_num
>> > per-feature-node instead of per-feature? This should really be a
>> > field in struct feat_ops (albeit the name "ops" then will be slightly
>> > misleading, but I think that's tolerable if you can't think of a better
>> > name).
>> > 
>> Ok, I got your meaning. How about 'feat_props'? No matter operations or
>> variables are all properties of the feature.
>> 
> One more thing here. If we move 'cos_max' into 'feat_ops', we cannot declare
> 'feat_ops' as const. Because we have to assign value to 'cos_max' in
> cat_init_feature().

I don't see a problem with this. It's only the static variable which
can't be const then anymore. The pointer used everywhere else
easily can be, afaict.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 11/25] x86: refactor psr: L3 CAT: set value: implement cos finding flow.
  2017-03-28  3:26     ` Yi Sun
@ 2017-03-28  8:41       ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-28  8:41 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 05:26, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 04:28:23, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
>> > --- a/xen/arch/x86/psr.c
>> > +++ b/xen/arch/x86/psr.c
> [...]
>  
>> > +static int cat_compare_val(const uint32_t val[],
>> > +                           const struct feat_node *feat,
>> > +                           unsigned int cos)
>> > +{
>> > +    /*
>> > +     * Different features' cos_max are different. If cos id of the feature
>> > +     * being set exceeds other feature's cos_max, the val of other feature
>> > +     * must be default value. HW supports such case.
>> > +     */
>> > +    if ( cos > feat->info.cat_info.cos_max )
>> > +    {
>> > +        /* cos_reg_val[0] is the default value. */
>> > +        if ( val[0] != feat->cos_reg_val[0] )
>> > +            return -EINVAL;
>> 
>> As you can see, with cos_max moved into the generic portion of the
>> feature node, this entire check can move into the caller.
>> 
> CDP has different behavior in this callback function. We need to check 
> val[0]
> and val[1] like below:
> static int l3_cdp_compare_val(...)
> {
>     if ( cos > feat->info.cat_info.cos_max )
>     {
>         if ( val[0] != get_cdp_data(feat, 0) ||
>              val[1] != get_cdp_code(feat, 0) )
>             return -EINVAL;
> 
>         /* Find */
>         return 1;
>     }
> 
>     if ( val[0] == get_cdp_data(feat, cos) &&
>          val[1] == get_cdp_code(feat, cos) )
>         /* Find */
>         return 1;
> ......
> }

There's no difference other than there being two values checked
here. Moving this to generic code should be easily possible as long
as the get_val() hook is flexible enough. Once again, please think
thoroughly about where to draw the line between generic code
and feature specific hooks - the latter should be reduced to a
minimum.

>> > +        return 1;
>> > +    }
>> > +
>> > +    if ( val[0] == feat->cos_reg_val[cos] )
>> > +        /* Find */
>> > +        return 1;
>> > +
>> > +    /* Not find */
>> > +    return 0;
>> > +}
>> 
>> Or actually, the entire function then becomes feature independent,
>> as it seems. And I think I did suggest that already during review of
>> an earlier version.
>> 
> Per above explanation, I think we have to keep this callback function.

Per above explanation, I think you don't need to keep this callback
function.

>> > +    int find = 0;
>> 
>> "found" again, or even simply "rc"? Also I think this would better
>> move into the outer for() scope.
>> 
> Ok, will use 'found' and move it.

Please check its use(s): It should be "found" only if that's the
meaning it always has. It's type being int (rather than bool)
suggests otherwise ...

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-28  4:58     ` Yi Sun
@ 2017-03-28  8:45       ` Jan Beulich
  2017-03-28 10:31         ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-28  8:45 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 06:58, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 04:37:37, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
>> > +static bool cat_fits_cos_max(const uint32_t val[],
>> > +                             const struct feat_node *feat,
>> > +                             unsigned int cos)
>> > +{
>> > +    if ( cos > feat->info.cat_info.cos_max &&
>> > +         val[0] != feat->cos_reg_val[0] )
>> > +            /*
>> > +             * Exceed cos_max and value to set is not default,
>> > +             * return error.
>> > +             */
>> > +            return false;
>> > +
>> > +    return true;
>> > +}
>> 
>> Same here - with cos_max moved out, the hook would seem to
>> become unnecessary.
>> 
> As explanation in previous patch, CDP has different behavior.
> static bool l3_cdp_fits_cos_max(...)
> {
>     if ( cos > feat->info.cat_info.cos_max &&
>          (val[0] != get_cdp_data(feat, 0) || val[1] != get_cdp_code(feat, 0)) )
>             /*
>              * Exceed cos_max and value to set is not default,
>              * return error.
>              */
>             return false;
> 
>     return true;
> 
> }

As said in reply, by making get_val() flexible enough you should
be able to avoid this. The caller knows how many values to
compare.

>> >  static int pick_avail_cos(const struct psr_socket_info *info,
>> >                            const uint32_t val[], uint32_t array_len,
>> >                            unsigned int old_cos,
>> >                            enum psr_feat_type feat_type)
>> >  {
>> > +    unsigned int cos;
>> > +    unsigned int cos_max = 0;
>> > +    const struct feat_node *feat;
>> > +    const unsigned int *ref = info->cos_ref;
>> > +
>> >      ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
>> > -    return -ENOENT;
>> > +
>> > +    /* cos_max is the one of the feature which is being set. */
>> > +    feat = info->features[feat_type];
>> > +    if ( !feat )
>> > +        return -ENOENT;
>> > +
>> > +    cos_max = feat->ops.get_cos_max(feat);
>> > +    if ( !cos_max )
>> > +        return -ENOENT;
>> > +
>> > +    /*
>> > +     * If old cos is referred only by the domain, then use it. And, we cannot
>> 
>> "the domain" here is lacking context - there's no domain involved
> 
> How about "the domain input through 'psr_set_val'"?

If you assume this going to remain a helper function for just this
one purpose, then I could live with that. Note however that if
ever a 2nd caller would appear, such a comment likely would
become stale. Therefore it is generally better to write comments
based on what the specific function does or assumes, without
regard to its caller(s) assumptions/restrictions.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 13/25] x86: refactor psr: L3 CAT: set value: implement write msr flow.
  2017-03-28  5:06     ` Yi Sun
@ 2017-03-28  8:48       ` Jan Beulich
  2017-03-28 10:20         ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-28  8:48 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 07:06, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-27 04:46:01, Jan Beulich wrote:
>> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
>> > @@ -421,6 +425,18 @@ static bool cat_fits_cos_max(const uint32_t val[],
>> >  }
>> >  
>> >  /* L3 CAT ops */
>> > +static void l3_cat_write_msr(unsigned int cos, uint32_t val,
>> > +                             enum cbm_type type, struct feat_node *feat)
>> 
>> "type" is an unused parameter. Please remove it from the hook
>> and this function.
>> 
> This is used by CDP callback function. We need it to decide whether CODE
> or DATA is written.

Strictly speaking you should then introduce it there - that'll also
allow better judgment as whether it's actually needed there, or
whether there are alternatives without.

>> > +    unsigned int cos            = info->cos;
>> > +    struct feat_node *feat      = info->feature;
>> > +
>> > +    if ( !feat )
>> > +        return;
>> 
>> You shouldn't even call this function when !feat.
>> 
> Ok, may I move the check to the 'write_psr_msr'?

Perhaps that's the right place, but it may also be that it would
better move even higher up. The question really is where it is
useful/needed.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-28  8:36         ` Jan Beulich
@ 2017-03-28  9:11           ` Yi Sun
  2017-03-28  9:20             ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28  9:11 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-28 02:36:05, Jan Beulich wrote:
> >>> On 28.03.17 at 10:05, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-28 11:12:43, Yi Sun wrote:
> >> On 17-03-27 04:17:28, Jan Beulich wrote:
> >> > >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> >> > > --- a/xen/arch/x86/psr.c
> >> > > +++ b/xen/arch/x86/psr.c
> > [...]
> > 
> >> > >  static int gather_val_array(uint32_t val[],
> >> > > @@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
> >> > >                              const struct psr_socket_info *info,
> >> > >                              unsigned int old_cos)
> >> > >  {
> >> > > -    return -EINVAL;
> >> > > +    const struct feat_node *feat;
> >> > > +    unsigned int i;
> >> > > +
> >> > > +    if ( !val )
> >> > > +        return -EINVAL;
> >> > > +
> >> > > +    /* Get all features current values according to old_cos. */
> >> > > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> >> > > +    {
> >> > > +        if ( !info->features[i] )
> >> > > +            continue;
> >> > > +
> >> > > +        feat = info->features[i];
> >> > > +
> >> > > +        if ( old_cos > feat->ops.get_cos_max(feat) )
> >> > > +            old_cos = 0;
> >> > > +
> >> > > +        /* value getting order is same as feature array */
> >> > > +        feat->ops.get_old_val(val, feat, old_cos);
> >> > > +
> >> > > +        array_len -= feat->cos_num;
> >> > 
> >> > So this I should really have asked about on a much earlier patch,
> >> > but I've recognize the oddity only now: Why is cos_num
> >> > per-feature-node instead of per-feature? This should really be a
> >> > field in struct feat_ops (albeit the name "ops" then will be slightly
> >> > misleading, but I think that's tolerable if you can't think of a better
> >> > name).
> >> > 
> >> Ok, I got your meaning. How about 'feat_props'? No matter operations or
> >> variables are all properties of the feature.
> >> 
> > One more thing here. If we move 'cos_max' into 'feat_ops', we cannot declare
> > 'feat_ops' as const. Because we have to assign value to 'cos_max' in
> > cat_init_feature().
> 
> I don't see a problem with this. It's only the static variable which
> can't be const then anymore. The pointer used everywhere else
> easily can be, afaict.
> 
Because I want to assign the l3_cat_props to feat->props before executing
cat_init_feature(). The codes sequence is below. Then, in cat_init_feature(),
I can use 'feat' but not 'l3_cat_props' which is feature specific.

static void cat_init_feature(...)
{
......
    feat->info.cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
    feat->props->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
......
}

static struct feat_props l3_cat_props = {
    .cos_num = 1,
};

static void psr_cpu_init(void)
{
......
        feat->props = &l3_cat_props;
        cat_init_feature(&regs, feat, info, PSR_SOCKET_L3_CAT);
......
}

Then, back to the origin of this. I think feature-node is feature itself.
Everything in it is feature specific thing. Is it necessary to move values
into a sub-structure, 'feat_props'? If not doing this, we can keep 'feat_ops'
to only handle callback functions.

Sorry that I output my thought until now.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-28  9:11           ` Yi Sun
@ 2017-03-28  9:20             ` Jan Beulich
  2017-03-28 10:18               ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-28  9:20 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 11:11, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-28 02:36:05, Jan Beulich wrote:
>> >>> On 28.03.17 at 10:05, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-28 11:12:43, Yi Sun wrote:
>> >> On 17-03-27 04:17:28, Jan Beulich wrote:
>> >> > >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
>> >> > > --- a/xen/arch/x86/psr.c
>> >> > > +++ b/xen/arch/x86/psr.c
>> > [...]
>> > 
>> >> > >  static int gather_val_array(uint32_t val[],
>> >> > > @@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
>> >> > >                              const struct psr_socket_info *info,
>> >> > >                              unsigned int old_cos)
>> >> > >  {
>> >> > > -    return -EINVAL;
>> >> > > +    const struct feat_node *feat;
>> >> > > +    unsigned int i;
>> >> > > +
>> >> > > +    if ( !val )
>> >> > > +        return -EINVAL;
>> >> > > +
>> >> > > +    /* Get all features current values according to old_cos. */
>> >> > > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
>> >> > > +    {
>> >> > > +        if ( !info->features[i] )
>> >> > > +            continue;
>> >> > > +
>> >> > > +        feat = info->features[i];
>> >> > > +
>> >> > > +        if ( old_cos > feat->ops.get_cos_max(feat) )
>> >> > > +            old_cos = 0;
>> >> > > +
>> >> > > +        /* value getting order is same as feature array */
>> >> > > +        feat->ops.get_old_val(val, feat, old_cos);
>> >> > > +
>> >> > > +        array_len -= feat->cos_num;
>> >> > 
>> >> > So this I should really have asked about on a much earlier patch,
>> >> > but I've recognize the oddity only now: Why is cos_num
>> >> > per-feature-node instead of per-feature? This should really be a
>> >> > field in struct feat_ops (albeit the name "ops" then will be slightly
>> >> > misleading, but I think that's tolerable if you can't think of a better
>> >> > name).
>> >> > 
>> >> Ok, I got your meaning. How about 'feat_props'? No matter operations or
>> >> variables are all properties of the feature.
>> >> 
>> > One more thing here. If we move 'cos_max' into 'feat_ops', we cannot 
> declare
>> > 'feat_ops' as const. Because we have to assign value to 'cos_max' in
>> > cat_init_feature().
>> 
>> I don't see a problem with this. It's only the static variable which
>> can't be const then anymore. The pointer used everywhere else
>> easily can be, afaict.
>> 
> Because I want to assign the l3_cat_props to feat->props before executing
> cat_init_feature(). The codes sequence is below. Then, in 
> cat_init_feature(),
> I can use 'feat' but not 'l3_cat_props' which is feature specific.
> 
> static void cat_init_feature(...)
> {
> ......
>     feat->info.cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
>     feat->props->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
> ......
> }
> 
> static struct feat_props l3_cat_props = {
>     .cos_num = 1,
> };
> 
> static void psr_cpu_init(void)
> {
> ......
>         feat->props = &l3_cat_props;
>         cat_init_feature(&regs, feat, info, PSR_SOCKET_L3_CAT);
> ......
> }

static void psr_cpu_init(void)
{
......
        cat_init_feature(&regs, &l3_cat_props, feat, info, PSR_SOCKET_L3_CAT);
        feat->props = &l3_cat_props;
......
}

> Then, back to the origin of this. I think feature-node is feature itself.
> Everything in it is feature specific thing. Is it necessary to move values
> into a sub-structure, 'feat_props'? If not doing this, we can keep
> 'feat_ops' to only handle callback functions.

I'm not sure I understand what you're trying to tell me. I can only
repeat what I've said before: The amount of feature specific
callbacks should be reduced to the minimum necessary - the more
generic code, the less code overall to maintain.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-28  8:34       ` Jan Beulich
@ 2017-03-28 10:12         ` Yi Sun
  2017-03-28 10:36           ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28 10:12 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau

On 17-03-28 02:34:51, Jan Beulich wrote:
> >>> On 28.03.17 at 05:12, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-27 04:17:28, Jan Beulich wrote:
> >> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> >> > --- a/xen/arch/x86/psr.c
> >> > +++ b/xen/arch/x86/psr.c
> >> > @@ -101,6 +101,28 @@ struct feat_node {
> >> >          /* get_val is used to get feature COS register value. */
> >> >          void (*get_val)(const struct feat_node *feat, unsigned int cos,
> >> >                          enum cbm_type type, uint32_t *val);
> >> > +
> >> > +        /*
> >> > +         * get_old_val and set_new_val are a pair of functions called in order.
> >> > +         * The caller will traverse all features in the array and call
> >> > +         * 'get_old_val' to get old_cos register value of all supported
> >> > +         * features. Then, call 'set_new_val' to set the new value for the
> >> > +         * designated feature.
> >> > +         *
> >> > +         * All the values are set into value array according to the traversal
> >> > +         * order, meaning the same order of feature array members.
> >> > +         *
> >> > +         * The return value meaning of set_new_val:
> >> > +         * 0 - success.
> >> > +         * negative - error.
> >> > +         */
> >> > +        void (*get_old_val)(uint32_t val[],
> >> > +                            const struct feat_node *feat,
> >> > +                            unsigned int old_cos);
> >> > +        int (*set_new_val)(uint32_t val[],
> >> > +                           const struct feat_node *feat,
> >> > +                           enum cbm_type type,
> >> > +                           uint32_t new_val);
> >> 
> >> Along the lines of an earlier comment - are "old" and "new" really
> >> meaningful here?
> >> 
> > Maybe 'old' is not accurate. How about 'current'? In fact, we use this
> > function to get domain's current CBM value. Furthermore, this is to distinguish
> > 'get_val' which is declared above.
> 
> I'm fine with "current", but the name collision - would "current" be
> omitted still bothers me. The fact that cat_get_old_val() calls
> cat_get_val(), however, strongly suggests that the hook here is
> redundant anyway. Even in the CDP case I think you can get
> away without it, but if this turns out really impossible or clumsy,
> then the hook could be introduced there (with a better name)
> and be an optional one (with the caller using ->get_val() if the
> one here is NULL).
> 
I am afraid we have to keep this hook. CDP uses this hook to get both CODE and
DATA at same time. But CDP uses get_val() hook to get either CODE or DATA.
So, they have different functionalitiy. I prefer to rename it to
'get_current_val'.

I can make it optional hook. But the codes in caller look a little strange. E.g.
static int gather_val_array()
{
...
    if ( feat->ops.get_old_val )
        feat->ops.get_old_val(val, feat, old_cos);
    else
        feat->ops.get_val(feat, old_cos, 0, &val[0]);
...
}

So, I think a wrapper like cat_get_old_val() may be a better choice. What is
your opinion?

> > I think 'new' is meaningful to express we are setting the newly input value.
> 
> Well, this is the meaning to its caller. The function itself doesn't
> care whether the value is a new one, or just some other value
> coming from an unnamed source.
> 
Ok, will remove 'new'.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-28  9:20             ` Jan Beulich
@ 2017-03-28 10:18               ` Yi Sun
  2017-03-28 10:39                 ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28 10:18 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-28 03:20:05, Jan Beulich wrote:
> >>> On 28.03.17 at 11:11, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-28 02:36:05, Jan Beulich wrote:
> >> >>> On 28.03.17 at 10:05, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-03-28 11:12:43, Yi Sun wrote:
> >> >> On 17-03-27 04:17:28, Jan Beulich wrote:
> >> >> > >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> >> >> > > --- a/xen/arch/x86/psr.c
> >> >> > > +++ b/xen/arch/x86/psr.c
> >> > [...]
> >> > 
> >> >> > >  static int gather_val_array(uint32_t val[],
> >> >> > > @@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
> >> >> > >                              const struct psr_socket_info *info,
> >> >> > >                              unsigned int old_cos)
> >> >> > >  {
> >> >> > > -    return -EINVAL;
> >> >> > > +    const struct feat_node *feat;
> >> >> > > +    unsigned int i;
> >> >> > > +
> >> >> > > +    if ( !val )
> >> >> > > +        return -EINVAL;
> >> >> > > +
> >> >> > > +    /* Get all features current values according to old_cos. */
> >> >> > > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
> >> >> > > +    {
> >> >> > > +        if ( !info->features[i] )
> >> >> > > +            continue;
> >> >> > > +
> >> >> > > +        feat = info->features[i];
> >> >> > > +
> >> >> > > +        if ( old_cos > feat->ops.get_cos_max(feat) )
> >> >> > > +            old_cos = 0;
> >> >> > > +
> >> >> > > +        /* value getting order is same as feature array */
> >> >> > > +        feat->ops.get_old_val(val, feat, old_cos);
> >> >> > > +
> >> >> > > +        array_len -= feat->cos_num;
> >> >> > 
> >> >> > So this I should really have asked about on a much earlier patch,
> >> >> > but I've recognize the oddity only now: Why is cos_num
> >> >> > per-feature-node instead of per-feature? This should really be a
> >> >> > field in struct feat_ops (albeit the name "ops" then will be slightly
> >> >> > misleading, but I think that's tolerable if you can't think of a better
> >> >> > name).
> >> >> > 
> >> >> Ok, I got your meaning. How about 'feat_props'? No matter operations or
> >> >> variables are all properties of the feature.
> >> >> 
> >> > One more thing here. If we move 'cos_max' into 'feat_ops', we cannot 
> > declare
> >> > 'feat_ops' as const. Because we have to assign value to 'cos_max' in
> >> > cat_init_feature().
> >> 
> >> I don't see a problem with this. It's only the static variable which
> >> can't be const then anymore. The pointer used everywhere else
> >> easily can be, afaict.
> >> 
> > Because I want to assign the l3_cat_props to feat->props before executing
> > cat_init_feature(). The codes sequence is below. Then, in 
> > cat_init_feature(),
> > I can use 'feat' but not 'l3_cat_props' which is feature specific.
> > 
> > static void cat_init_feature(...)
> > {
> > ......
> >     feat->info.cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
> >     feat->props->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
> > ......
> > }
> > 
> > static struct feat_props l3_cat_props = {
> >     .cos_num = 1,
> > };
> > 
> > static void psr_cpu_init(void)
> > {
> > ......
> >         feat->props = &l3_cat_props;
> >         cat_init_feature(&regs, feat, info, PSR_SOCKET_L3_CAT);
> > ......
> > }
> 
> static void psr_cpu_init(void)
> {
> ......
>         cat_init_feature(&regs, &l3_cat_props, feat, info, PSR_SOCKET_L3_CAT);
>         feat->props = &l3_cat_props;
> ......
> }
> 
> > Then, back to the origin of this. I think feature-node is feature itself.
> > Everything in it is feature specific thing. Is it necessary to move values
> > into a sub-structure, 'feat_props'? If not doing this, we can keep
> > 'feat_ops' to only handle callback functions.
> 
> I'm not sure I understand what you're trying to tell me. I can only
> repeat what I've said before: The amount of feature specific
> callbacks should be reduced to the minimum necessary - the more
> generic code, the less code overall to maintain.
> 
My key point is: can we keep 'cos_num' and 'cos_max' into 'feat_node' but not
'feat_ops'? Because I think 'feat_node' represents a feature. It can keep
all feature specific things.

If you still think it is not good, can we define 'struct feat_props' without
const? Then, I can keep above code sequence.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 13/25] x86: refactor psr: L3 CAT: set value: implement write msr flow.
  2017-03-28  8:48       ` Jan Beulich
@ 2017-03-28 10:20         ` Yi Sun
  0 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-28 10:20 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-28 02:48:53, Jan Beulich wrote:
> >>> On 28.03.17 at 07:06, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-27 04:46:01, Jan Beulich wrote:
> >> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> >> > @@ -421,6 +425,18 @@ static bool cat_fits_cos_max(const uint32_t val[],
> >> >  }
> >> >  
> >> >  /* L3 CAT ops */
> >> > +static void l3_cat_write_msr(unsigned int cos, uint32_t val,
> >> > +                             enum cbm_type type, struct feat_node *feat)
> >> 
> >> "type" is an unused parameter. Please remove it from the hook
> >> and this function.
> >> 
> > This is used by CDP callback function. We need it to decide whether CODE
> > or DATA is written.
> 
> Strictly speaking you should then introduce it there - that'll also
> allow better judgment as whether it's actually needed there, or
> whether there are alternatives without.
> 
Ok, I will introduce 'type' in CDP implementation.

> >> > +    unsigned int cos            = info->cos;
> >> > +    struct feat_node *feat      = info->feature;
> >> > +
> >> > +    if ( !feat )
> >> > +        return;
> >> 
> >> You shouldn't even call this function when !feat.
> >> 
> > Ok, may I move the check to the 'write_psr_msr'?
> 
> Perhaps that's the right place, but it may also be that it would
> better move even higher up. The question really is where it is
> useful/needed.
> 
Will consider the place to check it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-28  8:45       ` Jan Beulich
@ 2017-03-28 10:31         ` Yi Sun
  2017-03-28 10:40           ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28 10:31 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-28 02:45:13, Jan Beulich wrote:
> >>> On 28.03.17 at 06:58, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-27 04:37:37, Jan Beulich wrote:
> >> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> >> > +static bool cat_fits_cos_max(const uint32_t val[],
> >> > +                             const struct feat_node *feat,
> >> > +                             unsigned int cos)
> >> > +{
> >> > +    if ( cos > feat->info.cat_info.cos_max &&
> >> > +         val[0] != feat->cos_reg_val[0] )
> >> > +            /*
> >> > +             * Exceed cos_max and value to set is not default,
> >> > +             * return error.
> >> > +             */
> >> > +            return false;
> >> > +
> >> > +    return true;
> >> > +}
> >> 
> >> Same here - with cos_max moved out, the hook would seem to
> >> become unnecessary.
> >> 
> > As explanation in previous patch, CDP has different behavior.
> > static bool l3_cdp_fits_cos_max(...)
> > {
> >     if ( cos > feat->info.cat_info.cos_max &&
> >          (val[0] != get_cdp_data(feat, 0) || val[1] != get_cdp_code(feat, 0)) )
> >             /*
> >              * Exceed cos_max and value to set is not default,
> >              * return error.
> >              */
> >             return false;
> > 
> >     return true;
> > 
> > }
> 
> As said in reply, by making get_val() flexible enough you should
> be able to avoid this.

Sorry, I am confused here. 'fits_cos_max' is called during set value process.
Why "making get_val() flexible enough" can avoid this?

> The caller knows how many values to compare.
> 
My purpose to implement such hook is to avoid caller knows feature details.
Then, we can create a general flow to cover all features. So, I do not
understand your mearing here. Sorry.

> >> >  static int pick_avail_cos(const struct psr_socket_info *info,
> >> >                            const uint32_t val[], uint32_t array_len,
> >> >                            unsigned int old_cos,
> >> >                            enum psr_feat_type feat_type)
> >> >  {
> >> > +    unsigned int cos;
> >> > +    unsigned int cos_max = 0;
> >> > +    const struct feat_node *feat;
> >> > +    const unsigned int *ref = info->cos_ref;
> >> > +
> >> >      ASSERT(spin_is_locked((spinlock_t *)(&info->ref_lock)));
> >> > -    return -ENOENT;
> >> > +
> >> > +    /* cos_max is the one of the feature which is being set. */
> >> > +    feat = info->features[feat_type];
> >> > +    if ( !feat )
> >> > +        return -ENOENT;
> >> > +
> >> > +    cos_max = feat->ops.get_cos_max(feat);
> >> > +    if ( !cos_max )
> >> > +        return -ENOENT;
> >> > +
> >> > +    /*
> >> > +     * If old cos is referred only by the domain, then use it. And, we cannot
> >> 
> >> "the domain" here is lacking context - there's no domain involved
> > 
> > How about "the domain input through 'psr_set_val'"?
> 
> If you assume this going to remain a helper function for just this
> one purpose, then I could live with that. Note however that if
> ever a 2nd caller would appear, such a comment likely would
> become stale. Therefore it is generally better to write comments
> based on what the specific function does or assumes, without
> regard to its caller(s) assumptions/restrictions.
> 
Ok, I should explain this in caller and I think there already are
comments to explain this. So, I think I may remove this comment here.

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-28 10:12         ` Yi Sun
@ 2017-03-28 10:36           ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-28 10:36 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 12:12, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-28 02:34:51, Jan Beulich wrote:
>> >>> On 28.03.17 at 05:12, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-27 04:17:28, Jan Beulich wrote:
>> >> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
>> >> > --- a/xen/arch/x86/psr.c
>> >> > +++ b/xen/arch/x86/psr.c
>> >> > @@ -101,6 +101,28 @@ struct feat_node {
>> >> >          /* get_val is used to get feature COS register value. */
>> >> >          void (*get_val)(const struct feat_node *feat, unsigned int cos,
>> >> >                          enum cbm_type type, uint32_t *val);
>> >> > +
>> >> > +        /*
>> >> > +         * get_old_val and set_new_val are a pair of functions called in 
> order.
>> >> > +         * The caller will traverse all features in the array and call
>> >> > +         * 'get_old_val' to get old_cos register value of all supported
>> >> > +         * features. Then, call 'set_new_val' to set the new value for 
> the
>> >> > +         * designated feature.
>> >> > +         *
>> >> > +         * All the values are set into value array according to the 
> traversal
>> >> > +         * order, meaning the same order of feature array members.
>> >> > +         *
>> >> > +         * The return value meaning of set_new_val:
>> >> > +         * 0 - success.
>> >> > +         * negative - error.
>> >> > +         */
>> >> > +        void (*get_old_val)(uint32_t val[],
>> >> > +                            const struct feat_node *feat,
>> >> > +                            unsigned int old_cos);
>> >> > +        int (*set_new_val)(uint32_t val[],
>> >> > +                           const struct feat_node *feat,
>> >> > +                           enum cbm_type type,
>> >> > +                           uint32_t new_val);
>> >> 
>> >> Along the lines of an earlier comment - are "old" and "new" really
>> >> meaningful here?
>> >> 
>> > Maybe 'old' is not accurate. How about 'current'? In fact, we use this
>> > function to get domain's current CBM value. Furthermore, this is to 
> distinguish
>> > 'get_val' which is declared above.
>> 
>> I'm fine with "current", but the name collision - would "current" be
>> omitted still bothers me. The fact that cat_get_old_val() calls
>> cat_get_val(), however, strongly suggests that the hook here is
>> redundant anyway. Even in the CDP case I think you can get
>> away without it, but if this turns out really impossible or clumsy,
>> then the hook could be introduced there (with a better name)
>> and be an optional one (with the caller using ->get_val() if the
>> one here is NULL).
>> 
> I am afraid we have to keep this hook. CDP uses this hook to get both CODE and
> DATA at same time. But CDP uses get_val() hook to get either CODE or DATA.
> So, they have different functionalitiy. I prefer to rename it to
> 'get_current_val'.
> 
> I can make it optional hook. But the codes in caller look a little strange. 
> E.g.
> static int gather_val_array()
> {
> ...
>     if ( feat->ops.get_old_val )
>         feat->ops.get_old_val(val, feat, old_cos);
>     else
>         feat->ops.get_val(feat, old_cos, 0, &val[0]);
> ...
> }

So it looks like I have to repeat myself here: The caller knows
the number of slots. Make the get_val() hook flexible enough to
allow it to replace get_old_val() here (the caller would then
invoke it cos_num times).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array.
  2017-03-28 10:18               ` Yi Sun
@ 2017-03-28 10:39                 ` Jan Beulich
  0 siblings, 0 replies; 105+ messages in thread
From: Jan Beulich @ 2017-03-28 10:39 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 12:18, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-28 03:20:05, Jan Beulich wrote:
>> >>> On 28.03.17 at 11:11, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-28 02:36:05, Jan Beulich wrote:
>> >> >>> On 28.03.17 at 10:05, <yi.y.sun@linux.intel.com> wrote:
>> >> > On 17-03-28 11:12:43, Yi Sun wrote:
>> >> >> On 17-03-27 04:17:28, Jan Beulich wrote:
>> >> >> > >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
>> >> >> > > --- a/xen/arch/x86/psr.c
>> >> >> > > +++ b/xen/arch/x86/psr.c
>> >> > [...]
>> >> > 
>> >> >> > >  static int gather_val_array(uint32_t val[],
>> >> >> > > @@ -589,7 +672,34 @@ static int gather_val_array(uint32_t val[],
>> >> >> > >                              const struct psr_socket_info *info,
>> >> >> > >                              unsigned int old_cos)
>> >> >> > >  {
>> >> >> > > -    return -EINVAL;
>> >> >> > > +    const struct feat_node *feat;
>> >> >> > > +    unsigned int i;
>> >> >> > > +
>> >> >> > > +    if ( !val )
>> >> >> > > +        return -EINVAL;
>> >> >> > > +
>> >> >> > > +    /* Get all features current values according to old_cos. */
>> >> >> > > +    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
>> >> >> > > +    {
>> >> >> > > +        if ( !info->features[i] )
>> >> >> > > +            continue;
>> >> >> > > +
>> >> >> > > +        feat = info->features[i];
>> >> >> > > +
>> >> >> > > +        if ( old_cos > feat->ops.get_cos_max(feat) )
>> >> >> > > +            old_cos = 0;
>> >> >> > > +
>> >> >> > > +        /* value getting order is same as feature array */
>> >> >> > > +        feat->ops.get_old_val(val, feat, old_cos);
>> >> >> > > +
>> >> >> > > +        array_len -= feat->cos_num;
>> >> >> > 
>> >> >> > So this I should really have asked about on a much earlier patch,
>> >> >> > but I've recognize the oddity only now: Why is cos_num
>> >> >> > per-feature-node instead of per-feature? This should really be a
>> >> >> > field in struct feat_ops (albeit the name "ops" then will be slightly
>> >> >> > misleading, but I think that's tolerable if you can't think of a better
>> >> >> > name).
>> >> >> > 
>> >> >> Ok, I got your meaning. How about 'feat_props'? No matter operations or
>> >> >> variables are all properties of the feature.
>> >> >> 
>> >> > One more thing here. If we move 'cos_max' into 'feat_ops', we cannot 
>> > declare
>> >> > 'feat_ops' as const. Because we have to assign value to 'cos_max' in
>> >> > cat_init_feature().
>> >> 
>> >> I don't see a problem with this. It's only the static variable which
>> >> can't be const then anymore. The pointer used everywhere else
>> >> easily can be, afaict.
>> >> 
>> > Because I want to assign the l3_cat_props to feat->props before executing
>> > cat_init_feature(). The codes sequence is below. Then, in 
>> > cat_init_feature(),
>> > I can use 'feat' but not 'l3_cat_props' which is feature specific.
>> > 
>> > static void cat_init_feature(...)
>> > {
>> > ......
>> >     feat->info.cbm_len = (regs->a & CAT_CBM_LEN_MASK) + 1;
>> >     feat->props->cos_max = min(opt_cos_max, regs->d & CAT_COS_MAX_MASK);
>> > ......
>> > }
>> > 
>> > static struct feat_props l3_cat_props = {
>> >     .cos_num = 1,
>> > };
>> > 
>> > static void psr_cpu_init(void)
>> > {
>> > ......
>> >         feat->props = &l3_cat_props;
>> >         cat_init_feature(&regs, feat, info, PSR_SOCKET_L3_CAT);
>> > ......
>> > }
>> 
>> static void psr_cpu_init(void)
>> {
>> ......
>>         cat_init_feature(&regs, &l3_cat_props, feat, info, PSR_SOCKET_L3_CAT);
>>         feat->props = &l3_cat_props;
>> ......
>> }
>> 
>> > Then, back to the origin of this. I think feature-node is feature itself.
>> > Everything in it is feature specific thing. Is it necessary to move values
>> > into a sub-structure, 'feat_props'? If not doing this, we can keep
>> > 'feat_ops' to only handle callback functions.
>> 
>> I'm not sure I understand what you're trying to tell me. I can only
>> repeat what I've said before: The amount of feature specific
>> callbacks should be reduced to the minimum necessary - the more
>> generic code, the less code overall to maintain.
>> 
> My key point is: can we keep 'cos_num' and 'cos_max' into 'feat_node' but not
> 'feat_ops'? Because I think 'feat_node' represents a feature. It can keep
> all feature specific things.

Let me ask the question this way - for a given feature, can
cos_max and cos_num vary between individual nodes created
for this feature? If they can, the values need to be per-node.
If they can't, there's no point in replicating them in every node.

> If you still think it is not good, can we define 'struct feat_props' without
> const? Then, I can keep above code sequence.

Sure - I've already outlined how that would work with keeping const
in most places.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-28 10:31         ` Yi Sun
@ 2017-03-28 10:40           ` Jan Beulich
  2017-03-28 11:59             ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-28 10:40 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 12:31, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-28 02:45:13, Jan Beulich wrote:
>> >>> On 28.03.17 at 06:58, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-27 04:37:37, Jan Beulich wrote:
>> >> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
>> >> > +static bool cat_fits_cos_max(const uint32_t val[],
>> >> > +                             const struct feat_node *feat,
>> >> > +                             unsigned int cos)
>> >> > +{
>> >> > +    if ( cos > feat->info.cat_info.cos_max &&
>> >> > +         val[0] != feat->cos_reg_val[0] )
>> >> > +            /*
>> >> > +             * Exceed cos_max and value to set is not default,
>> >> > +             * return error.
>> >> > +             */
>> >> > +            return false;
>> >> > +
>> >> > +    return true;
>> >> > +}
>> >> 
>> >> Same here - with cos_max moved out, the hook would seem to
>> >> become unnecessary.
>> >> 
>> > As explanation in previous patch, CDP has different behavior.
>> > static bool l3_cdp_fits_cos_max(...)
>> > {
>> >     if ( cos > feat->info.cat_info.cos_max &&
>> >          (val[0] != get_cdp_data(feat, 0) || val[1] != get_cdp_code(feat, 0)) )
>> >             /*
>> >              * Exceed cos_max and value to set is not default,
>> >              * return error.
>> >              */
>> >             return false;
>> > 
>> >     return true;
>> > 
>> > }
>> 
>> As said in reply, by making get_val() flexible enough you should
>> be able to avoid this.
> 
> Sorry, I am confused here. 'fits_cos_max' is called during set value process.
> Why "making get_val() flexible enough" can avoid this?

Because that would then replace the get_cdp_data() and
get_cdp_code() calls above.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-28 10:40           ` Jan Beulich
@ 2017-03-28 11:59             ` Yi Sun
  2017-03-28 12:20               ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-28 11:59 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-28 04:40:41, Jan Beulich wrote:
> >>> On 28.03.17 at 12:31, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-28 02:45:13, Jan Beulich wrote:
> >> >>> On 28.03.17 at 06:58, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-03-27 04:37:37, Jan Beulich wrote:
> >> >> >>> On 16.03.17 at 12:08, <yi.y.sun@linux.intel.com> wrote:
> >> >> > +static bool cat_fits_cos_max(const uint32_t val[],
> >> >> > +                             const struct feat_node *feat,
> >> >> > +                             unsigned int cos)
> >> >> > +{
> >> >> > +    if ( cos > feat->info.cat_info.cos_max &&
> >> >> > +         val[0] != feat->cos_reg_val[0] )
> >> >> > +            /*
> >> >> > +             * Exceed cos_max and value to set is not default,
> >> >> > +             * return error.
> >> >> > +             */
> >> >> > +            return false;
> >> >> > +
> >> >> > +    return true;
> >> >> > +}
> >> >> 
> >> >> Same here - with cos_max moved out, the hook would seem to
> >> >> become unnecessary.
> >> >> 
> >> > As explanation in previous patch, CDP has different behavior.
> >> > static bool l3_cdp_fits_cos_max(...)
> >> > {
> >> >     if ( cos > feat->info.cat_info.cos_max &&
> >> >          (val[0] != get_cdp_data(feat, 0) || val[1] != get_cdp_code(feat, 0)) )
> >> >             /*
> >> >              * Exceed cos_max and value to set is not default,
> >> >              * return error.
> >> >              */
> >> >             return false;
> >> > 
> >> >     return true;
> >> > 
> >> > }
> >> 
> >> As said in reply, by making get_val() flexible enough you should
> >> be able to avoid this.
> > 
> > Sorry, I am confused here. 'fits_cos_max' is called during set value process.
> > Why "making get_val() flexible enough" can avoid this?
> 
> Because that would then replace the get_cdp_data() and
> get_cdp_code() calls above.
> 
I think we at least need a 'get_val()' hook. I try to implement CAT/CDP hook.
Please help to check if this is what you thought.

static void cat_get_val(const struct feat_node *feat, unsigned int cos,
                        enum cbm_type type, int flag, uint32_t *val)
{
    *val = feat->cos_reg_val[cos];
}

static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
                           enum cbm_type type, int flag, uint32_t *val)
{
    if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
        *val = get_cdp_data(feat, cos);
    if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
        *val = get_cdp_code(feat, cos);
}

static bool fits_cos_max(const uint32_t val[],
                         uint32_t array_len,
                         const struct psr_socket_info *info,
                         unsigned int cos)
{
    unsigned int i, j;
    const struct feat_node *feat;
    uint32_t default_val;
	
    for ( i = 0; i < PSR_SOCKET_MAX_FEAT; i++ )
    {
        if ( !info->features[i] )
            continue;

        feat = info->features[i];

        if ( array_len < feat->cos_num )
            return false;
	
        /* Move the fits_cos_max() hook content into caller here. */
        if ( cos > feat->props->cos_max )
        {
            for ( j = 0; j < feat->cos_num; j++ )
            {
                /* cos_reg_val[0] always stores the default value so set cos to 0. */
                feat->props->get_val(feat, 0, 0, j, &default_val);
                if ( val[j] != default_val )
                    return false;
            }
        }

        val += feat->cos_num;
    }

    return true;
}

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-28 11:59             ` Yi Sun
@ 2017-03-28 12:20               ` Jan Beulich
  2017-03-29  1:20                 ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-28 12:20 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
> I think we at least need a 'get_val()' hook.

Of course.

> I try to implement CAT/CDP hook.
> Please help to check if this is what you thought.

One remark below, but other than that - yes.

> static void cat_get_val(const struct feat_node *feat, unsigned int cos,
>                         enum cbm_type type, int flag, uint32_t *val)
> {
>     *val = feat->cos_reg_val[cos];
> }
> 
> static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
>                            enum cbm_type type, int flag, uint32_t *val)
> {
>     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
>         *val = get_cdp_data(feat, cos);
>     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
>         *val = get_cdp_code(feat, cos);
> }

Why the redundancy between type and flag?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 24/25] tools: L2 CAT: support set cbm for L2 CAT.
  2017-03-16 11:08 ` [PATCH v9 24/25] tools: L2 CAT: support set " Yi Sun
@ 2017-03-28 14:04   ` Wei Liu
  2017-03-29  1:21     ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Wei Liu @ 2017-03-28 14:04 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, andrew.cooper3, dario.faggioli, he.chen,
	ian.jackson, mengxu, jbeulich, chao.p.peng, xen-devel, roger.pau

On Thu, Mar 16, 2017 at 07:08:14PM +0800, Yi Sun wrote:
> This patch implements the xl/xc changes to support set CBM
> for L2 CAT.
> 
> The new level option is introduced to original CAT setting
> command in order to set CBM for specified level CAT.
> - 'xl psr-cat-set' is updated to set cache capacity bitmasks(CBM)
>   for a domain according to input cache level.
> 
> root@:~$ xl psr-cat-set -l2 1 0x7f
> 
> root@:~$ xl psr-cat-show -l2 1
> Socket ID       : 0
> Default CBM     : 0xff
>    ID                     NAME             CBM
>     1                 ubuntu14            0x7f
> 
> Signed-off-by: He Chen <he.chen@linux.intel.com>
> Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> Acked-by: Wei Liu <wei.liu2@citrix.com>
> ---
> v9:
>     - handle the case to set both CODE and DATA for CDP at same time.
>       For such case, user does not input '-c' or '-d' to set CDP cbm.
>     - move xl_cmdimpl.c changes into xl/xl_psr.c.
>     - move xl_cmdtable.c changes into xl/xl_cmdtable.c.

Right. Since you've changed how this patch works, it'd be better to drop
my ack, so that I know it requires my review.

>  
>      rc = libxl__count_physical_sockets(gc, &nr_sockets);
>      if (rc) {
> @@ -331,10 +332,43 @@ int libxl_psr_cat_set_cbm(libxl_ctx *ctx, uint32_t domid,
>              break;
>  
>          xc_type = libxl__psr_cbm_type_to_libxc_psr_cat_type(type);
> -        if (xc_psr_cat_set_domain_data(ctx->xch, domid, xc_type,
> -                                       socketid, cbm)) {
> -            libxl__psr_cat_log_err_msg(gc, errno);
> -            rc = ERROR_FAIL;
> +
> +        if (xc_type == XC_PSR_CAT_L3_CBM) {
> +            if (xc_psr_cat_get_info(ctx->xch, socketid, 3, &cat_info.cos_max,
> +                                    &cat_info.cbm_len, &cat_info.cdp_enabled)) {
> +                libxl__psr_cat_log_err_msg(gc, errno);
> +                rc = ERROR_FAIL;
> +                goto out;
> +            }
> +        }
> +
> +        /*
> +         * User can set both CDP CODE and DATA at same time when the '-c' or
> +         * '-d' is not input. In such case, the input type is
> +         * LIBXL_PSR_CBM_TYPE_L3_CBM. So, we need check if cdp_enabled is true.
> +         * If it is true, we need set both CODE and DATA.

"When user sets both CDP CODE and DATA at the same time, the input type
is..."

No need to mention xl options here.  Please document xl manpage instead.

The rest looks sensible.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-28 12:20               ` Jan Beulich
@ 2017-03-29  1:20                 ` Yi Sun
  2017-03-29  1:36                   ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-29  1:20 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-28 06:20:48, Jan Beulich wrote:
> >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
> > I think we at least need a 'get_val()' hook.
> 
> Of course.
> 
> > I try to implement CAT/CDP hook.
> > Please help to check if this is what you thought.
> 
> One remark below, but other than that - yes.
> 
> > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
> >                         enum cbm_type type, int flag, uint32_t *val)
> > {
> >     *val = feat->cos_reg_val[cos];
> > }
> > 
> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> >                            enum cbm_type type, int flag, uint32_t *val)
> > {
> >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
> >         *val = get_cdp_data(feat, cos);
> >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
> >         *val = get_cdp_code(feat, cos);
> > }
> 
> Why the redundancy between type and flag?
> 
For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
value. For other cases, we use flag as cos_num index to get either DATA or
CODE.

> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 24/25] tools: L2 CAT: support set cbm for L2 CAT.
  2017-03-28 14:04   ` Wei Liu
@ 2017-03-29  1:21     ` Yi Sun
  0 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-29  1:21 UTC (permalink / raw)
  To: Wei Liu
  Cc: kevin.tian, andrew.cooper3, dario.faggioli, he.chen, ian.jackson,
	mengxu, jbeulich, chao.p.peng, xen-devel, roger.pau

On 17-03-28 15:04:03, Wei Liu wrote:
> On Thu, Mar 16, 2017 at 07:08:14PM +0800, Yi Sun wrote:
> > This patch implements the xl/xc changes to support set CBM
> > for L2 CAT.
> > 
> > The new level option is introduced to original CAT setting
> > command in order to set CBM for specified level CAT.
> > - 'xl psr-cat-set' is updated to set cache capacity bitmasks(CBM)
> >   for a domain according to input cache level.
> > 
> > root@:~$ xl psr-cat-set -l2 1 0x7f
> > 
> > root@:~$ xl psr-cat-show -l2 1
> > Socket ID       : 0
> > Default CBM     : 0xff
> >    ID                     NAME             CBM
> >     1                 ubuntu14            0x7f
> > 
> > Signed-off-by: He Chen <he.chen@linux.intel.com>
> > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
> > Acked-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> > v9:
> >     - handle the case to set both CODE and DATA for CDP at same time.
> >       For such case, user does not input '-c' or '-d' to set CDP cbm.
> >     - move xl_cmdimpl.c changes into xl/xl_psr.c.
> >     - move xl_cmdtable.c changes into xl/xl_cmdtable.c.
> 
> Right. Since you've changed how this patch works, it'd be better to drop
> my ack, so that I know it requires my review.
> 
Sorry for that. I will remember this.

> >  
> >      rc = libxl__count_physical_sockets(gc, &nr_sockets);
> >      if (rc) {
> > @@ -331,10 +332,43 @@ int libxl_psr_cat_set_cbm(libxl_ctx *ctx, uint32_t domid,
> >              break;
> >  
> >          xc_type = libxl__psr_cbm_type_to_libxc_psr_cat_type(type);
> > -        if (xc_psr_cat_set_domain_data(ctx->xch, domid, xc_type,
> > -                                       socketid, cbm)) {
> > -            libxl__psr_cat_log_err_msg(gc, errno);
> > -            rc = ERROR_FAIL;
> > +
> > +        if (xc_type == XC_PSR_CAT_L3_CBM) {
> > +            if (xc_psr_cat_get_info(ctx->xch, socketid, 3, &cat_info.cos_max,
> > +                                    &cat_info.cbm_len, &cat_info.cdp_enabled)) {
> > +                libxl__psr_cat_log_err_msg(gc, errno);
> > +                rc = ERROR_FAIL;
> > +                goto out;
> > +            }
> > +        }
> > +
> > +        /*
> > +         * User can set both CDP CODE and DATA at same time when the '-c' or
> > +         * '-d' is not input. In such case, the input type is
> > +         * LIBXL_PSR_CBM_TYPE_L3_CBM. So, we need check if cdp_enabled is true.
> > +         * If it is true, we need set both CODE and DATA.
> 
> "When user sets both CDP CODE and DATA at the same time, the input type
> is..."
> 
> No need to mention xl options here.  Please document xl manpage instead.
> 
> The rest looks sensible.
> 
Thanks a lot for your review and comments!

> Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-29  1:20                 ` Yi Sun
@ 2017-03-29  1:36                   ` Yi Sun
  2017-03-29  9:57                     ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-29  1:36 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, chao.p.peng, xen-devel, roger.pau

On 17-03-29 09:20:21, Yi Sun wrote:
> On 17-03-28 06:20:48, Jan Beulich wrote:
> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
> > > I think we at least need a 'get_val()' hook.
> > 
> > Of course.
> > 
> > > I try to implement CAT/CDP hook.
> > > Please help to check if this is what you thought.
> > 
> > One remark below, but other than that - yes.
> > 
> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
> > >                         enum cbm_type type, int flag, uint32_t *val)
> > > {
> > >     *val = feat->cos_reg_val[cos];
> > > }
> > > 
> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> > >                            enum cbm_type type, int flag, uint32_t *val)
> > > {
> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
> > >         *val = get_cdp_data(feat, cos);
> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
> > >         *val = get_cdp_code(feat, cos);
> > > }
> > 
> > Why the redundancy between type and flag?
> > 
> For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
> value. For other cases, we use flag as cos_num index to get either DATA or
> CODE.
> 
Let me explain more to avoid confusion. For other cases, we use cos_num as
index to get values from a feature. In these cases, we do not know the
cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
know which value should be returned.

> > Jan
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-29  1:36                   ` Yi Sun
@ 2017-03-29  9:57                     ` Jan Beulich
  2017-03-30  1:37                       ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-29  9:57 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-29 09:20:21, Yi Sun wrote:
>> On 17-03-28 06:20:48, Jan Beulich wrote:
>> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
>> > > I think we at least need a 'get_val()' hook.
>> > 
>> > Of course.
>> > 
>> > > I try to implement CAT/CDP hook.
>> > > Please help to check if this is what you thought.
>> > 
>> > One remark below, but other than that - yes.
>> > 
>> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
>> > >                         enum cbm_type type, int flag, uint32_t *val)
>> > > {
>> > >     *val = feat->cos_reg_val[cos];
>> > > }
>> > > 
>> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
>> > >                            enum cbm_type type, int flag, uint32_t *val)
>> > > {
>> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
>> > >         *val = get_cdp_data(feat, cos);
>> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
>> > >         *val = get_cdp_code(feat, cos);
>> > > }
>> > 
>> > Why the redundancy between type and flag?
>> > 
>> For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
>> value. For other cases, we use flag as cos_num index to get either DATA or
>> CODE.
>> 
> Let me explain more to avoid confusion. For other cases, we use cos_num as
> index to get values from a feature. In these cases, we do not know the
> cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
> know which value should be returned.

I'm pretty sure this redundancy can be avoided.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-29  9:57                     ` Jan Beulich
@ 2017-03-30  1:37                       ` Yi Sun
  2017-03-30  1:39                         ` Yi Sun
  2017-03-30 11:55                         ` Jan Beulich
  0 siblings, 2 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-30  1:37 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-29 03:57:52, Jan Beulich wrote:
> >>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-29 09:20:21, Yi Sun wrote:
> >> On 17-03-28 06:20:48, Jan Beulich wrote:
> >> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
> >> > > I think we at least need a 'get_val()' hook.
> >> > 
> >> > Of course.
> >> > 
> >> > > I try to implement CAT/CDP hook.
> >> > > Please help to check if this is what you thought.
> >> > 
> >> > One remark below, but other than that - yes.
> >> > 
> >> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
> >> > >                         enum cbm_type type, int flag, uint32_t *val)
> >> > > {
> >> > >     *val = feat->cos_reg_val[cos];
> >> > > }
> >> > > 
> >> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> >> > >                            enum cbm_type type, int flag, uint32_t *val)
> >> > > {
> >> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
> >> > >         *val = get_cdp_data(feat, cos);
> >> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
> >> > >         *val = get_cdp_code(feat, cos);
> >> > > }
> >> > 
> >> > Why the redundancy between type and flag?
> >> > 
> >> For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
> >> value. For other cases, we use flag as cos_num index to get either DATA or
> >> CODE.
> >> 
> > Let me explain more to avoid confusion. For other cases, we use cos_num as
> > index to get values from a feature. In these cases, we do not know the
> > cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
> > know which value should be returned.
> 
> I'm pretty sure this redundancy can be avoided.
> 
Then, I think I have to reuse the 'type'. As only CDP needs type to decide
which value to be returned so far, I think I can implement codes like below
to make CDP can handle all scenarios.

static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
                           enum cbm_type type, uint32_t *val)
{
    if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0xF000 )
        *val = get_cdp_data(feat, cos);
    if ( type == PSR_CBM_TYPE_L3_CODE || flag == 0xF001 )
        *val = get_cdp_code(feat, cos);
}

static bool fits_cos_max(...)
{
......
    for (i = 0; i < feat->props->cos_num; i++)
    {
        feat->props->get_val(feat, cos, i + 0xF000, &default_val);
        if ( val[i] == default_val )
            ......
    }
......
}

Is that good for you?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-30  1:37                       ` Yi Sun
@ 2017-03-30  1:39                         ` Yi Sun
  2017-03-30 11:55                         ` Jan Beulich
  1 sibling, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-30  1:39 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-30 09:37:33, Yi Sun wrote:
> On 17-03-29 03:57:52, Jan Beulich wrote:
> > >>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
> > > On 17-03-29 09:20:21, Yi Sun wrote:
> > >> On 17-03-28 06:20:48, Jan Beulich wrote:
> > >> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
> > >> > > I think we at least need a 'get_val()' hook.
> > >> > 
> > >> > Of course.
> > >> > 
> > >> > > I try to implement CAT/CDP hook.
> > >> > > Please help to check if this is what you thought.
> > >> > 
> > >> > One remark below, but other than that - yes.
> > >> > 
> > >> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
> > >> > >                         enum cbm_type type, int flag, uint32_t *val)
> > >> > > {
> > >> > >     *val = feat->cos_reg_val[cos];
> > >> > > }
> > >> > > 
> > >> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> > >> > >                            enum cbm_type type, int flag, uint32_t *val)
> > >> > > {
> > >> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
> > >> > >         *val = get_cdp_data(feat, cos);
> > >> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
> > >> > >         *val = get_cdp_code(feat, cos);
> > >> > > }
> > >> > 
> > >> > Why the redundancy between type and flag?
> > >> > 
> > >> For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
> > >> value. For other cases, we use flag as cos_num index to get either DATA or
> > >> CODE.
> > >> 
> > > Let me explain more to avoid confusion. For other cases, we use cos_num as
> > > index to get values from a feature. In these cases, we do not know the
> > > cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
> > > know which value should be returned.
> > 
> > I'm pretty sure this redundancy can be avoided.
> > 
> Then, I think I have to reuse the 'type'. As only CDP needs type to decide
> which value to be returned so far, I think I can implement codes like below
> to make CDP can handle all scenarios.
> 
> static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
>                            enum cbm_type type, uint32_t *val)
> {
>     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0xF000 )
>         *val = get_cdp_data(feat, cos);
>     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 0xF001 )
>         *val = get_cdp_code(feat, cos);
> }
> 
> static bool fits_cos_max(...)
> {
> ......
>     for (i = 0; i < feat->props->cos_num; i++)
>     {
>         feat->props->get_val(feat, cos, i + 0xF000, &default_val);
>         if ( val[i] == default_val )
>             ......
>     }
> ......
> }
> 
> Is that good for you?

Sorry, a mistake, forgot to change 'flag' to 'type' in 'l3_cdp_get_val'.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-30  1:37                       ` Yi Sun
  2017-03-30  1:39                         ` Yi Sun
@ 2017-03-30 11:55                         ` Jan Beulich
  2017-03-30 12:10                           ` Yi Sun
  1 sibling, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-30 11:55 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 30.03.17 at 03:37, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-29 03:57:52, Jan Beulich wrote:
>> >>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-29 09:20:21, Yi Sun wrote:
>> >> On 17-03-28 06:20:48, Jan Beulich wrote:
>> >> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
>> >> > > I think we at least need a 'get_val()' hook.
>> >> > 
>> >> > Of course.
>> >> > 
>> >> > > I try to implement CAT/CDP hook.
>> >> > > Please help to check if this is what you thought.
>> >> > 
>> >> > One remark below, but other than that - yes.
>> >> > 
>> >> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
>> >> > >                         enum cbm_type type, int flag, uint32_t *val)
>> >> > > {
>> >> > >     *val = feat->cos_reg_val[cos];
>> >> > > }
>> >> > > 
>> >> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int 
> cos,
>> >> > >                            enum cbm_type type, int flag, uint32_t *val)
>> >> > > {
>> >> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
>> >> > >         *val = get_cdp_data(feat, cos);
>> >> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
>> >> > >         *val = get_cdp_code(feat, cos);
>> >> > > }
>> >> > 
>> >> > Why the redundancy between type and flag?
>> >> > 
>> >> For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
>> >> value. For other cases, we use flag as cos_num index to get either DATA or
>> >> CODE.
>> >> 
>> > Let me explain more to avoid confusion. For other cases, we use cos_num as
>> > index to get values from a feature. In these cases, we do not know the
>> > cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
>> > know which value should be returned.
>> 
>> I'm pretty sure this redundancy can be avoided.
>> 
> Then, I think I have to reuse the 'type'. As only CDP needs type to decide
> which value to be returned so far, I think I can implement codes like below
> to make CDP can handle all scenarios.
> 
> static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
>                            enum cbm_type type, uint32_t *val)
> {
>     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0xF000 )
>         *val = get_cdp_data(feat, cos);
>     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 0xF001 )
>         *val = get_cdp_code(feat, cos);
> }
> 
> static bool fits_cos_max(...)
> {
> ......
>     for (i = 0; i < feat->props->cos_num; i++)
>     {
>         feat->props->get_val(feat, cos, i + 0xF000, &default_val);
>         if ( val[i] == default_val )
>             ......
>     }
> ......
> }
> 
> Is that good for you?

Urgh - no, not really: This is hackery. Do you have a tree
somewhere so I could look at the file after at least the CDP
patches have all been applied, to see whether I have a
better idea? Alternatively, could you attach psr.c the way
you have it right now with the entire series applied?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-30 11:55                         ` Jan Beulich
@ 2017-03-30 12:10                           ` Yi Sun
  2017-03-31  8:47                             ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-30 12:10 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-30 05:55:52, Jan Beulich wrote:
> >>> On 30.03.17 at 03:37, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-29 03:57:52, Jan Beulich wrote:
> >> >>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-03-29 09:20:21, Yi Sun wrote:
> >> >> On 17-03-28 06:20:48, Jan Beulich wrote:
> >> >> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
> >> >> > > I think we at least need a 'get_val()' hook.
> >> >> > 
> >> >> > Of course.
> >> >> > 
> >> >> > > I try to implement CAT/CDP hook.
> >> >> > > Please help to check if this is what you thought.
> >> >> > 
> >> >> > One remark below, but other than that - yes.
> >> >> > 
> >> >> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
> >> >> > >                         enum cbm_type type, int flag, uint32_t *val)
> >> >> > > {
> >> >> > >     *val = feat->cos_reg_val[cos];
> >> >> > > }
> >> >> > > 
> >> >> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int 
> > cos,
> >> >> > >                            enum cbm_type type, int flag, uint32_t *val)
> >> >> > > {
> >> >> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
> >> >> > >         *val = get_cdp_data(feat, cos);
> >> >> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
> >> >> > >         *val = get_cdp_code(feat, cos);
> >> >> > > }
> >> >> > 
> >> >> > Why the redundancy between type and flag?
> >> >> > 
> >> >> For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
> >> >> value. For other cases, we use flag as cos_num index to get either DATA or
> >> >> CODE.
> >> >> 
> >> > Let me explain more to avoid confusion. For other cases, we use cos_num as
> >> > index to get values from a feature. In these cases, we do not know the
> >> > cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
> >> > know which value should be returned.
> >> 
> >> I'm pretty sure this redundancy can be avoided.
> >> 
> > Then, I think I have to reuse the 'type'. As only CDP needs type to decide
> > which value to be returned so far, I think I can implement codes like below
> > to make CDP can handle all scenarios.
> > 
> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> >                            enum cbm_type type, uint32_t *val)
> > {
> >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0xF000 )
> >         *val = get_cdp_data(feat, cos);
> >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 0xF001 )
> >         *val = get_cdp_code(feat, cos);
> > }
> > 
> > static bool fits_cos_max(...)
> > {
> > ......
> >     for (i = 0; i < feat->props->cos_num; i++)
> >     {
> >         feat->props->get_val(feat, cos, i + 0xF000, &default_val);
> >         if ( val[i] == default_val )
> >             ......
> >     }
> > ......
> > }
> > 
> > Is that good for you?
> 
> Urgh - no, not really: This is hackery. Do you have a tree
> somewhere so I could look at the file after at least the CDP
> patches have all been applied, to see whether I have a
> better idea? Alternatively, could you attach psr.c the way
> you have it right now with the entire series applied?
> 
I think you can check v9 codes here:
https://github.com/yisun-git/xen/tree/l2_cat_v9

Thanks,
Sun Yi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-30 12:10                           ` Yi Sun
@ 2017-03-31  8:47                             ` Jan Beulich
  2017-03-31  9:12                               ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-31  8:47 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 30.03.17 at 14:10, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-30 05:55:52, Jan Beulich wrote:
>> >>> On 30.03.17 at 03:37, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-29 03:57:52, Jan Beulich wrote:
>> >> >>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
>> >> > On 17-03-29 09:20:21, Yi Sun wrote:
>> >> >> On 17-03-28 06:20:48, Jan Beulich wrote:
>> >> >> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
>> >> >> > > I think we at least need a 'get_val()' hook.
>> >> >> > 
>> >> >> > Of course.
>> >> >> > 
>> >> >> > > I try to implement CAT/CDP hook.
>> >> >> > > Please help to check if this is what you thought.
>> >> >> > 
>> >> >> > One remark below, but other than that - yes.
>> >> >> > 
>> >> >> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
>> >> >> > >                         enum cbm_type type, int flag, uint32_t *val)
>> >> >> > > {
>> >> >> > >     *val = feat->cos_reg_val[cos];
>> >> >> > > }
>> >> >> > > 
>> >> >> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int 
>> > cos,
>> >> >> > >                            enum cbm_type type, int flag, uint32_t *val)
>> >> >> > > {
>> >> >> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
>> >> >> > >         *val = get_cdp_data(feat, cos);
>> >> >> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
>> >> >> > >         *val = get_cdp_code(feat, cos);
>> >> >> > > }
>> >> >> > 
>> >> >> > Why the redundancy between type and flag?
>> >> >> > 
>> >> >> For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
>> >> >> value. For other cases, we use flag as cos_num index to get either DATA or
>> >> >> CODE.
>> >> >> 
>> >> > Let me explain more to avoid confusion. For other cases, we use cos_num as
>> >> > index to get values from a feature. In these cases, we do not know the
>> >> > cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
>> >> > know which value should be returned.
>> >> 
>> >> I'm pretty sure this redundancy can be avoided.
>> >> 
>> > Then, I think I have to reuse the 'type'. As only CDP needs type to decide
>> > which value to be returned so far, I think I can implement codes like below
>> > to make CDP can handle all scenarios.
>> > 
>> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
>> >                            enum cbm_type type, uint32_t *val)
>> > {
>> >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0xF000 )
>> >         *val = get_cdp_data(feat, cos);
>> >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 0xF001 )
>> >         *val = get_cdp_code(feat, cos);
>> > }
>> > 
>> > static bool fits_cos_max(...)
>> > {
>> > ......
>> >     for (i = 0; i < feat->props->cos_num; i++)
>> >     {
>> >         feat->props->get_val(feat, cos, i + 0xF000, &default_val);
>> >         if ( val[i] == default_val )
>> >             ......
>> >     }
>> > ......
>> > }
>> > 
>> > Is that good for you?
>> 
>> Urgh - no, not really: This is hackery. Do you have a tree
>> somewhere so I could look at the file after at least the CDP
>> patches have all been applied, to see whether I have a
>> better idea? Alternatively, could you attach psr.c the way
>> you have it right now with the entire series applied?
>> 
> I think you can check v9 codes here:
> https://github.com/yisun-git/xen/tree/l2_cat_v9 

Looking at this made me notice that cat_get_old_val() passes a
bogus literal 0 to cat_get_val(), which needs taking care of too.
One option I can see is for each feature to make available an
array of type enum cbm_type, with cos_num elements. The order
would match that of the order of values in their arrays. This will
allow elimination of all of the get_old_val, compare_val, and
fits_cos_max hooks afaict. At that point the "new" in
set_new_val is also no longer needed to contrast with the "old"
in the no longer existing get_old_val. Arguably insert_val may
be even slightly more precise in describing what the function
does.

The checks done first in what is currently *_set_new_val() also
look like they could be moved into the (generic) caller(s), making
clear that at least for now even cbm_len (just like suggested for
cos_max) should be a generic rather than unionized field in
struct feat_node. In the worst case a new check_val() hook
might be needed.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-31  8:47                             ` Jan Beulich
@ 2017-03-31  9:12                               ` Yi Sun
  2017-03-31  9:18                                 ` Yi Sun
  2017-03-31 10:19                                 ` Jan Beulich
  0 siblings, 2 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-31  9:12 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-31 02:47:25, Jan Beulich wrote:
> >>> On 30.03.17 at 14:10, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-30 05:55:52, Jan Beulich wrote:
> >> >>> On 30.03.17 at 03:37, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-03-29 03:57:52, Jan Beulich wrote:
> >> >> >>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
> >> >> > On 17-03-29 09:20:21, Yi Sun wrote:
> >> >> >> On 17-03-28 06:20:48, Jan Beulich wrote:
> >> >> >> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
> >> >> >> > > I think we at least need a 'get_val()' hook.
> >> >> >> > 
> >> >> >> > Of course.
> >> >> >> > 
> >> >> >> > > I try to implement CAT/CDP hook.
> >> >> >> > > Please help to check if this is what you thought.
> >> >> >> > 
> >> >> >> > One remark below, but other than that - yes.
> >> >> >> > 
> >> >> >> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
> >> >> >> > >                         enum cbm_type type, int flag, uint32_t *val)
> >> >> >> > > {
> >> >> >> > >     *val = feat->cos_reg_val[cos];
> >> >> >> > > }
> >> >> >> > > 
> >> >> >> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int 
> >> > cos,
> >> >> >> > >                            enum cbm_type type, int flag, uint32_t *val)
> >> >> >> > > {
> >> >> >> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
> >> >> >> > >         *val = get_cdp_data(feat, cos);
> >> >> >> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
> >> >> >> > >         *val = get_cdp_code(feat, cos);
> >> >> >> > > }
> >> >> >> > 
> >> >> >> > Why the redundancy between type and flag?
> >> >> >> > 
> >> >> >> For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
> >> >> >> value. For other cases, we use flag as cos_num index to get either DATA or
> >> >> >> CODE.
> >> >> >> 
> >> >> > Let me explain more to avoid confusion. For other cases, we use cos_num as
> >> >> > index to get values from a feature. In these cases, we do not know the
> >> >> > cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
> >> >> > know which value should be returned.
> >> >> 
> >> >> I'm pretty sure this redundancy can be avoided.
> >> >> 
> >> > Then, I think I have to reuse the 'type'. As only CDP needs type to decide
> >> > which value to be returned so far, I think I can implement codes like below
> >> > to make CDP can handle all scenarios.
> >> > 
> >> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> >> >                            enum cbm_type type, uint32_t *val)
> >> > {
> >> >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0xF000 )
> >> >         *val = get_cdp_data(feat, cos);
> >> >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 0xF001 )
> >> >         *val = get_cdp_code(feat, cos);
> >> > }
> >> > 
> >> > static bool fits_cos_max(...)
> >> > {
> >> > ......
> >> >     for (i = 0; i < feat->props->cos_num; i++)
> >> >     {
> >> >         feat->props->get_val(feat, cos, i + 0xF000, &default_val);
> >> >         if ( val[i] == default_val )
> >> >             ......
> >> >     }
> >> > ......
> >> > }
> >> > 
> >> > Is that good for you?
> >> 
> >> Urgh - no, not really: This is hackery. Do you have a tree
> >> somewhere so I could look at the file after at least the CDP
> >> patches have all been applied, to see whether I have a
> >> better idea? Alternatively, could you attach psr.c the way
> >> you have it right now with the entire series applied?
> >> 
> > I think you can check v9 codes here:
> > https://github.com/yisun-git/xen/tree/l2_cat_v9 
> 
> Looking at this made me notice that cat_get_old_val() passes a
> bogus literal 0 to cat_get_val(), which needs taking care of too.
> One option I can see is for each feature to make available an
> array of type enum cbm_type, with cos_num elements. The order
> would match that of the order of values in their arrays. This will

Sorry, not very clear your meaning. How to do that? Could you please
provide pieces of codes? Thanks!

> allow elimination of all of the get_old_val, compare_val, and
> fits_cos_max hooks afaict. At that point the "new" in
> set_new_val is also no longer needed to contrast with the "old"
> in the no longer existing get_old_val. Arguably insert_val may
> be even slightly more precise in describing what the function
> does.

I have modified it to 'set_val_to_array'.

> 
> The checks done first in what is currently *_set_new_val() also
> look like they could be moved into the (generic) caller(s), making

MBA value is throttling value which is different with CBM. So, the
check is different. So, we have to have 'check_val()' at least.

> clear that at least for now even cbm_len (just like suggested for
> cos_max) should be a generic rather than unionized field in

MBA does not have cbm_len.

> struct feat_node. In the worst case a new check_val() hook
> might be needed.
> 
After analyzing codes again, for 'gather_val', 'compare_val' and
'fits_cos_max', we need get all values of a feature. E.g. we need
get both DATA and CODE for CDP. But for 'psr_get_val', we only need
get one value per cbm_type. So, I think these should be different
hooks. Can I implement 'get_val' for getting one value per cbm_type,
and 'get_vals' to get all values?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-31  9:12                               ` Yi Sun
@ 2017-03-31  9:18                                 ` Yi Sun
  2017-03-31 10:19                                 ` Jan Beulich
  1 sibling, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-31  9:18 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-31 17:12:36, Yi Sun wrote:
> On 17-03-31 02:47:25, Jan Beulich wrote:
> > >>> On 30.03.17 at 14:10, <yi.y.sun@linux.intel.com> wrote:
> > > On 17-03-30 05:55:52, Jan Beulich wrote:
> > >> >>> On 30.03.17 at 03:37, <yi.y.sun@linux.intel.com> wrote:
> > >> > On 17-03-29 03:57:52, Jan Beulich wrote:
> > >> >> >>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
> > >> >> > On 17-03-29 09:20:21, Yi Sun wrote:
> > >> >> >> On 17-03-28 06:20:48, Jan Beulich wrote:
> > >> >> >> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
> > >> >> >> > > I think we at least need a 'get_val()' hook.
> > >> >> >> > 
> > >> >> >> > Of course.
> > >> >> >> > 
> > >> >> >> > > I try to implement CAT/CDP hook.
> > >> >> >> > > Please help to check if this is what you thought.
> > >> >> >> > 
> > >> >> >> > One remark below, but other than that - yes.
> > >> >> >> > 
> > >> >> >> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
> > >> >> >> > >                         enum cbm_type type, int flag, uint32_t *val)
> > >> >> >> > > {
> > >> >> >> > >     *val = feat->cos_reg_val[cos];
> > >> >> >> > > }
> > >> >> >> > > 
> > >> >> >> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int 
> > >> > cos,
> > >> >> >> > >                            enum cbm_type type, int flag, uint32_t *val)
> > >> >> >> > > {
> > >> >> >> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
> > >> >> >> > >         *val = get_cdp_data(feat, cos);
> > >> >> >> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
> > >> >> >> > >         *val = get_cdp_code(feat, cos);
> > >> >> >> > > }
> > >> >> >> > 
> > >> >> >> > Why the redundancy between type and flag?
> > >> >> >> > 
> > >> >> >> For psr_get_val, upper layer input the cbm_type to get either DATA or CODE
> > >> >> >> value. For other cases, we use flag as cos_num index to get either DATA or
> > >> >> >> CODE.
> > >> >> >> 
> > >> >> > Let me explain more to avoid confusion. For other cases, we use cos_num as
> > >> >> > index to get values from a feature. In these cases, we do not know the
> > >> >> > cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
> > >> >> > know which value should be returned.
> > >> >> 
> > >> >> I'm pretty sure this redundancy can be avoided.
> > >> >> 
> > >> > Then, I think I have to reuse the 'type'. As only CDP needs type to decide
> > >> > which value to be returned so far, I think I can implement codes like below
> > >> > to make CDP can handle all scenarios.
> > >> > 
> > >> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> > >> >                            enum cbm_type type, uint32_t *val)
> > >> > {
> > >> >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0xF000 )
> > >> >         *val = get_cdp_data(feat, cos);
> > >> >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 0xF001 )
> > >> >         *val = get_cdp_code(feat, cos);
> > >> > }
> > >> > 
> > >> > static bool fits_cos_max(...)
> > >> > {
> > >> > ......
> > >> >     for (i = 0; i < feat->props->cos_num; i++)
> > >> >     {
> > >> >         feat->props->get_val(feat, cos, i + 0xF000, &default_val);
> > >> >         if ( val[i] == default_val )
> > >> >             ......
> > >> >     }
> > >> > ......
> > >> > }
> > >> > 
> > >> > Is that good for you?
> > >> 
> > >> Urgh - no, not really: This is hackery. Do you have a tree
> > >> somewhere so I could look at the file after at least the CDP
> > >> patches have all been applied, to see whether I have a
> > >> better idea? Alternatively, could you attach psr.c the way
> > >> you have it right now with the entire series applied?
> > >> 
> > > I think you can check v9 codes here:
> > > https://github.com/yisun-git/xen/tree/l2_cat_v9 
> > 
> > Looking at this made me notice that cat_get_old_val() passes a
> > bogus literal 0 to cat_get_val(), which needs taking care of too.
> > One option I can see is for each feature to make available an
> > array of type enum cbm_type, with cos_num elements. The order
> > would match that of the order of values in their arrays. This will
> 
> Sorry, not very clear your meaning. How to do that? Could you please
> provide pieces of codes? Thanks!
> 
> > allow elimination of all of the get_old_val, compare_val, and
> > fits_cos_max hooks afaict. At that point the "new" in
> > set_new_val is also no longer needed to contrast with the "old"
> > in the no longer existing get_old_val. Arguably insert_val may
> > be even slightly more precise in describing what the function
> > does.
> 
> I have modified it to 'set_val_to_array'.
> 
> > 
> > The checks done first in what is currently *_set_new_val() also
> > look like they could be moved into the (generic) caller(s), making
> 
> MBA value is throttling value which is different with CBM. So, the
> check is different. So, we have to have 'check_val()' at least.
> 
> > clear that at least for now even cbm_len (just like suggested for
> > cos_max) should be a generic rather than unionized field in
> 
> MBA does not have cbm_len.
> 
> > struct feat_node. In the worst case a new check_val() hook
> > might be needed.
> > 
> After analyzing codes again, for 'gather_val', 'compare_val' and
> 'fits_cos_max', we need get all values of a feature. E.g. we need
> get both DATA and CODE for CDP. But for 'psr_get_val', we only need
> get one value per cbm_type. So, I think these should be different
> hooks. Can I implement 'get_val' for getting one value per cbm_type,
> and 'get_vals' to get all values?

Of course, for cos_num=1 features, 'get_vals' is NULL and 'get_val'
will be used.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-31  9:12                               ` Yi Sun
  2017-03-31  9:18                                 ` Yi Sun
@ 2017-03-31 10:19                                 ` Jan Beulich
  2017-03-31 12:40                                   ` Yi Sun
  1 sibling, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-31 10:19 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 31.03.17 at 11:12, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-31 02:47:25, Jan Beulich wrote:
>> >>> On 30.03.17 at 14:10, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-30 05:55:52, Jan Beulich wrote:
>> >> >>> On 30.03.17 at 03:37, <yi.y.sun@linux.intel.com> wrote:
>> >> > On 17-03-29 03:57:52, Jan Beulich wrote:
>> >> >> >>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
>> >> >> > On 17-03-29 09:20:21, Yi Sun wrote:
>> >> >> >> On 17-03-28 06:20:48, Jan Beulich wrote:
>> >> >> >> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
>> >> >> >> > > I think we at least need a 'get_val()' hook.
>> >> >> >> > 
>> >> >> >> > Of course.
>> >> >> >> > 
>> >> >> >> > > I try to implement CAT/CDP hook.
>> >> >> >> > > Please help to check if this is what you thought.
>> >> >> >> > 
>> >> >> >> > One remark below, but other than that - yes.
>> >> >> >> > 
>> >> >> >> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
>> >> >> >> > >                         enum cbm_type type, int flag, uint32_t *val)
>> >> >> >> > > {
>> >> >> >> > >     *val = feat->cos_reg_val[cos];
>> >> >> >> > > }
>> >> >> >> > > 
>> >> >> >> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int 
>> >> > cos,
>> >> >> >> > >                            enum cbm_type type, int flag, uint32_t *val)
>> >> >> >> > > {
>> >> >> >> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
>> >> >> >> > >         *val = get_cdp_data(feat, cos);
>> >> >> >> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
>> >> >> >> > >         *val = get_cdp_code(feat, cos);
>> >> >> >> > > }
>> >> >> >> > 
>> >> >> >> > Why the redundancy between type and flag?
>> >> >> >> > 
>> >> >> >> For psr_get_val, upper layer input the cbm_type to get either DATA or 
> CODE
>> >> >> >> value. For other cases, we use flag as cos_num index to get either DATA 
> or
>> >> >> >> CODE.
>> >> >> >> 
>> >> >> > Let me explain more to avoid confusion. For other cases, we use cos_num 
> as
>> >> >> > index to get values from a feature. In these cases, we do not know the
>> >> >> > cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
>> >> >> > know which value should be returned.
>> >> >> 
>> >> >> I'm pretty sure this redundancy can be avoided.
>> >> >> 
>> >> > Then, I think I have to reuse the 'type'. As only CDP needs type to decide
>> >> > which value to be returned so far, I think I can implement codes like 
> below
>> >> > to make CDP can handle all scenarios.
>> >> > 
>> >> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
>> >> >                            enum cbm_type type, uint32_t *val)
>> >> > {
>> >> >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0xF000 )
>> >> >         *val = get_cdp_data(feat, cos);
>> >> >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 0xF001 )
>> >> >         *val = get_cdp_code(feat, cos);
>> >> > }
>> >> > 
>> >> > static bool fits_cos_max(...)
>> >> > {
>> >> > ......
>> >> >     for (i = 0; i < feat->props->cos_num; i++)
>> >> >     {
>> >> >         feat->props->get_val(feat, cos, i + 0xF000, &default_val);
>> >> >         if ( val[i] == default_val )
>> >> >             ......
>> >> >     }
>> >> > ......
>> >> > }
>> >> > 
>> >> > Is that good for you?
>> >> 
>> >> Urgh - no, not really: This is hackery. Do you have a tree
>> >> somewhere so I could look at the file after at least the CDP
>> >> patches have all been applied, to see whether I have a
>> >> better idea? Alternatively, could you attach psr.c the way
>> >> you have it right now with the entire series applied?
>> >> 
>> > I think you can check v9 codes here:
>> > https://github.com/yisun-git/xen/tree/l2_cat_v9 
>> 
>> Looking at this made me notice that cat_get_old_val() passes a
>> bogus literal 0 to cat_get_val(), which needs taking care of too.
>> One option I can see is for each feature to make available an
>> array of type enum cbm_type, with cos_num elements. The order
>> would match that of the order of values in their arrays. This will
> 
> Sorry, not very clear your meaning. How to do that? Could you please
> provide pieces of codes? Thanks!

I'm sorry, but I'm afraid I don't see how I would reasonably supply
code here without taking over your series altogether (which I don't
intend to do). What is unclear with, at the example of CDP, you
needing to add an array at initialization time, slot 0 of which holds
PSR_CBM_TYPE_L3_DATA and slot 1 PSR_CBM_TYPE_L3_CODE (or
the other way around). Granted I was wrong with the type of the
array (as the above aren't enum psr_feat_type enumerators, but
enum cbm_type ones), but I think the basic idea should have been
clear anyway: You need to provide a way for generic code to pass
suitable type information into ->get_val().

>> allow elimination of all of the get_old_val, compare_val, and
>> fits_cos_max hooks afaict. At that point the "new" in
>> set_new_val is also no longer needed to contrast with the "old"
>> in the no longer existing get_old_val. Arguably insert_val may
>> be even slightly more precise in describing what the function
>> does.
> 
> I have modified it to 'set_val_to_array'.

I think we had that name already before, and I still consider it
unsuitable. If insert_val() doesn't suit you, use the slower to
type insert_val_into_array().

>> The checks done first in what is currently *_set_new_val() also
>> look like they could be moved into the (generic) caller(s), making
> 
> MBA value is throttling value which is different with CBM. So, the
> check is different. So, we have to have 'check_val()' at least.

At the point you introduce MBA support, which isn't in this series.

>> clear that at least for now even cbm_len (just like suggested for
>> cos_max) should be a generic rather than unionized field in
> 
> MBA does not have cbm_len.

As per above, it would need abstracting out _then_, not _now_.

>> struct feat_node. In the worst case a new check_val() hook
>> might be needed.
>> 
> After analyzing codes again, for 'gather_val', 'compare_val' and
> 'fits_cos_max', we need get all values of a feature. E.g. we need
> get both DATA and CODE for CDP. But for 'psr_get_val', we only need
> get one value per cbm_type. So, I think these should be different
> hooks. Can I implement 'get_val' for getting one value per cbm_type,
> and 'get_vals' to get all values?

Why? Generic code knows how many times to call ->get_val(), by
way of knowing cos_num.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-31 10:19                                 ` Jan Beulich
@ 2017-03-31 12:40                                   ` Yi Sun
  2017-03-31 12:51                                     ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-31 12:40 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-31 04:19:49, Jan Beulich wrote:
> >>> On 31.03.17 at 11:12, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-31 02:47:25, Jan Beulich wrote:
> >> >>> On 30.03.17 at 14:10, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-03-30 05:55:52, Jan Beulich wrote:
> >> >> >>> On 30.03.17 at 03:37, <yi.y.sun@linux.intel.com> wrote:
> >> >> > On 17-03-29 03:57:52, Jan Beulich wrote:
> >> >> >> >>> On 29.03.17 at 03:36, <yi.y.sun@linux.intel.com> wrote:
> >> >> >> > On 17-03-29 09:20:21, Yi Sun wrote:
> >> >> >> >> On 17-03-28 06:20:48, Jan Beulich wrote:
> >> >> >> >> > >>> On 28.03.17 at 13:59, <yi.y.sun@linux.intel.com> wrote:
> >> >> >> >> > > I think we at least need a 'get_val()' hook.
> >> >> >> >> > 
> >> >> >> >> > Of course.
> >> >> >> >> > 
> >> >> >> >> > > I try to implement CAT/CDP hook.
> >> >> >> >> > > Please help to check if this is what you thought.
> >> >> >> >> > 
> >> >> >> >> > One remark below, but other than that - yes.
> >> >> >> >> > 
> >> >> >> >> > > static void cat_get_val(const struct feat_node *feat, unsigned int cos,
> >> >> >> >> > >                         enum cbm_type type, int flag, uint32_t *val)
> >> >> >> >> > > {
> >> >> >> >> > >     *val = feat->cos_reg_val[cos];
> >> >> >> >> > > }
> >> >> >> >> > > 
> >> >> >> >> > > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int 
> >> >> > cos,
> >> >> >> >> > >                            enum cbm_type type, int flag, uint32_t *val)
> >> >> >> >> > > {
> >> >> >> >> > >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0 )
> >> >> >> >> > >         *val = get_cdp_data(feat, cos);
> >> >> >> >> > >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 1 )
> >> >> >> >> > >         *val = get_cdp_code(feat, cos);
> >> >> >> >> > > }
> >> >> >> >> > 
> >> >> >> >> > Why the redundancy between type and flag?
> >> >> >> >> > 
> >> >> >> >> For psr_get_val, upper layer input the cbm_type to get either DATA or 
> > CODE
> >> >> >> >> value. For other cases, we use flag as cos_num index to get either DATA 
> > or
> >> >> >> >> CODE.
> >> >> >> >> 
> >> >> >> > Let me explain more to avoid confusion. For other cases, we use cos_num 
> > as
> >> >> >> > index to get values from a feature. In these cases, we do not know the
> >> >> >> > cbm_type of the feature. So, I use the cos_num as flag to make 'get_val'
> >> >> >> > know which value should be returned.
> >> >> >> 
> >> >> >> I'm pretty sure this redundancy can be avoided.
> >> >> >> 
> >> >> > Then, I think I have to reuse the 'type'. As only CDP needs type to decide
> >> >> > which value to be returned so far, I think I can implement codes like 
> > below
> >> >> > to make CDP can handle all scenarios.
> >> >> > 
> >> >> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> >> >> >                            enum cbm_type type, uint32_t *val)
> >> >> > {
> >> >> >     if ( type == PSR_CBM_TYPE_L3_DATA || flag == 0xF000 )
> >> >> >         *val = get_cdp_data(feat, cos);
> >> >> >     if ( type == PSR_CBM_TYPE_L3_CODE || flag == 0xF001 )
> >> >> >         *val = get_cdp_code(feat, cos);
> >> >> > }
> >> >> > 
> >> >> > static bool fits_cos_max(...)
> >> >> > {
> >> >> > ......
> >> >> >     for (i = 0; i < feat->props->cos_num; i++)
> >> >> >     {
> >> >> >         feat->props->get_val(feat, cos, i + 0xF000, &default_val);
> >> >> >         if ( val[i] == default_val )
> >> >> >             ......
> >> >> >     }
> >> >> > ......
> >> >> > }
> >> >> > 
> >> >> > Is that good for you?
> >> >> 
> >> >> Urgh - no, not really: This is hackery. Do you have a tree
> >> >> somewhere so I could look at the file after at least the CDP
> >> >> patches have all been applied, to see whether I have a
> >> >> better idea? Alternatively, could you attach psr.c the way
> >> >> you have it right now with the entire series applied?
> >> >> 
> >> > I think you can check v9 codes here:
> >> > https://github.com/yisun-git/xen/tree/l2_cat_v9 
> >> 
> >> Looking at this made me notice that cat_get_old_val() passes a
> >> bogus literal 0 to cat_get_val(), which needs taking care of too.
> >> One option I can see is for each feature to make available an
> >> array of type enum cbm_type, with cos_num elements. The order
> >> would match that of the order of values in their arrays. This will
> > 
> > Sorry, not very clear your meaning. How to do that? Could you please
> > provide pieces of codes? Thanks!
> 
> I'm sorry, but I'm afraid I don't see how I would reasonably supply
> code here without taking over your series altogether (which I don't
> intend to do). What is unclear with, at the example of CDP, you
> needing to add an array at initialization time, slot 0 of which holds
> PSR_CBM_TYPE_L3_DATA and slot 1 PSR_CBM_TYPE_L3_CODE (or
> the other way around). Granted I was wrong with the type of the
> array (as the above aren't enum psr_feat_type enumerators, but
> enum cbm_type ones), but I think the basic idea should have been
> clear anyway: You need to provide a way for generic code to pass
> suitable type information into ->get_val().
> 
May I change the 'get_val()' parameter 'enum cbm_type' to a generic type
'unsigned int' to make it be a flexible type,  and then combine feature
type with cos_num together as a flag to indicate which feature it is,
which value to get and distinguish it with cbm_type? For example:

#define CDP_GATHER_BOTH_DATA ( PSR_SOCKET_L3_CDP << 16 )
#define CDP_GATHER_BOTH_CODE ( PSR_SOCKET_L3_CDP << 16 + 1 )
static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
                           unsigned int type, uint32_t *val)
{
    switch ( type )
    {
    case PSR_CBM_TYPE_L3_DATA:
    case CDP_GATHER_BOTH_DATA:
        *val = get_cdp_data(feat, cos);
        break;
    case PSR_CBM_TYPE_L3_CODE:
    case CDP_GATHER_BOTH_CODE:
        *val = get_cdp_code(feat, cos);
        break;
    }
}

/* Example for getting value by cos_num. */
static bool fits_cos_max(...)
{
    enum psr_feat_type feat_type;
...
    for ( j = 0; j < feat->props->cos_num; j++ )
    {
        unsigned int type = feat_type << 16 + j;
        feat->props->get_val(feat, 0, type, &default_val);
...
}

/* Example for getting value by cbm_type. */
int psr_get_val(enum cbm_type type...)
{
...
    feat->props->get_val(feat, cos, type, val);
...
}

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-31 12:40                                   ` Yi Sun
@ 2017-03-31 12:51                                     ` Jan Beulich
  2017-03-31 13:22                                       ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-31 12:51 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 31.03.17 at 14:40, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-31 04:19:49, Jan Beulich wrote:
>> >>> On 31.03.17 at 11:12, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-31 02:47:25, Jan Beulich wrote:
>> >> >>> On 30.03.17 at 14:10, <yi.y.sun@linux.intel.com> wrote:
>> >> > I think you can check v9 codes here:
>> >> > https://github.com/yisun-git/xen/tree/l2_cat_v9 
>> >> 
>> >> Looking at this made me notice that cat_get_old_val() passes a
>> >> bogus literal 0 to cat_get_val(), which needs taking care of too.
>> >> One option I can see is for each feature to make available an
>> >> array of type enum cbm_type, with cos_num elements. The order
>> >> would match that of the order of values in their arrays. This will
>> > 
>> > Sorry, not very clear your meaning. How to do that? Could you please
>> > provide pieces of codes? Thanks!
>> 
>> I'm sorry, but I'm afraid I don't see how I would reasonably supply
>> code here without taking over your series altogether (which I don't
>> intend to do). What is unclear with, at the example of CDP, you
>> needing to add an array at initialization time, slot 0 of which holds
>> PSR_CBM_TYPE_L3_DATA and slot 1 PSR_CBM_TYPE_L3_CODE (or
>> the other way around). Granted I was wrong with the type of the
>> array (as the above aren't enum psr_feat_type enumerators, but
>> enum cbm_type ones), but I think the basic idea should have been
>> clear anyway: You need to provide a way for generic code to pass
>> suitable type information into ->get_val().
>> 
> May I change the 'get_val()' parameter 'enum cbm_type' to a generic type
> 'unsigned int' to make it be a flexible type,  and then combine feature
> type with cos_num together as a flag to indicate which feature it is,
> which value to get and distinguish it with cbm_type? For example:
> 
> #define CDP_GATHER_BOTH_DATA ( PSR_SOCKET_L3_CDP << 16 )
> #define CDP_GATHER_BOTH_CODE ( PSR_SOCKET_L3_CDP << 16 + 1 )
> static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
>                            unsigned int type, uint32_t *val)
> {
>     switch ( type )
>     {
>     case PSR_CBM_TYPE_L3_DATA:
>     case CDP_GATHER_BOTH_DATA:
>         *val = get_cdp_data(feat, cos);
>         break;
>     case PSR_CBM_TYPE_L3_CODE:
>     case CDP_GATHER_BOTH_CODE:
>         *val = get_cdp_code(feat, cos);
>         break;
>     }
> }

The two case labels are still indicative of unnecessary redundancy
(and, even right now only highly theoretical, risk of collisions). What's
wrong with the model I've proposed?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-31 12:51                                     ` Jan Beulich
@ 2017-03-31 13:22                                       ` Yi Sun
  2017-03-31 14:35                                         ` Jan Beulich
  0 siblings, 1 reply; 105+ messages in thread
From: Yi Sun @ 2017-03-31 13:22 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-31 06:51:07, Jan Beulich wrote:
> >>> On 31.03.17 at 14:40, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-31 04:19:49, Jan Beulich wrote:
> >> >>> On 31.03.17 at 11:12, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-03-31 02:47:25, Jan Beulich wrote:
> >> >> >>> On 30.03.17 at 14:10, <yi.y.sun@linux.intel.com> wrote:
> >> >> > I think you can check v9 codes here:
> >> >> > https://github.com/yisun-git/xen/tree/l2_cat_v9 
> >> >> 
> >> >> Looking at this made me notice that cat_get_old_val() passes a
> >> >> bogus literal 0 to cat_get_val(), which needs taking care of too.
> >> >> One option I can see is for each feature to make available an
> >> >> array of type enum cbm_type, with cos_num elements. The order
> >> >> would match that of the order of values in their arrays. This will
> >> > 
> >> > Sorry, not very clear your meaning. How to do that? Could you please
> >> > provide pieces of codes? Thanks!
> >> 
> >> I'm sorry, but I'm afraid I don't see how I would reasonably supply
> >> code here without taking over your series altogether (which I don't
> >> intend to do). What is unclear with, at the example of CDP, you
> >> needing to add an array at initialization time, slot 0 of which holds
> >> PSR_CBM_TYPE_L3_DATA and slot 1 PSR_CBM_TYPE_L3_CODE (or
> >> the other way around). Granted I was wrong with the type of the
> >> array (as the above aren't enum psr_feat_type enumerators, but
> >> enum cbm_type ones), but I think the basic idea should have been
> >> clear anyway: You need to provide a way for generic code to pass
> >> suitable type information into ->get_val().
> >> 
> > May I change the 'get_val()' parameter 'enum cbm_type' to a generic type
> > 'unsigned int' to make it be a flexible type,  and then combine feature
> > type with cos_num together as a flag to indicate which feature it is,
> > which value to get and distinguish it with cbm_type? For example:
> > 
> > #define CDP_GATHER_BOTH_DATA ( PSR_SOCKET_L3_CDP << 16 )
> > #define CDP_GATHER_BOTH_CODE ( PSR_SOCKET_L3_CDP << 16 + 1 )
> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> >                            unsigned int type, uint32_t *val)
> > {
> >     switch ( type )
> >     {
> >     case PSR_CBM_TYPE_L3_DATA:
> >     case CDP_GATHER_BOTH_DATA:
> >         *val = get_cdp_data(feat, cos);
> >         break;
> >     case PSR_CBM_TYPE_L3_CODE:
> >     case CDP_GATHER_BOTH_CODE:
> >         *val = get_cdp_code(feat, cos);
> >         break;
> >     }
> > }
> 
> The two case labels are still indicative of unnecessary redundancy
> (and, even right now only highly theoretical, risk of collisions). What's
> wrong with the model I've proposed?
> 
Oh, sorry. I did not understand your proposal just now so I provided another
solution.

After reading your suggestion again, I think your meaing is below in codes:

struct feat_props {
...
    unsigned int cos_num;
    enum cbm_type cos_to_type[2];
...
}

static void cat_init_feature(...)
{
...
    case PSR_SOCKET_L3_CDP:
        feat->props->cos_to_type[0] = PSR_CBM_TYPE_L3_DATA;
        feat->props->cos_to_type[1] = PSR_CBM_TYPE_L3_CODE;
...
}

Then, in functions to iterate 'cos_num', we can input 'cos_to_type[i]'. Right?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-31 13:22                                       ` Yi Sun
@ 2017-03-31 14:35                                         ` Jan Beulich
  2017-03-31 14:46                                           ` Yi Sun
  0 siblings, 1 reply; 105+ messages in thread
From: Jan Beulich @ 2017-03-31 14:35 UTC (permalink / raw)
  To: Yi Sun
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

>>> On 31.03.17 at 15:22, <yi.y.sun@linux.intel.com> wrote:
> On 17-03-31 06:51:07, Jan Beulich wrote:
>> >>> On 31.03.17 at 14:40, <yi.y.sun@linux.intel.com> wrote:
>> > On 17-03-31 04:19:49, Jan Beulich wrote:
>> >> >>> On 31.03.17 at 11:12, <yi.y.sun@linux.intel.com> wrote:
>> >> > On 17-03-31 02:47:25, Jan Beulich wrote:
>> >> >> >>> On 30.03.17 at 14:10, <yi.y.sun@linux.intel.com> wrote:
>> >> >> > I think you can check v9 codes here:
>> >> >> > https://github.com/yisun-git/xen/tree/l2_cat_v9 
>> >> >> 
>> >> >> Looking at this made me notice that cat_get_old_val() passes a
>> >> >> bogus literal 0 to cat_get_val(), which needs taking care of too.
>> >> >> One option I can see is for each feature to make available an
>> >> >> array of type enum cbm_type, with cos_num elements. The order
>> >> >> would match that of the order of values in their arrays. This will
>> >> > 
>> >> > Sorry, not very clear your meaning. How to do that? Could you please
>> >> > provide pieces of codes? Thanks!
>> >> 
>> >> I'm sorry, but I'm afraid I don't see how I would reasonably supply
>> >> code here without taking over your series altogether (which I don't
>> >> intend to do). What is unclear with, at the example of CDP, you
>> >> needing to add an array at initialization time, slot 0 of which holds
>> >> PSR_CBM_TYPE_L3_DATA and slot 1 PSR_CBM_TYPE_L3_CODE (or
>> >> the other way around). Granted I was wrong with the type of the
>> >> array (as the above aren't enum psr_feat_type enumerators, but
>> >> enum cbm_type ones), but I think the basic idea should have been
>> >> clear anyway: You need to provide a way for generic code to pass
>> >> suitable type information into ->get_val().
>> >> 
>> > May I change the 'get_val()' parameter 'enum cbm_type' to a generic type
>> > 'unsigned int' to make it be a flexible type,  and then combine feature
>> > type with cos_num together as a flag to indicate which feature it is,
>> > which value to get and distinguish it with cbm_type? For example:
>> > 
>> > #define CDP_GATHER_BOTH_DATA ( PSR_SOCKET_L3_CDP << 16 )
>> > #define CDP_GATHER_BOTH_CODE ( PSR_SOCKET_L3_CDP << 16 + 1 )
>> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
>> >                            unsigned int type, uint32_t *val)
>> > {
>> >     switch ( type )
>> >     {
>> >     case PSR_CBM_TYPE_L3_DATA:
>> >     case CDP_GATHER_BOTH_DATA:
>> >         *val = get_cdp_data(feat, cos);
>> >         break;
>> >     case PSR_CBM_TYPE_L3_CODE:
>> >     case CDP_GATHER_BOTH_CODE:
>> >         *val = get_cdp_code(feat, cos);
>> >         break;
>> >     }
>> > }
>> 
>> The two case labels are still indicative of unnecessary redundancy
>> (and, even right now only highly theoretical, risk of collisions). What's
>> wrong with the model I've proposed?
>> 
> Oh, sorry. I did not understand your proposal just now so I provided another
> solution.
> 
> After reading your suggestion again, I think your meaing is below in codes:
> 
> struct feat_props {
> ...
>     unsigned int cos_num;
>     enum cbm_type cos_to_type[2];
> ...
> }
> 
> static void cat_init_feature(...)
> {
> ...
>     case PSR_SOCKET_L3_CDP:
>         feat->props->cos_to_type[0] = PSR_CBM_TYPE_L3_DATA;
>         feat->props->cos_to_type[1] = PSR_CBM_TYPE_L3_CODE;
> ...
> }
> 
> Then, in functions to iterate 'cos_num', we can input 'cos_to_type[i]'. 

Yes (albeit I'd call the array field just "type").

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

* Re: [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow.
  2017-03-31 14:35                                         ` Jan Beulich
@ 2017-03-31 14:46                                           ` Yi Sun
  0 siblings, 0 replies; 105+ messages in thread
From: Yi Sun @ 2017-03-31 14:46 UTC (permalink / raw)
  To: Jan Beulich
  Cc: kevin.tian, wei.liu2, he.chen, andrew.cooper3, dario.faggioli,
	ian.jackson, mengxu, xen-devel, chao.p.peng, roger.pau

On 17-03-31 08:35:14, Jan Beulich wrote:
> >>> On 31.03.17 at 15:22, <yi.y.sun@linux.intel.com> wrote:
> > On 17-03-31 06:51:07, Jan Beulich wrote:
> >> >>> On 31.03.17 at 14:40, <yi.y.sun@linux.intel.com> wrote:
> >> > On 17-03-31 04:19:49, Jan Beulich wrote:
> >> >> >>> On 31.03.17 at 11:12, <yi.y.sun@linux.intel.com> wrote:
> >> >> > On 17-03-31 02:47:25, Jan Beulich wrote:
> >> >> >> >>> On 30.03.17 at 14:10, <yi.y.sun@linux.intel.com> wrote:
> >> >> >> > I think you can check v9 codes here:
> >> >> >> > https://github.com/yisun-git/xen/tree/l2_cat_v9 
> >> >> >> 
> >> >> >> Looking at this made me notice that cat_get_old_val() passes a
> >> >> >> bogus literal 0 to cat_get_val(), which needs taking care of too.
> >> >> >> One option I can see is for each feature to make available an
> >> >> >> array of type enum cbm_type, with cos_num elements. The order
> >> >> >> would match that of the order of values in their arrays. This will
> >> >> > 
> >> >> > Sorry, not very clear your meaning. How to do that? Could you please
> >> >> > provide pieces of codes? Thanks!
> >> >> 
> >> >> I'm sorry, but I'm afraid I don't see how I would reasonably supply
> >> >> code here without taking over your series altogether (which I don't
> >> >> intend to do). What is unclear with, at the example of CDP, you
> >> >> needing to add an array at initialization time, slot 0 of which holds
> >> >> PSR_CBM_TYPE_L3_DATA and slot 1 PSR_CBM_TYPE_L3_CODE (or
> >> >> the other way around). Granted I was wrong with the type of the
> >> >> array (as the above aren't enum psr_feat_type enumerators, but
> >> >> enum cbm_type ones), but I think the basic idea should have been
> >> >> clear anyway: You need to provide a way for generic code to pass
> >> >> suitable type information into ->get_val().
> >> >> 
> >> > May I change the 'get_val()' parameter 'enum cbm_type' to a generic type
> >> > 'unsigned int' to make it be a flexible type,  and then combine feature
> >> > type with cos_num together as a flag to indicate which feature it is,
> >> > which value to get and distinguish it with cbm_type? For example:
> >> > 
> >> > #define CDP_GATHER_BOTH_DATA ( PSR_SOCKET_L3_CDP << 16 )
> >> > #define CDP_GATHER_BOTH_CODE ( PSR_SOCKET_L3_CDP << 16 + 1 )
> >> > static void l3_cdp_get_val(const struct feat_node *feat, unsigned int cos,
> >> >                            unsigned int type, uint32_t *val)
> >> > {
> >> >     switch ( type )
> >> >     {
> >> >     case PSR_CBM_TYPE_L3_DATA:
> >> >     case CDP_GATHER_BOTH_DATA:
> >> >         *val = get_cdp_data(feat, cos);
> >> >         break;
> >> >     case PSR_CBM_TYPE_L3_CODE:
> >> >     case CDP_GATHER_BOTH_CODE:
> >> >         *val = get_cdp_code(feat, cos);
> >> >         break;
> >> >     }
> >> > }
> >> 
> >> The two case labels are still indicative of unnecessary redundancy
> >> (and, even right now only highly theoretical, risk of collisions). What's
> >> wrong with the model I've proposed?
> >> 
> > Oh, sorry. I did not understand your proposal just now so I provided another
> > solution.
> > 
> > After reading your suggestion again, I think your meaing is below in codes:
> > 
> > struct feat_props {
> > ...
> >     unsigned int cos_num;
> >     enum cbm_type cos_to_type[2];
> > ...
> > }
> > 
> > static void cat_init_feature(...)
> > {
> > ...
> >     case PSR_SOCKET_L3_CDP:
> >         feat->props->cos_to_type[0] = PSR_CBM_TYPE_L3_DATA;
> >         feat->props->cos_to_type[1] = PSR_CBM_TYPE_L3_CODE;
> > ...
> > }
> > 
> > Then, in functions to iterate 'cos_num', we can input 'cos_to_type[i]'. 
> 
> Yes (albeit I'd call the array field just "type").
> 
Got it. This should be the last issue not locked down. I will send out
v10 in soon. Thank you very much for the review!

BRs,
Sun Yi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 105+ messages in thread

end of thread, other threads:[~2017-03-31 14:46 UTC | newest]

Thread overview: 105+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-16 11:07 [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Yi Sun
2017-03-16 11:07 ` [PATCH v9 01/25] docs: create Cache Allocation Technology (CAT) and Code and Data Prioritization (CDP) feature document Yi Sun
2017-03-16 11:07 ` [PATCH v9 02/25] x86: refactor psr: remove L3 CAT/CDP codes Yi Sun
2017-03-16 11:07 ` [PATCH v9 03/25] x86: refactor psr: implement main data structures Yi Sun
2017-03-24 16:19   ` Jan Beulich
2017-03-27  2:38     ` Yi Sun
2017-03-27  6:20       ` Jan Beulich
2017-03-27  7:12         ` Yi Sun
2017-03-27  7:37           ` Jan Beulich
2017-03-16 11:07 ` [PATCH v9 04/25] x86: move cpuid_count_leaf from cpuid.c to processor.h Yi Sun
2017-03-24 16:22   ` Jan Beulich
2017-03-16 11:07 ` [PATCH v9 05/25] x86: refactor psr: L3 CAT: implement CPU init and free flow Yi Sun
2017-03-24 16:52   ` Jan Beulich
2017-03-27  4:41     ` Yi Sun
2017-03-27  6:34       ` Jan Beulich
2017-03-27  8:16         ` Yi Sun
2017-03-27  8:43           ` Jan Beulich
2017-03-16 11:07 ` [PATCH v9 06/25] x86: refactor psr: L3 CAT: implement Domain init/free and schedule flows Yi Sun
2017-03-16 11:07 ` [PATCH v9 07/25] x86: refactor psr: L3 CAT: implement get hw info flow Yi Sun
2017-03-27  9:07   ` Jan Beulich
2017-03-27 12:24     ` Yi Sun
2017-03-27 12:51       ` Jan Beulich
2017-03-27 13:19         ` Yi Sun
2017-03-27 13:32           ` Jan Beulich
2017-03-16 11:07 ` [PATCH v9 08/25] x86: refactor psr: L3 CAT: implement get value flow Yi Sun
2017-03-27  9:23   ` Jan Beulich
2017-03-27 12:59     ` Yi Sun
2017-03-27 13:34       ` Jan Beulich
2017-03-28  2:13         ` Yi Sun
2017-03-28  8:10           ` Jan Beulich
2017-03-16 11:07 ` [PATCH v9 09/25] x86: refactor psr: L3 CAT: set value: implement framework Yi Sun
2017-03-27  9:59   ` Jan Beulich
2017-03-28  1:21     ` Yi Sun
2017-03-28  8:21       ` Jan Beulich
2017-03-16 11:08 ` [PATCH v9 10/25] x86: refactor psr: L3 CAT: set value: assemble features value array Yi Sun
2017-03-27 10:17   ` Jan Beulich
2017-03-28  3:12     ` Yi Sun
2017-03-28  8:05       ` Yi Sun
2017-03-28  8:36         ` Jan Beulich
2017-03-28  9:11           ` Yi Sun
2017-03-28  9:20             ` Jan Beulich
2017-03-28 10:18               ` Yi Sun
2017-03-28 10:39                 ` Jan Beulich
2017-03-28  8:34       ` Jan Beulich
2017-03-28 10:12         ` Yi Sun
2017-03-28 10:36           ` Jan Beulich
2017-03-16 11:08 ` [PATCH v9 11/25] x86: refactor psr: L3 CAT: set value: implement cos finding flow Yi Sun
2017-03-27 10:28   ` Jan Beulich
2017-03-28  3:26     ` Yi Sun
2017-03-28  8:41       ` Jan Beulich
2017-03-16 11:08 ` [PATCH v9 12/25] x86: refactor psr: L3 CAT: set value: implement cos id picking flow Yi Sun
2017-03-27 10:37   ` Jan Beulich
2017-03-28  4:58     ` Yi Sun
2017-03-28  8:45       ` Jan Beulich
2017-03-28 10:31         ` Yi Sun
2017-03-28 10:40           ` Jan Beulich
2017-03-28 11:59             ` Yi Sun
2017-03-28 12:20               ` Jan Beulich
2017-03-29  1:20                 ` Yi Sun
2017-03-29  1:36                   ` Yi Sun
2017-03-29  9:57                     ` Jan Beulich
2017-03-30  1:37                       ` Yi Sun
2017-03-30  1:39                         ` Yi Sun
2017-03-30 11:55                         ` Jan Beulich
2017-03-30 12:10                           ` Yi Sun
2017-03-31  8:47                             ` Jan Beulich
2017-03-31  9:12                               ` Yi Sun
2017-03-31  9:18                                 ` Yi Sun
2017-03-31 10:19                                 ` Jan Beulich
2017-03-31 12:40                                   ` Yi Sun
2017-03-31 12:51                                     ` Jan Beulich
2017-03-31 13:22                                       ` Yi Sun
2017-03-31 14:35                                         ` Jan Beulich
2017-03-31 14:46                                           ` Yi Sun
2017-03-16 11:08 ` [PATCH v9 13/25] x86: refactor psr: L3 CAT: set value: implement write msr flow Yi Sun
2017-03-27 10:46   ` Jan Beulich
2017-03-28  5:06     ` Yi Sun
2017-03-28  8:48       ` Jan Beulich
2017-03-28 10:20         ` Yi Sun
2017-03-16 11:08 ` [PATCH v9 14/25] x86: refactor psr: CDP: implement CPU init and free flow Yi Sun
2017-03-27 13:58   ` Jan Beulich
2017-03-16 11:08 ` [PATCH v9 15/25] x86: refactor psr: CDP: implement get hw info flow Yi Sun
2017-03-27 14:08   ` Jan Beulich
2017-03-28  5:13     ` Yi Sun
2017-03-16 11:08 ` [PATCH v9 16/25] x86: refactor psr: CDP: implement get value flow Yi Sun
2017-03-16 11:08 ` [PATCH v9 17/25] x86: refactor psr: CDP: implement set value callback functions Yi Sun
2017-03-27 14:17   ` Jan Beulich
2017-03-28  5:14     ` Yi Sun
2017-03-16 11:08 ` [PATCH v9 18/25] x86: L2 CAT: implement CPU init and free flow Yi Sun
2017-03-16 11:08 ` [PATCH v9 19/25] x86: L2 CAT: implement get hw info flow Yi Sun
2017-03-27 14:38   ` Jan Beulich
2017-03-28  5:16     ` Yi Sun
2017-03-16 11:08 ` [PATCH v9 20/25] x86: L2 CAT: implement get value flow Yi Sun
2017-03-27 14:39   ` Jan Beulich
2017-03-16 11:08 ` [PATCH v9 21/25] x86: L2 CAT: implement set " Yi Sun
2017-03-27 14:40   ` Jan Beulich
2017-03-16 11:08 ` [PATCH v9 22/25] tools: L2 CAT: support get HW info for L2 CAT Yi Sun
2017-03-16 11:08 ` [PATCH v9 23/25] tools: L2 CAT: support show cbm " Yi Sun
2017-03-16 11:08 ` [PATCH v9 24/25] tools: L2 CAT: support set " Yi Sun
2017-03-28 14:04   ` Wei Liu
2017-03-29  1:21     ` Yi Sun
2017-03-16 11:08 ` [PATCH v9 25/25] docs: add L2 CAT description in docs Yi Sun
2017-03-16 11:20 ` [PATCH v9 00/25] Enable L2 Cache Allocation Technology & Refactor psr.c Jan Beulich
2017-03-17  1:29   ` Yi Sun
2017-03-17  7:25     ` Jan Beulich

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.