* [v2 0/3] Enable L2 Cache Allocation Technology
@ 2016-09-22 2:14 Yi Sun
2016-09-22 10:18 ` Wei Liu
2016-09-30 20:32 ` Konrad Rzeszutek Wilk
0 siblings, 2 replies; 7+ messages in thread
From: Yi Sun @ 2016-09-22 2:14 UTC (permalink / raw)
To: xen-devel
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, Yi Sun, jbeulich,
chao.p.peng
Design document is below:
=======================================================================
% Intel L2 Cache Allocation Technology (L2 CAT) Feature
% Revision 2.0
\clearpage
Hi all,
We plan to bring a new PSR (Platform Shared Resource) feature called
Intel L2 Cache Allocation Technology (L2 CAT) to Xen.
This is the design of L2 CAT. It might be a little long and detailed, hope
it doesn't matter.
Besides the L2 CAT implementaion, we refactor the psr.c to make it more
flexible to add new features and fulfill the principle, open for extension
but closed for modification.
Comments and suggestions are welcome :-)
# Basics
---------------- ----------------------------------------------------
Status: **Tech Preview**
Architecture(s): Intel x86
Component(s): Hypervisor, toolstack
Hardware: Atom codename Goldmont and beyond
---------------- ----------------------------------------------------
# Overview
L2 CAT allows an OS or Hypervisor/VMM to control allocation of a
CPU's shared L2 cache based on application priority or Class of Service
(COS). Each CLOS is configured using capacity bitmasks (CBM) which
represent cache capacity and indicate the degree of overlap and
isolation between classes. Once L2 CAT is configured, the processor
allows access to portions of L2 cache according to the established
class of service (COS).
# Technical information
L2 CAT is a member of Intel PSR features and part of CAT, it shares
some base PSR infrastructure in Xen.
## Hardware perspective
L2 CAT defines a new range MSRs to assign different L2 cache access
patterns which are known as CBMs (Capacity BitMask), each CBM is
associated with a COS.
```
+----------------------------+----------------+
IA32_PQR_ASSOC | MSR (per socket) | Address |
+----+---+-------+ +----------------------------+----------------+
| |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 |
+----+---+-------+ +----------------------------+----------------+
└-------------> | ... | ... |
+----------------------------+----------------+
| IA32_L2_QOS_MASK_n | 0xD10+n (n<64) |
+----------------------------+----------------+
```
When context switch happens, the COS of VCPU is written to per-thread
MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation
according to the corresponding CBM.
## The relationship between L2 CAT and L3 CAT/CDP
L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled
while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled.
L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by
the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing
patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of
L2 CAT must be continuous too.
N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same
associate register `IA32_PQR_ASSOC`, which means one COS associate to a
pair of L2 CBM and L3 CBM.
Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or
other PSR features in future). In some cases, a VM is permitted to have a
COS that is beyond one (or more) of PSR features but within the others.
For instance, let's assume the max COS of L2 CAT is 8 but the max COS of
L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to
COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no
limit) since COS 9 is beyond the max COS (8) of L2 CAT.
## Design Overview
* Core COS/CBM association
When enforcing L2 CAT, all cores of domains have the same default
COS (COS0) which associated to the fully open CBM (all ones bitmask)
to access all L2 cache. The default COS is used only in hypervisor
and is transparent to tool stack and user.
System administrator can change PSR allocation policy at runtime by
tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds
to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP
is enabled, one COS corresponds to a 3-tuple, like [L2 CBM,
L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled,
things would be easier, one COS corresponds to one L2 CBM.
* VCPU schedule
This part reuses L3 CAT COS infrastructure.
* Multi-sockets
Different sockets may have different L2 CAT capability (e.g. max COS)
although it is consistent on the same socket. So the capability of
per-socket L2 CAT is specified.
## Implementation Description
* Hypervisor interfaces:
1. Ext: Boot line parameter "psr=cat" now will enable L2 CAT and L3
CAT if hardware supported.
2. New: SYSCTL:
- XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information.
3. New: DOMCTL:
- XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain.
- XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain.
* xl interfaces:
1. Ext: psr-cat-show -l2 domain-id
Show L2 cbm for a domain.
=> XEN_SYSCTL_PSR_CAT_get_l2_info /
XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM
2. Ext: psr-mba-set -l2 domain-id cbm
Set L2 cbm for a domain.
=> XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM
3. Ext: psr-hwinfo
Show PSR HW information, including L2 CAT
=> XEN_SYSCTL_PSR_CAT_get_l2_info
* Key data structure:
1. Feature info
```
struct feat_info {
union {
struct psr_cat_lvl_info l3_info;
struct psr_cat_lvl_info l2_info;
};
};
```
- Member `l3_info`
`l3_info` is the hardware info of L3 CAT.
- Member `l2_info`
`l2_info` is the hardware info of L2 CAT.
2. Feature list
```
struct feat_list {
unsigned int feature;
struct feat_ops ops;
unsigned feat_info info;
uint64_t cos_reg_val[MAX_COS_REG_NUM];
struct list_head list;
};
```
When a PSR enforcement feature is enabled, it will be added into a
feature list. The head of the list is created in psr initialization.
- Member `feature`
`feature` is an integer number, to indicate which feature the list member
corresponds to.
- Member `ops`
`ops` maintains a callback function list of the feature. It will be introduced
in details later.
- Member `info`
`info` maintains the feature HW information which can be got through
psr_hwinfo command.
- Member `cos_reg_val`
`cos_ref_val` is an array to maintain the value set in all COS registers of
the feature.
3. Per-socket PSR features information structure
```
struct psr_socket_info {
unsigned int features;
unsigned int nr_feat;
struct list_head feat_list;
struct psr_ref *cos_ref;
spinlock_t mask_lock;
};
```
We collect all PSR allocation features information of a socket in
this `struct psr_socket_alloc_info`.
- Member `features`
`features` is a bitmap, to indicate which feature is enabled on
current socket. We define `features` bitmap as:
bit 0~1: L3 CAT status, [01] stands for L3 CAT only and [10]
stands for L3 CDP is enalbed.
bit 2: L2 CAT status.
- Member `cos_ref`
`cos_ref` is an array which maintains the reference of one COS.
If the COS is used by one domain, the reference will increase one.
If a domain releases the COS, the reference will decrease one. The
array is indexed by COS.
4. Feature operation functions structure
```
struct feat_ops {
void (*init_feature)(unsigned int eax, unsigned int ebx,
unsigned int ecx, unsigned int edx,
struct feat_list *feat,
struct psr_socket_info *info);
unsigned int (*get_cos_num)(const struct feat_list *feat);
int (*get_old_val)(uint64_t val[],
const struct feat_list *feat,
unsigned int old_cos,
enum psr_val_type type);
int (*set_new_val)(uint64_t val[],
const struct feat_list *feat,
unsigned int old_cos,
enum psr_val_type type,
uint64_t m);
int (*compare_val)(const uint64_t val[], const struct feat_list *feat,
unsigned int cos, bool *found);
unsigned int (*get_cos_max_from_type)(const struct feat_list *feat,
enum psr_val_type type);
unsigned int (*exceeds_cos_max)(const uint64_t val[],
const struct feat_list *feat,
unsigned int cos);
int (*write_msr)(unsigned int cos, const uint64_t val[],
struct feat_list *feat);
int (*get_val)(const struct feat_list *feat, unsigned int cos,
enum psr_val_type type, uint64_t *val);
int (*get_val)(const struct feat_list *feat, unsigned int cos,
enum psr_val_type type, uint64_t *val);
unsigned int (*get_max_cos_max)(const struct feat_list *feat);
};
```
We abstract above callback functions to encapsulate the feature specific
behaviors into them. Then, it is easy to add a new feature. We just need:
1) Implement such ops and callback functions for every feature.
2) Register the ops into `struct feat_list`.
3) Add the feature into feature list during CPU initialization.
# User information
* Feature Enabling:
Add "psr=cat" to boot line parameter to enable all supported level CAT
features.
* xl interfaces:
1. `psr-cat-show [OPTIONS] domain-id`:
Show domain L2 or L3 CAT CBM.
New option `-l` is added.
`-l2`: Specify cbm for L2 cache.
`-l3`: Specify cbm for L3 cache.
If neither `-l2` nor `-l3` is given, a prompt message will be shown
to ask user input cache level.
2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`:
Set domain L2 or L3 CBM.
New option `-l` is added.
`-l2`: Specify cbm for L2 cache.
`-l3`: Specify cbm for L3 cache.
If neither `-l2` nor `-l3` is given, a prompt message will be shown
to ask user input cache level. Since there is no CDP support on L2 cache,
so if `-l2`, `--code` or `--data` are both given, error message shows.
3. `psr-hwinfo [OPTIONS]`:
Show L2 & L3 CAT HW informations on every socket.
# References
[Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3, chapter 17.17](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
# History
------------------------------------------------------------------------
Date Revision Version Notes
---------- -------- -------- -------------------------------------------
2016-08-12 1.0 Xen 4.7 Initial design
2016-09-21 2.0 Xen 4.7 Changes according to review comments.
1. L2 CBM bits should be continuous too.
2. Describe 'struct feat_info'
3. Update 'struct feat_list" description.
4. Update 'struct feat_ops" description.
5. Update `psr-cat-show` description.
6. Update `psr-cat-cbm-set` description.
7. Add volume and chapter info in References.
---------- -------- -------- -------------------------------------------
Yi Sun (3):
x86: refactor psr implementation in hypervisor.
x86: add support for L2 CAT in hypervisor.
tools & docs: add L2 CAT support in tools and docs.
docs/man/xl.pod.1.in | 25 +-
docs/misc/xl-psr.markdown | 10 +-
tools/libxc/include/xenctrl.h | 7 +-
tools/libxc/xc_psr.c | 47 +-
tools/libxl/libxl.h | 2 +
tools/libxl/libxl_psr.c | 50 +-
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 195 +++++-
tools/libxl/xl_cmdtable.c | 4 +-
xen/arch/x86/domctl.c | 49 +-
xen/arch/x86/psr.c | 1317 ++++++++++++++++++++++++++++++++-------
xen/arch/x86/sysctl.c | 25 +-
xen/include/asm-x86/msr-index.h | 1 +
xen/include/asm-x86/psr.h | 22 +-
xen/include/public/domctl.h | 2 +
xen/include/public/sysctl.h | 6 +
16 files changed, 1440 insertions(+), 323 deletions(-)
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [v2 0/3] Enable L2 Cache Allocation Technology
2016-09-22 2:14 [v2 0/3] Enable L2 Cache Allocation Technology Yi Sun
@ 2016-09-22 10:18 ` Wei Liu
2016-09-23 2:06 ` Yi Sun
2016-09-30 20:32 ` Konrad Rzeszutek Wilk
1 sibling, 1 reply; 7+ messages in thread
From: Wei Liu @ 2016-09-22 10:18 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
chao.p.peng, xen-devel
Hi Yi
Thanks for submitting this series. I see that all the actual patches
are not chained to 0/3. It would be better if they show up as replies to
0/3.
The workflow for sending patch series differs from person to person, so
we don't want to dictate how you use your tool. But we have written a
wiki page on how to submit patches to xen-devel. That contains some tips
about git.
https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches#Sending_the_patches_to_the_list
Feel free to ask questions here. I would be happy to share how I do it
if you're interested.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [v2 0/3] Enable L2 Cache Allocation Technology
2016-09-22 10:18 ` Wei Liu
@ 2016-09-23 2:06 ` Yi Sun
2016-09-23 9:41 ` Wei Liu
0 siblings, 1 reply; 7+ messages in thread
From: Yi Sun @ 2016-09-23 2:06 UTC (permalink / raw)
To: Wei Liu
Cc: he.chen, andrew.cooper3, ian.jackson, jbeulich, chao.p.peng, xen-devel
On 16-09-22 11:18:12, Wei Liu wrote:
> Hi Yi
>
> Thanks for submitting this series. I see that all the actual patches
> are not chained to 0/3. It would be better if they show up as replies to
> 0/3.
>
> The workflow for sending patch series differs from person to person, so
> we don't want to dictate how you use your tool. But we have written a
> wiki page on how to submit patches to xen-devel. That contains some tips
> about git.
>
> https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches#Sending_the_patches_to_the_list
>
> Feel free to ask questions here. I would be happy to share how I do it
> if you're interested.
>
> Wei.
Hi, Wei,
Thanks a lot for your kindness! I just went through the wiki and found I
should use '--in-reply-to=', right? If you can share your steps that would
be great.
Can I send the patches again after addressing Ian's comments?
Thanks,
Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [v2 0/3] Enable L2 Cache Allocation Technology
2016-09-23 2:06 ` Yi Sun
@ 2016-09-23 9:41 ` Wei Liu
2016-09-23 10:03 ` Yi Sun
0 siblings, 1 reply; 7+ messages in thread
From: Wei Liu @ 2016-09-23 9:41 UTC (permalink / raw)
To: Yi Sun
Cc: Wei Liu, he.chen, andrew.cooper3, ian.jackson, jbeulich,
chao.p.peng, xen-devel
On Fri, Sep 23, 2016 at 10:06:54AM +0800, Yi Sun wrote:
> On 16-09-22 11:18:12, Wei Liu wrote:
> > Hi Yi
> >
> > Thanks for submitting this series. I see that all the actual patches
> > are not chained to 0/3. It would be better if they show up as replies to
> > 0/3.
> >
> > The workflow for sending patch series differs from person to person, so
> > we don't want to dictate how you use your tool. But we have written a
> > wiki page on how to submit patches to xen-devel. That contains some tips
> > about git.
> >
> > https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches#Sending_the_patches_to_the_list
> >
> > Feel free to ask questions here. I would be happy to share how I do it
> > if you're interested.
> >
> > Wei.
>
> Hi, Wei,
>
> Thanks a lot for your kindness! I just went through the wiki and found I
> should use '--in-reply-to=', right? If you can share your steps that would
> be great.
Assuming you send out your cover letter separately from your patches,
yes, --in-reply-to= is the option that you need to add to `git
send-email' invocation. The <identifier> (see `git help send-email')
part is the message id of your cover letter. You should be able to get
that after sending out the cover letter.
My current preferred workflow is to have git handle everything for me.
I have a separate file for the *content* of cover letter. And to generate
everything including the cover letter *email*:
$ git format-patch <revision range> --reroll-count=$N --cover-letter
Then you will have v$N-0000-cover-letter.patch plus a few v$N-*.patch.
Use you favourite text editor to edit the cover letter, import the
content, edit it as you wish.
To send out all you patches including the cover letter:
$ git send-email *.patch --to xen-devel@... --cc maintainers@... --dry-run
When you're confident the list of emails looks correct, delete --dry-run
to actually send them.
This way git send-email will automatically chain all actual patches to
cover letter.
Some people have built other tools on top of porcelain git to track more
stuff. I'm experimenting with a tool called git-series, but it is too
early to say if it is the right tool for me.
Anyway, read git manpage. It has a lot of information in it.
>
> Can I send the patches again after addressing Ian's comments?
>
It's better to wait a bit for hypervisor maintainers to comment on the
first three patches.
Wei.
> Thanks,
> Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [v2 0/3] Enable L2 Cache Allocation Technology
2016-09-23 9:41 ` Wei Liu
@ 2016-09-23 10:03 ` Yi Sun
0 siblings, 0 replies; 7+ messages in thread
From: Yi Sun @ 2016-09-23 10:03 UTC (permalink / raw)
To: Wei Liu
Cc: he.chen, andrew.cooper3, ian.jackson, jbeulich, chao.p.peng, xen-devel
On 16-09-23 10:41:45, Wei Liu wrote:
> On Fri, Sep 23, 2016 at 10:06:54AM +0800, Yi Sun wrote:
> > On 16-09-22 11:18:12, Wei Liu wrote:
> > > Hi Yi
> > >
> > > Thanks for submitting this series. I see that all the actual patches
> > > are not chained to 0/3. It would be better if they show up as replies to
> > > 0/3.
> > >
> > > The workflow for sending patch series differs from person to person, so
> > > we don't want to dictate how you use your tool. But we have written a
> > > wiki page on how to submit patches to xen-devel. That contains some tips
> > > about git.
> > >
> > > https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches#Sending_the_patches_to_the_list
> > >
> > > Feel free to ask questions here. I would be happy to share how I do it
> > > if you're interested.
> > >
> > > Wei.
> >
> > Hi, Wei,
> >
> > Thanks a lot for your kindness! I just went through the wiki and found I
> > should use '--in-reply-to=', right? If you can share your steps that would
> > be great.
>
> Assuming you send out your cover letter separately from your patches,
> yes, --in-reply-to= is the option that you need to add to `git
> send-email' invocation. The <identifier> (see `git help send-email')
> part is the message id of your cover letter. You should be able to get
> that after sending out the cover letter.
>
> My current preferred workflow is to have git handle everything for me.
> I have a separate file for the *content* of cover letter. And to generate
> everything including the cover letter *email*:
>
> $ git format-patch <revision range> --reroll-count=$N --cover-letter
>
> Then you will have v$N-0000-cover-letter.patch plus a few v$N-*.patch.
> Use you favourite text editor to edit the cover letter, import the
> content, edit it as you wish.
>
> To send out all you patches including the cover letter:
>
> $ git send-email *.patch --to xen-devel@... --cc maintainers@... --dry-run
>
> When you're confident the list of emails looks correct, delete --dry-run
> to actually send them.
>
> This way git send-email will automatically chain all actual patches to
> cover letter.
>
> Some people have built other tools on top of porcelain git to track more
> stuff. I'm experimenting with a tool called git-series, but it is too
> early to say if it is the right tool for me.
>
> Anyway, read git manpage. It has a lot of information in it.
>
Many thanks for the details! They are very helpful. I will read manual and
test the flow.
> >
> > Can I send the patches again after addressing Ian's comments?
> >
>
> It's better to wait a bit for hypervisor maintainers to comment on the
> first three patches.
>
Sure. Thank you!
> Wei.
>
> > Thanks,
> > Yi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [v2 0/3] Enable L2 Cache Allocation Technology
2016-09-22 2:14 [v2 0/3] Enable L2 Cache Allocation Technology Yi Sun
2016-09-22 10:18 ` Wei Liu
@ 2016-09-30 20:32 ` Konrad Rzeszutek Wilk
2016-10-09 1:52 ` Yi Sun
1 sibling, 1 reply; 7+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-09-30 20:32 UTC (permalink / raw)
To: Yi Sun
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
chao.p.peng, xen-devel
On Thu, Sep 22, 2016 at 10:14:00AM +0800, Yi Sun wrote:
> Design document is below:
Could the design document be a patch itself please?
Meaning it will be added in docs/misc/ ?
Also is there a git tree for your patches?
Thanks.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [v2 0/3] Enable L2 Cache Allocation Technology
2016-09-30 20:32 ` Konrad Rzeszutek Wilk
@ 2016-10-09 1:52 ` Yi Sun
0 siblings, 0 replies; 7+ messages in thread
From: Yi Sun @ 2016-10-09 1:52 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: wei.liu2, he.chen, andrew.cooper3, ian.jackson, jbeulich,
chao.p.peng, xen-devel
Hi,
Many thanks for reviewing the patches and provide your comments!
Please check my comments below. Thanks!
On 16-09-30 16:32:33, Konrad Rzeszutek Wilk wrote:
> On Thu, Sep 22, 2016 at 10:14:00AM +0800, Yi Sun wrote:
> > Design document is below:
>
> Could the design document be a patch itself please?
>
Yes, I will do it. Thanks!
> Meaning it will be added in docs/misc/ ?
>
> Also is there a git tree for your patches?
>
No. Because the number of patches is not much, I do not create git tree.
If you strongly recommend it, I can create it. Thanks!
> Thanks.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-10-09 1:58 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-09-22 2:14 [v2 0/3] Enable L2 Cache Allocation Technology Yi Sun
2016-09-22 10:18 ` Wei Liu
2016-09-23 2:06 ` Yi Sun
2016-09-23 9:41 ` Wei Liu
2016-09-23 10:03 ` Yi Sun
2016-09-30 20:32 ` Konrad Rzeszutek Wilk
2016-10-09 1:52 ` Yi Sun
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.