* [PATCH 0/3] x86/intel_rdt: Fix exclusive mode with CDP resources @ 2018-09-26 18:59 Reinette Chatre 2018-09-26 18:59 ` [PATCH 1/3] x86/intel_rdt: Introduce utility to obtain CDP peer Reinette Chatre ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Reinette Chatre @ 2018-09-26 18:59 UTC (permalink / raw) To: tglx, fenghua.yu, tony.luck Cc: jithu.joseph, gavin.hindman, dave.hansen, mingo, hpa, x86, linux-kernel, Reinette Chatre Dear Maintainers, CDP resources do not currently behave as expected when there are resource groups with mode 'exclusive'. In the example below it was possible to create two resource groups, p1 and p2, that are both in exclusive mode but their usage of the underlying L2 cache actually overlaps. root@glk:/sys/fs/resctrl# ls cpus cpus_list info mode p1 p2 schemata size tasks root@glk:/sys/fs/resctrl# cat schemata L2DATA:0=fff0 L2CODE:0=fff0 root@glk:/sys/fs/resctrl# cat mode shareable root@glk:/sys/fs/resctrl# cat p1/schemata L2DATA:0=0003 L2CODE:0=000c root@glk:/sys/fs/resctrl# cat p1/mode exclusive root@glk:/sys/fs/resctrl# cat p2/schemata L2DATA:0=000c L2CODE:0=0003 root@glk:/sys/fs/resctrl# cat p2/mode exclusive root@glk:/sys/fs/resctrl# cat info/L2CODE/bit_usage 0=SSSSSSSSSSSSEEEE root@glk:/sys/fs/resctrl# cat info/L2DATA/bit_usage 0=SSSSSSSSSSSSEEEE root@glk:/sys/fs/resctrl# In the above example, the CBM of L2DATA in p1 overlaps with the CBM of L2CODE in p2 while they are both in exclusive mode. While it may reflect no overlap among the L2DATA resources specifically it does actually imply overlap of use of the underlying hardware that is not the intention of 'exclusive' mode. This happens because the current implementation treats L2CODE and L2DATA as totally independent, when it is actually referring to the same underlying hardware. This series fixes the potential for overlap of hardware resource use when resource groups are in 'exclusive' mode by ensuring that if there is a CDP peer on the same hardware then any overlap test would consider it also. Allocations of data and code resources within the same exclusive resource group are allowed to overlap. Your feedback will be greatly appreciated. Reinette Reinette Chatre (3): x86/intel_rdt: Introduce utility to obtain CDP peer x86/intel_rdt: CBM overlap should also check for overlap with CDP peer x86/intel_rdt: Fix initial allocation to consider CDP arch/x86/kernel/cpu/intel_rdt_rdtgroup.c | 133 ++++++++++++++++++++++- 1 file changed, 128 insertions(+), 5 deletions(-) -- 2.17.0 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/3] x86/intel_rdt: Introduce utility to obtain CDP peer 2018-09-26 18:59 [PATCH 0/3] x86/intel_rdt: Fix exclusive mode with CDP resources Reinette Chatre @ 2018-09-26 18:59 ` Reinette Chatre 2018-10-03 6:46 ` Thomas Gleixner 2018-09-26 18:59 ` [PATCH 2/3] x86/intel_rdt: CBM overlap should also check for overlap with " Reinette Chatre 2018-09-26 18:59 ` [PATCH 3/3] x86/intel_rdt: Fix initial allocation to consider CDP Reinette Chatre 2 siblings, 1 reply; 9+ messages in thread From: Reinette Chatre @ 2018-09-26 18:59 UTC (permalink / raw) To: tglx, fenghua.yu, tony.luck Cc: jithu.joseph, gavin.hindman, dave.hansen, mingo, hpa, x86, linux-kernel, Reinette Chatre Introduce a utility that, when provided with a RDT resource and an instance of this RDT resource (a RDT domain), would return pointers to the RDT resource and RDT domain that share the same hardware. This is specific to the CDP resources that share the same hardware. For example, if a pointer to the RDT_RESOURCE_L2DATA resource (struct rdt_resource) and a pointer to an instance of this resource (struct rdt_domain) is provided, then it will return a pointer to the RDT_RESOURCE_L2CODE resource as well as the specific instance that shares the same hardware as the provided rdt_domain. This utility is created in support of the "exclusive" resource group mode where overlap of resource allocation between resource groups need to be avoided. The overlap test need to consider not just the matching resources, but also the resources that share the same hardware. Temporarily mark it as unused in support of patch testing to avoid compile warnings until it is used. Fixes: 49f7b4efa110 ("x86/intel_rdt: Enable setting of exclusive mode") Signed-off-by: Reinette Chatre <reinette.chatre@intel.com> Tested-by: Jithu Joseph <jithu.joseph@intel.com> Acked-by: Fenghua Yu <fenghua.yu@intel.com> --- arch/x86/kernel/cpu/intel_rdt_rdtgroup.c | 72 ++++++++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c index 1b8e86a5d5e1..c8c02c16d072 100644 --- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c +++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c @@ -960,6 +960,78 @@ static int rdtgroup_mode_show(struct kernfs_open_file *of, return 0; } +/** + * rdt_cdp_peer_get - Retrieve CDP peer if it exists + * @r: RDT resource to which RDT domain @d belongs + * @d: Cache instance for which a CDP peer is requested + * @r_cdp: RDT resource that shares hardware with @r (RDT resource peer) + * Used to return the result. + * @d_cdp: RDT domain that shares hardware with @d (RDT domain peer) + * Used to return the result. + * + * RDT resources are managed independently and by extension the RDT domains + * (RDT resource instances) are managed independently also. The Code and + * Data Prioritization (CDP) RDT resources, while managed independently, + * could refer to the same underlying hardware. For example, + * RDT_RESOURCE_L2CODE and RDT_RESOURCE_L2DATA both refer to the L2 cache. + * + * When provided with an RDT resource @r and an instance of that RDT + * resource @d rdt_cdp_peer_get() will return if there is a peer RDT + * resource and the exact instance that shares the same hardware. + * + * Return: 0 if a CDP peer was found, <0 on error or if no CDP peer exists. + * If a CDP peer was found, @r_cdp will point to the peer RDT resource + * and @d_cdp will point to the peer RDT domain. + */ +static int __attribute__((unused)) rdt_cdp_peer_get(struct rdt_resource *r, + struct rdt_domain *d, + struct rdt_resource **r_cdp, + struct rdt_domain **d_cdp) +{ + struct rdt_resource *_r_cdp = NULL; + struct rdt_domain *_d_cdp = NULL; + int ret = 0; + + switch (r->rid) { + case RDT_RESOURCE_L3DATA: + _r_cdp = &rdt_resources_all[RDT_RESOURCE_L3CODE]; + break; + case RDT_RESOURCE_L3CODE: + _r_cdp = &rdt_resources_all[RDT_RESOURCE_L3DATA]; + break; + case RDT_RESOURCE_L2DATA: + _r_cdp = &rdt_resources_all[RDT_RESOURCE_L2CODE]; + break; + case RDT_RESOURCE_L2CODE: + _r_cdp = &rdt_resources_all[RDT_RESOURCE_L2DATA]; + break; + default: + ret = -ENOENT; + goto out; + } + + /* + * When a new CPU comes online and CDP is enabled then the new + * RDT domains (if any) associated with both CDP RDT resources + * are added in the same CPU online routine while the + * rdtgroup_mutex is held. It should thus not happen for one + * RDT domain to exist and be associated with its RDT CDP + * resource but there is no RDT domain associated with the + * peer RDT CDP resource. Hence the WARN. + */ + _d_cdp = rdt_find_domain(_r_cdp, d->id, NULL); + if (WARN_ON(!_d_cdp)) { + _r_cdp = NULL; + ret = -ENOENT; + } + +out: + *r_cdp = _r_cdp; + *d_cdp = _d_cdp; + + return ret; +} + /** * rdtgroup_cbm_overlaps - Does CBM for intended closid overlap with other * @r: Resource to which domain instance @d belongs. -- 2.17.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 1/3] x86/intel_rdt: Introduce utility to obtain CDP peer 2018-09-26 18:59 ` [PATCH 1/3] x86/intel_rdt: Introduce utility to obtain CDP peer Reinette Chatre @ 2018-10-03 6:46 ` Thomas Gleixner 0 siblings, 0 replies; 9+ messages in thread From: Thomas Gleixner @ 2018-10-03 6:46 UTC (permalink / raw) To: Reinette Chatre Cc: fenghua.yu, tony.luck, jithu.joseph, gavin.hindman, dave.hansen, mingo, hpa, x86, linux-kernel On Wed, 26 Sep 2018, Reinette Chatre wrote: > + * Return: 0 if a CDP peer was found, <0 on error or if no CDP peer exists. > + * If a CDP peer was found, @r_cdp will point to the peer RDT resource > + * and @d_cdp will point to the peer RDT domain. > + */ > +static int __attribute__((unused)) rdt_cdp_peer_get(struct rdt_resource *r, > + struct rdt_domain *d, > + struct rdt_resource **r_cdp, > + struct rdt_domain **d_cdp) > +{ > + struct rdt_resource *_r_cdp = NULL; > + struct rdt_domain *_d_cdp = NULL; > + int ret = 0; > + > + switch (r->rid) { > + case RDT_RESOURCE_L3DATA: > + _r_cdp = &rdt_resources_all[RDT_RESOURCE_L3CODE]; > + break; > + case RDT_RESOURCE_L3CODE: > + _r_cdp = &rdt_resources_all[RDT_RESOURCE_L3DATA]; > + break; > + case RDT_RESOURCE_L2DATA: > + _r_cdp = &rdt_resources_all[RDT_RESOURCE_L2CODE]; > + break; > + case RDT_RESOURCE_L2CODE: > + _r_cdp = &rdt_resources_all[RDT_RESOURCE_L2DATA]; > + break; > + default: > + ret = -ENOENT; > + goto out; > + } > + > + /* > + * When a new CPU comes online and CDP is enabled then the new > + * RDT domains (if any) associated with both CDP RDT resources > + * are added in the same CPU online routine while the > + * rdtgroup_mutex is held. It should thus not happen for one > + * RDT domain to exist and be associated with its RDT CDP > + * resource but there is no RDT domain associated with the > + * peer RDT CDP resource. Hence the WARN. > + */ > + _d_cdp = rdt_find_domain(_r_cdp, d->id, NULL); > + if (WARN_ON(!_d_cdp)) { > + _r_cdp = NULL; > + ret = -ENOENT; While this should never happen, the return value is ambiguous. I'd rather use EINVAL or such and propagate it further down at the call site. Thanks, tglx ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/3] x86/intel_rdt: CBM overlap should also check for overlap with CDP peer 2018-09-26 18:59 [PATCH 0/3] x86/intel_rdt: Fix exclusive mode with CDP resources Reinette Chatre 2018-09-26 18:59 ` [PATCH 1/3] x86/intel_rdt: Introduce utility to obtain CDP peer Reinette Chatre @ 2018-09-26 18:59 ` Reinette Chatre 2018-10-03 7:02 ` Thomas Gleixner 2018-09-26 18:59 ` [PATCH 3/3] x86/intel_rdt: Fix initial allocation to consider CDP Reinette Chatre 2 siblings, 1 reply; 9+ messages in thread From: Reinette Chatre @ 2018-09-26 18:59 UTC (permalink / raw) To: tglx, fenghua.yu, tony.luck Cc: jithu.joseph, gavin.hindman, dave.hansen, mingo, hpa, x86, linux-kernel, Reinette Chatre The CBM overlap test is used to manage the allocations of RDT resources where overlap is possible between resource groups. When a resource group is in exclusive mode then there should be no overlap between resource groups. The current overlap test only considers overlap between the same resources, for example, that usage of a RDT_RESOURCE_L2DATA resource in one resource group does not overlap with usage of a RDT_RESOURCE_L2DATA resource in another resource group. The problem with this is that it allows overlap between a RDT_RESOURCE_L2DATA resource in one resource group with a RDT_RESOURCE_L2CODE resource in another resource group - even if both resource groups are in exclusive mode. This is a problem because even though these appear to be different resources they end up sharing the same underlying hardware and thus does not fulfill the user's request for exclusive use of hardware resources. Fix this by including the CDP peer (if there is one) in every CBM overlap test. This does not impact the overlap between resources within the same exclusive resource group that is allowed. Fixes: 49f7b4efa110 ("x86/intel_rdt: Enable setting of exclusive mode") Reported-by: Jithu Joseph <jithu.joseph@intel.com> Signed-off-by: Reinette Chatre <reinette.chatre@intel.com> Tested-by: Jithu Joseph <jithu.joseph@intel.com> Acked-by: Fenghua Yu <fenghua.yu@intel.com> --- arch/x86/kernel/cpu/intel_rdt_rdtgroup.c | 51 ++++++++++++++++++++---- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c index c8c02c16d072..3f56b4e624ea 100644 --- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c +++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c @@ -983,10 +983,9 @@ static int rdtgroup_mode_show(struct kernfs_open_file *of, * If a CDP peer was found, @r_cdp will point to the peer RDT resource * and @d_cdp will point to the peer RDT domain. */ -static int __attribute__((unused)) rdt_cdp_peer_get(struct rdt_resource *r, - struct rdt_domain *d, - struct rdt_resource **r_cdp, - struct rdt_domain **d_cdp) +static int rdt_cdp_peer_get(struct rdt_resource *r, struct rdt_domain *d, + struct rdt_resource **r_cdp, + struct rdt_domain **d_cdp) { struct rdt_resource *_r_cdp = NULL; struct rdt_domain *_d_cdp = NULL; @@ -1033,7 +1032,7 @@ static int __attribute__((unused)) rdt_cdp_peer_get(struct rdt_resource *r, } /** - * rdtgroup_cbm_overlaps - Does CBM for intended closid overlap with other + * _rdtgroup_cbm_overlaps - Does CBM for intended closid overlap with other * @r: Resource to which domain instance @d belongs. * @d: The domain instance for which @closid is being tested. * @cbm: Capacity bitmask being tested. @@ -1049,8 +1048,8 @@ static int __attribute__((unused)) rdt_cdp_peer_get(struct rdt_resource *r, * * Return: false if CBM does not overlap, true if it does. */ -bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, - u32 _cbm, int closid, bool exclusive) +static bool _rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, + u32 _cbm, int closid, bool exclusive) { unsigned long *cbm = (unsigned long *)&_cbm; unsigned long *ctrl_b; @@ -1087,6 +1086,44 @@ bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, return false; } +/** + * rdtgroup_cbm_overlaps - Does CBM overlap with other use of hardware + * @r: Resource to which domain instance @d belongs. + * @d: The domain instance for which @closid is being tested. + * @cbm: Capacity bitmask being tested. + * @closid: Intended closid for @cbm. + * @exclusive: Only check if overlaps with exclusive resource groups + * + * Resources that can be allocated using a CBM can use the CBM to control + * the overlap of these allocations. rdtgroup_cmb_overlaps() is the test + * for overlap. Overlap test is not limited to the specific resource for + * which the CBM is intended though - when dealing with CDP resources that + * share the underlying hardware the overlap check should be performed on + * the CDP resource sharing the hardware also. + * + * Refer to description of _rdtgroup_cbm_overlaps() for the details of the + * overlap test. + * + * Return: true if CBM overlap detected, false if there is no overlap + */ +bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, + u32 _cbm, int closid, bool exclusive) +{ + struct rdt_resource *r_cdp; + struct rdt_domain *d_cdp; + bool ret; + + ret = _rdtgroup_cbm_overlaps(r, d, _cbm, closid, exclusive); + if (ret) + return ret; + + if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) == 0) + return _rdtgroup_cbm_overlaps(r_cdp, d_cdp, _cbm, + closid, exclusive); + + return ret; +} + /** * rdtgroup_mode_test_exclusive - Test if this resource group can be exclusive * -- 2.17.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 2/3] x86/intel_rdt: CBM overlap should also check for overlap with CDP peer 2018-09-26 18:59 ` [PATCH 2/3] x86/intel_rdt: CBM overlap should also check for overlap with " Reinette Chatre @ 2018-10-03 7:02 ` Thomas Gleixner 2018-10-03 18:16 ` Reinette Chatre 0 siblings, 1 reply; 9+ messages in thread From: Thomas Gleixner @ 2018-10-03 7:02 UTC (permalink / raw) To: Reinette Chatre Cc: fenghua.yu, tony.luck, jithu.joseph, gavin.hindman, dave.hansen, mingo, hpa, x86, linux-kernel On Wed, 26 Sep 2018, Reinette Chatre wrote: > /** > - * rdtgroup_cbm_overlaps - Does CBM for intended closid overlap with other > + * _rdtgroup_cbm_overlaps - Does CBM for intended closid overlap with other > * @r: Resource to which domain instance @d belongs. > * @d: The domain instance for which @closid is being tested. > * @cbm: Capacity bitmask being tested. > @@ -1049,8 +1048,8 @@ static int __attribute__((unused)) rdt_cdp_peer_get(struct rdt_resource *r, > * > * Return: false if CBM does not overlap, true if it does. > */ > -bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, > - u32 _cbm, int closid, bool exclusive) > +static bool _rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, > + u32 _cbm, int closid, bool exclusive) Existing issue. The documentation uses @cbm, but the argument is _cbm. Also please make this __rdtgroup_cbm_overlaps(). Double underscores are standing more out. > { > unsigned long *cbm = (unsigned long *)&_cbm; > unsigned long *ctrl_b; > @@ -1087,6 +1086,44 @@ bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, > return false; > } > > +/** > + * rdtgroup_cbm_overlaps - Does CBM overlap with other use of hardware > + * @r: Resource to which domain instance @d belongs. > + * @d: The domain instance for which @closid is being tested. > + * @cbm: Capacity bitmask being tested. > + * @closid: Intended closid for @cbm. > + * @exclusive: Only check if overlaps with exclusive resource groups > + * > + * Resources that can be allocated using a CBM can use the CBM to control > + * the overlap of these allocations. rdtgroup_cmb_overlaps() is the test > + * for overlap. Overlap test is not limited to the specific resource for > + * which the CBM is intended though - when dealing with CDP resources that > + * share the underlying hardware the overlap check should be performed on > + * the CDP resource sharing the hardware also. > + * > + * Refer to description of _rdtgroup_cbm_overlaps() for the details of the > + * overlap test. > + * > + * Return: true if CBM overlap detected, false if there is no overlap > + */ > +bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, > + u32 _cbm, int closid, bool exclusive) Ditto. And here is no reason for using _cbm. > +{ > + struct rdt_resource *r_cdp; > + struct rdt_domain *d_cdp; > + bool ret; > + > + ret = _rdtgroup_cbm_overlaps(r, d, _cbm, closid, exclusive); > + if (ret) > + return ret; if (__rdtgroup_cbm_overlaps(r, d, _cbm, closid, exclusive)) return true; > + > + if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) == 0) > + return _rdtgroup_cbm_overlaps(r_cdp, d_cdp, _cbm, > + closid, exclusive); if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) < 0) return false; return __rdtgroup_cbm_overlaps(r_cpd, d_cdp, _cbm, closid, exclusive); Makes the whole thing more obvious. Thanks, tglx ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/3] x86/intel_rdt: CBM overlap should also check for overlap with CDP peer 2018-10-03 7:02 ` Thomas Gleixner @ 2018-10-03 18:16 ` Reinette Chatre 2018-10-03 19:43 ` Thomas Gleixner 0 siblings, 1 reply; 9+ messages in thread From: Reinette Chatre @ 2018-10-03 18:16 UTC (permalink / raw) To: Thomas Gleixner Cc: fenghua.yu, tony.luck, jithu.joseph, gavin.hindman, dave.hansen, mingo, hpa, x86, linux-kernel Hi Thomas, On 10/3/2018 12:02 AM, Thomas Gleixner wrote: > On Wed, 26 Sep 2018, Reinette Chatre wrote: >> /** >> - * rdtgroup_cbm_overlaps - Does CBM for intended closid overlap with other >> + * _rdtgroup_cbm_overlaps - Does CBM for intended closid overlap with other >> * @r: Resource to which domain instance @d belongs. >> * @d: The domain instance for which @closid is being tested. >> * @cbm: Capacity bitmask being tested. >> @@ -1049,8 +1048,8 @@ static int __attribute__((unused)) rdt_cdp_peer_get(struct rdt_resource *r, >> * >> * Return: false if CBM does not overlap, true if it does. >> */ >> -bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, >> - u32 _cbm, int closid, bool exclusive) >> +static bool _rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, >> + u32 _cbm, int closid, bool exclusive) > > Existing issue. The documentation uses @cbm, but the argument is _cbm. Thanks for spotting this. > > Also please make this __rdtgroup_cbm_overlaps(). Double underscores are > standing more out. Will do. > >> { >> unsigned long *cbm = (unsigned long *)&_cbm; >> unsigned long *ctrl_b; >> @@ -1087,6 +1086,44 @@ bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, >> return false; >> } >> >> +/** >> + * rdtgroup_cbm_overlaps - Does CBM overlap with other use of hardware >> + * @r: Resource to which domain instance @d belongs. >> + * @d: The domain instance for which @closid is being tested. >> + * @cbm: Capacity bitmask being tested. >> + * @closid: Intended closid for @cbm. >> + * @exclusive: Only check if overlaps with exclusive resource groups >> + * >> + * Resources that can be allocated using a CBM can use the CBM to control >> + * the overlap of these allocations. rdtgroup_cmb_overlaps() is the test >> + * for overlap. Overlap test is not limited to the specific resource for >> + * which the CBM is intended though - when dealing with CDP resources that >> + * share the underlying hardware the overlap check should be performed on >> + * the CDP resource sharing the hardware also. >> + * >> + * Refer to description of _rdtgroup_cbm_overlaps() for the details of the >> + * overlap test. >> + * >> + * Return: true if CBM overlap detected, false if there is no overlap >> + */ >> +bool rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, >> + u32 _cbm, int closid, bool exclusive) > > Ditto. And here is no reason for using _cbm. Thanks for spotting this also, will do. > >> +{ >> + struct rdt_resource *r_cdp; >> + struct rdt_domain *d_cdp; >> + bool ret; >> + >> + ret = _rdtgroup_cbm_overlaps(r, d, _cbm, closid, exclusive); >> + if (ret) >> + return ret; > > if (__rdtgroup_cbm_overlaps(r, d, _cbm, closid, exclusive)) > return true; > >> + >> + if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) == 0) >> + return _rdtgroup_cbm_overlaps(r_cdp, d_cdp, _cbm, >> + closid, exclusive); > > if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) < 0) > return false; > > return __rdtgroup_cbm_overlaps(r_cpd, d_cdp, _cbm, closid, exclusive); > > Makes the whole thing more obvious. I think a different change is needed to support the request from your review of the first patch to propagate that unthinkable error where only one of the CDP peers could have an rdt_domain associated with it. In the above that error in question from rdt_cdp_peer_get() will be lost. I could do the following in support of propagating that error (note that in support of the code below __rdtgroup_cbm_overlaps() also changes to return int instead of bool): int rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, u32 cbm, int closid, bool exclusive) { struct rdt_resource *r_cdp; struct rdt_domain *d_cdp; int ret; if (__rdtgroup_cbm_overlaps(r, d, cbm, closid, exclusive)) return 1; ret = rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp); if (ret == -ENOENT) { return 0; } else if (ret == -EINVAL) { rdt_last_cmd_puts("Error finding CDP peer\n"); return ret; } else { return __rdtgroup_cbm_overlaps(r_cdp, d_cdp, cbm, closid, exclusive); } return -EINVAL; } With the above change in rdtgroup_cbm_overlaps() the call sites then change to for example: ret = rdtgroup_cbm_overlaps(r, d, cbm_val, rdtgrp->closid, true); if (ret < 0) { /* last_cmd_status already populated with error */ return -EINVAL; } else if (ret == 1) { rdt_last_cmd_puts("overlaps with exclusive group\n"); return -EINVAL; } /* fall through when no overlap detected */ Would this be acceptable? Thank you very much Reinette ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/3] x86/intel_rdt: CBM overlap should also check for overlap with CDP peer 2018-10-03 18:16 ` Reinette Chatre @ 2018-10-03 19:43 ` Thomas Gleixner 2018-10-03 19:51 ` Reinette Chatre 0 siblings, 1 reply; 9+ messages in thread From: Thomas Gleixner @ 2018-10-03 19:43 UTC (permalink / raw) To: Reinette Chatre Cc: fenghua.yu, tony.luck, jithu.joseph, gavin.hindman, dave.hansen, mingo, hpa, x86, linux-kernel Reinette, On Wed, 3 Oct 2018, Reinette Chatre wrote: > On 10/3/2018 12:02 AM, Thomas Gleixner wrote: > >> +{ > >> + struct rdt_resource *r_cdp; > >> + struct rdt_domain *d_cdp; > >> + bool ret; > >> + > >> + ret = _rdtgroup_cbm_overlaps(r, d, _cbm, closid, exclusive); > >> + if (ret) > >> + return ret; > > > > if (__rdtgroup_cbm_overlaps(r, d, _cbm, closid, exclusive)) > > return true; > > > >> + > >> + if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) == 0) > >> + return _rdtgroup_cbm_overlaps(r_cdp, d_cdp, _cbm, > >> + closid, exclusive); > > > > if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) < 0) > > return false; > > > > return __rdtgroup_cbm_overlaps(r_cpd, d_cdp, _cbm, closid, exclusive); > > > > Makes the whole thing more obvious. > > I think a different change is needed to support the request from your > review of the first patch to propagate that unthinkable error where only > one of the CDP peers could have an rdt_domain associated with it. > > In the above that error in question from rdt_cdp_peer_get() will be lost. > > I could do the following in support of propagating that error (note that > in support of the code below __rdtgroup_cbm_overlaps() also changes to > return int instead of bool): > > int rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, > u32 cbm, int closid, bool exclusive) > { > struct rdt_resource *r_cdp; > struct rdt_domain *d_cdp; > int ret; > > if (__rdtgroup_cbm_overlaps(r, d, cbm, closid, exclusive)) > return 1; > > ret = rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp); > if (ret == -ENOENT) { > return 0; > } else if (ret == -EINVAL) { > rdt_last_cmd_puts("Error finding CDP peer\n"); > return ret; > } else { > return __rdtgroup_cbm_overlaps(r_cdp, d_cdp, cbm, > closid, exclusive); > } > > return -EINVAL; > } > > With the above change in rdtgroup_cbm_overlaps() the call sites then > change to for example: > > ret = rdtgroup_cbm_overlaps(r, d, cbm_val, rdtgrp->closid, true); > if (ret < 0) { > /* last_cmd_status already populated with error */ > return -EINVAL; > } else if (ret == 1) { > rdt_last_cmd_puts("overlaps with exclusive group\n"); > return -EINVAL; > } > /* fall through when no overlap detected */ > > Would this be acceptable? We really have to think about that whether it's worth it. Looking at the resulting code I doubt it. Then I'd rather prefer the warnon and the simpler code. But either way works for me. Thanks, tglx ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/3] x86/intel_rdt: CBM overlap should also check for overlap with CDP peer 2018-10-03 19:43 ` Thomas Gleixner @ 2018-10-03 19:51 ` Reinette Chatre 0 siblings, 0 replies; 9+ messages in thread From: Reinette Chatre @ 2018-10-03 19:51 UTC (permalink / raw) To: Thomas Gleixner Cc: fenghua.yu, tony.luck, jithu.joseph, gavin.hindman, dave.hansen, mingo, hpa, x86, linux-kernel Hi Thomas, On 10/3/2018 12:43 PM, Thomas Gleixner wrote: > On Wed, 3 Oct 2018, Reinette Chatre wrote: >> On 10/3/2018 12:02 AM, Thomas Gleixner wrote: >>>> +{ >>>> + struct rdt_resource *r_cdp; >>>> + struct rdt_domain *d_cdp; >>>> + bool ret; >>>> + >>>> + ret = _rdtgroup_cbm_overlaps(r, d, _cbm, closid, exclusive); >>>> + if (ret) >>>> + return ret; >>> >>> if (__rdtgroup_cbm_overlaps(r, d, _cbm, closid, exclusive)) >>> return true; >>> >>>> + >>>> + if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) == 0) >>>> + return _rdtgroup_cbm_overlaps(r_cdp, d_cdp, _cbm, >>>> + closid, exclusive); >>> >>> if (rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp) < 0) >>> return false; >>> >>> return __rdtgroup_cbm_overlaps(r_cpd, d_cdp, _cbm, closid, exclusive); >>> >>> Makes the whole thing more obvious. >> >> I think a different change is needed to support the request from your >> review of the first patch to propagate that unthinkable error where only >> one of the CDP peers could have an rdt_domain associated with it. >> >> In the above that error in question from rdt_cdp_peer_get() will be lost. >> >> I could do the following in support of propagating that error (note that >> in support of the code below __rdtgroup_cbm_overlaps() also changes to >> return int instead of bool): >> >> int rdtgroup_cbm_overlaps(struct rdt_resource *r, struct rdt_domain *d, >> u32 cbm, int closid, bool exclusive) >> { >> struct rdt_resource *r_cdp; >> struct rdt_domain *d_cdp; >> int ret; >> >> if (__rdtgroup_cbm_overlaps(r, d, cbm, closid, exclusive)) >> return 1; >> >> ret = rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp); >> if (ret == -ENOENT) { >> return 0; >> } else if (ret == -EINVAL) { >> rdt_last_cmd_puts("Error finding CDP peer\n"); >> return ret; >> } else { >> return __rdtgroup_cbm_overlaps(r_cdp, d_cdp, cbm, >> closid, exclusive); >> } >> >> return -EINVAL; >> } >> >> With the above change in rdtgroup_cbm_overlaps() the call sites then >> change to for example: >> >> ret = rdtgroup_cbm_overlaps(r, d, cbm_val, rdtgrp->closid, true); >> if (ret < 0) { >> /* last_cmd_status already populated with error */ >> return -EINVAL; >> } else if (ret == 1) { >> rdt_last_cmd_puts("overlaps with exclusive group\n"); >> return -EINVAL; >> } >> /* fall through when no overlap detected */ >> >> Would this be acceptable? > > We really have to think about that whether it's worth it. Looking at the > resulting code I doubt it. Then I'd rather prefer the warnon and the > simpler code. But either way works for me. Thank you very much. I'll resubmit with the changes you prefer. Reinette ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 3/3] x86/intel_rdt: Fix initial allocation to consider CDP 2018-09-26 18:59 [PATCH 0/3] x86/intel_rdt: Fix exclusive mode with CDP resources Reinette Chatre 2018-09-26 18:59 ` [PATCH 1/3] x86/intel_rdt: Introduce utility to obtain CDP peer Reinette Chatre 2018-09-26 18:59 ` [PATCH 2/3] x86/intel_rdt: CBM overlap should also check for overlap with " Reinette Chatre @ 2018-09-26 18:59 ` Reinette Chatre 2 siblings, 0 replies; 9+ messages in thread From: Reinette Chatre @ 2018-09-26 18:59 UTC (permalink / raw) To: tglx, fenghua.yu, tony.luck Cc: jithu.joseph, gavin.hindman, dave.hansen, mingo, hpa, x86, linux-kernel, Reinette Chatre When a new resource group is created it is initialized with a default allocation that considers which portions of cache are currently available for sharing across all resource groups or which portions of cache are currently unused. If a CDP allocation forms part of a resource group that is in exclusive mode then it should be ensured that no new allocation overlaps with any resource that shares the underlying hardware. The current initial allocation does not take this sharing of hardware into account and a new allocation in a resource that shares the same hardware would affect the exclusive resource group. Fix this by considering the allocation of a peer RDT domain - a RDT domain sharing the same hardware - as part of the test to determine which portion of cache is in use and available for use. Fixes: 95f0b77efa57 ("x86/intel_rdt: Initialize new resource group with sane defaults") Signed-off-by: Reinette Chatre <reinette.chatre@intel.com> Acked-by: Fenghua Yu <fenghua.yu@intel.com> --- arch/x86/kernel/cpu/intel_rdt_rdtgroup.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c index 3f56b4e624ea..5b7932af522a 100644 --- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c +++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c @@ -2459,11 +2459,14 @@ static void cbm_ensure_valid(u32 *_val, struct rdt_resource *r) */ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp) { + struct rdt_resource *r_cdp = NULL; + struct rdt_domain *d_cdp = NULL; u32 used_b = 0, unused_b = 0; u32 closid = rdtgrp->closid; struct rdt_resource *r; enum rdtgrp_mode mode; struct rdt_domain *d; + u32 peer_ctl; int i, ret; u32 *ctrl; @@ -2475,6 +2478,7 @@ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp) if (r->rid == RDT_RESOURCE_MBA) continue; list_for_each_entry(d, &r->domains, list) { + rdt_cdp_peer_get(r, d, &r_cdp, &d_cdp); d->have_new_ctrl = false; d->new_ctrl = r->cache.shareable_bits; used_b = r->cache.shareable_bits; @@ -2484,9 +2488,19 @@ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp) mode = rdtgroup_mode_by_closid(i); if (mode == RDT_MODE_PSEUDO_LOCKSETUP) break; - used_b |= *ctrl; + /* + * If CDP is active include peer + * domain's usage to ensure there + * is no overlap with an exclusive + * group. + */ + if (d_cdp) + peer_ctl = d_cdp->ctrl_val[i]; + else + peer_ctl = 0; + used_b |= *ctrl | peer_ctl; if (mode == RDT_MODE_SHAREABLE) - d->new_ctrl |= *ctrl; + d->new_ctrl |= *ctrl | peer_ctl; } } if (d->plr && d->plr->cbm > 0) -- 2.17.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-10-03 19:51 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-09-26 18:59 [PATCH 0/3] x86/intel_rdt: Fix exclusive mode with CDP resources Reinette Chatre 2018-09-26 18:59 ` [PATCH 1/3] x86/intel_rdt: Introduce utility to obtain CDP peer Reinette Chatre 2018-10-03 6:46 ` Thomas Gleixner 2018-09-26 18:59 ` [PATCH 2/3] x86/intel_rdt: CBM overlap should also check for overlap with " Reinette Chatre 2018-10-03 7:02 ` Thomas Gleixner 2018-10-03 18:16 ` Reinette Chatre 2018-10-03 19:43 ` Thomas Gleixner 2018-10-03 19:51 ` Reinette Chatre 2018-09-26 18:59 ` [PATCH 3/3] x86/intel_rdt: Fix initial allocation to consider CDP Reinette Chatre
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).