Re: [PATCH 00/24] x86/resctrl: Merge the CDP resources

[Jamie Iles <jamie@nuviainc.com>]

Hi James,

On Fri, Oct 30, 2020 at 04:10:56PM +0000, James Morse wrote:
> Hi folks,
> 
> This series re-folds the resctrl code so the CDP resources (L3CODE et al)
> behaviour is all contained in the filesystem parts, with a minimum amount
> of arch specific code.
> 
> Arm have some CPU support for dividing caches into portions, and
> applying bandwidth limits at various points in the SoC. The collective term
> for these features is MPAM: Memory Partitioning and Monitoring.
> 
> MPAM is similar enough to Intel RDT, that it should use the defacto linux
> interface: resctrl. This filesystem currently lives under arch/x86, and is
> tightly coupled to the architecture.
> Ultimately, my plan is to split the existing resctrl code up to have an
> arch<->fs abstraction, then move all the bits out to fs/resctrl. From there
> MPAM can be wired up.
> 
> x86 might have two resources with cache controls, (L2 and L3) but has
> extra copies for CDP: L{2,3}{CODE,DATA}, which are marked as enabled
> if CDP is enabled for the corresponding cache.
> 
> MPAM has an equivalent feature to CDP, but its a property of the CPU,
> not the cache. Resctrl needs to have x86's odd/even behaviour, as that
> its the ABI, but this isn't how the MPAM hardware works. It is entirely
> possible that an in-kernel user of MPAM would not be using CDP, whereas
> resctrl is.
> Pretending L3CODE and L3DATA are entirely separate resources is a neat
> trick, but doing this is specific to x86.
> Doing this leaves the arch code in control of various parts of the
> filesystem ABI: the resources names, and the way the schemata are parsed.
> Allowing this stuff to vary between architectures is bad for user space.
> 
> 
> This series collapses the CODE/DATA resources, moving all the user-visible
> resctrl ABI into the filesystem code. CDP becomes the type of configuration
> being applied to a cache. This is done by adding a struct resctrl_schema to
> the parts of resctrl that will move to fs. This holds the arch-code resource
> that is in use for this schema, along with other properties like the name,
> and whether the configuration being applied is CODE/DATA/BOTH.
> 
> This lets us fold the extra resources out of the arch code so that they
> don't need to be duplicated if the equivalent feature to CDP is missing, or
> implemented in a different way.
> 
> 
> The first two patches split the resource and domain structs to have an
> arch specific 'hw' portion, and the rest that is visible to resctrl.
> Future series massage the resctrl code so there are no accesses to 'hw'
> structures in the parts of resctrl that will move to fs, providing helpers
> where necessary.
> 
> 
> Since anyone last looked at this, the CDP property has been made per-resource
> instead of global. MPAM will need to make this global in the arch code, as
> CODE/DATA closid are based on how the CPU tags traffic, not how the cache
> interprets it. resctrl sets CDP enabled on a resource, but reads it back on
> each one.
> The attempt to keep closids as-used-by-resctrl and closids as-written-to-hw
> appart has been dropped.
> There are two copies of num_closid. The version private to the arch code is
> the value discovered from hardware. resctrl has its own version, which it
> may write to, which is exposed to user-space. This lets resctrl do its
> odd/even thing, even if thats not how the hardware works.
> 
> This series adds temporary scaffolding, which it removes a few patches
> later. This is to allow things like the ctrlval arrays and resources to be
> merged separately, which should make is easier to bisect. These things
> are marked temporary, and should all be gone by the end of the series.
> 
> This series is a little rough around the monitors, would a fake
> struct resctrl_schema for the monitors simplify things, or be a source
> of bugs?
> 
> This series is based on v5.10-rc1, and can be retrieved from:
> git://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/resctrl_merge_cdp/v1
> 
> Parts were previously posted as an RFC here:
> https://lore.kernel.org/lkml/20200214182947.39194-1-james.morse@arm.com/

Reviewed-by: Jamie Iles <jamie@nuviainc.com>

Jamie