Re: [PATCH 05/32] x86/intel_rdt: Implement scheduling support for Intel RDT

[Nilay Vaish <nilayvaish@gmail.com>]

On 12 July 2016 at 20:02, Fenghua Yu <fenghua.yu@intel.com> wrote:
> From: Vikas Shivappa <vikas.shivappa@linux.intel.com>
>
> Adds support for IA32_PQR_ASSOC MSR writes during task scheduling. For
> Cache Allocation, MSR write would let the task fill in the cache
> 'subset' represented by the task's capacity bit mask.
>
> The high 32 bits in the per processor MSR IA32_PQR_ASSOC represents the
> CLOSid. During context switch kernel implements this by writing the
> CLOSid of the task belongs to the CPU's IA32_PQR_ASSOC MSR.
>
> This patch also implements a common software cache for IA32_PQR_MSR
> (RMID 0:9, CLOSId 32:63) to be used by both Cache monitoring (CMT) and
> Cache allocation. CMT updates the RMID where as cache_alloc updates the
> CLOSid in the software cache. During scheduling when the new RMID/CLOSid
> value is different from the cached values, IA32_PQR_MSR is updated.
> Since the measured rdmsr latency for IA32_PQR_MSR is very high (~250
>  cycles) this software cache is necessary to avoid reading the MSR to
> compare the current CLOSid value.
>
> The following considerations are done for the PQR MSR write so that it
> minimally impacts scheduler hot path:
>  - This path does not exist on any non-intel platforms.
>  - On Intel platforms, this would not exist by default unless INTEL_RDT
>  is enabled.
>  - remains a no-op when INTEL_RDT is enabled and intel SKU does not
>  support the feature.
>  - When feature is available and enabled, never does MSR write till the
>  user manually starts using one of the capacity bit masks.
>  - MSR write is only done when there is a task with different Closid is
>  scheduled on the CPU. Typically if the task groups are bound to be
>  scheduled on a set of CPUs, the number of MSR writes is greatly
>  reduced.
>  - A per CPU cache of CLOSids is maintained to do the check so that we
>  don't have to do a rdmsr which actually costs a lot of cycles.
>

I was thinking more about this software caching of CLOSids.  How
likely do you think these CLOSids would be found cached?  I think the
software cache would be very infrequently accessed, so it seems you
are likely to miss these in all levels of cache hierarchy and more
likely to have to fetch these from the main memory, which itself might
cost ~250 cycles.

--
Nilay