Re: Question about the CAT and CMT in Xen

[Meng Xu <xumengpanda@gmail.com>]

Hi Andrew and Chao,

[Important things go first] It turns out my machine (Intel E5-2618L
v3) does have CAT capability!
Xen gives the false alarm that my machine does not have it. This
should be a bug, IMO. :-)

2015-09-01 10:42 GMT-04:00 Meng Xu <xumengpanda@gmail.com>:
> 2015-09-01 10:30 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>> On 01/09/15 15:20, Meng Xu wrote:
>>> 2015-09-01 9:04 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>>>> On 01/09/15 13:55, Meng Xu wrote:
>>>>> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
>>>>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
>>>>>>> I looked into the xen/arch/x86/psr.c and found that the function
>>>>>>> cat_cpu_init() just returned without initializing the variable
>>>>>>> "cat_socket_enable".
>>>>>>>
>>>>>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
>>>>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
>>>>>>> cat_cpu_init().

I'm thinking this check could be wrong for Intel E5-2618L v3. It
should work on Chao's machine but not on mine. There should be a
better way to check this probably. :-)

---

I used another way to check the CAT capability, as suggested by Priya
(cc.ed) from Intel.
I did the following steps as Priya suggested:
1. Download msr-tools utility on your linux distribution to perform
msr read write operations./ if you already have it installed modprobe
msr
2. rdmsr 0xc91
which returns 0xfffff
3. wrmsr -p 1 0xc91 0xf
which does not return anything
4. wrmsr -p 1 0xc8f 0x100000000
which does not return anything
5. rdmsr 0xc91
which returns 0xf

This shows that the CPU does have the MSRs that are used for CAT.

I also run the CAT tools on Linux provided by Intel, which can be
downloaded at https://01.org/packet-processing/cache-monitoring-technology-memory-bandwidth-monitoring-cache-allocation-technology.
It shows me that I can set up different cache partitions for different COS.
Basically, the pre-configured cache setting provided by the CAT tools
work perfectly on my machine. :-)

------
OK. That just check the registers are there and tools do not return
error. It may still not work, right? :-)
Well, I also did some performance evaluation by running a small simple
benchmark I wrote.

The benchmark task sequentially access a 6MB array;

I run the benchmark on core 0 in the following scenarios:

Scenario 1): Core 0 is allocated for 8MB cache with CAT, the latency
of accessing the 6MB array is around 5.5M cycles;
Scenario 2): Core 0 is allocated for 4MB cache with CAT, the latency
of accessing the 6MB array is around 16.9M cycles.
The slowdown in scenario 2) is 16.9M / 5.5M ~=3x.

------ISSUES-------
I tried to run some noisy neighbors on another core to see how good
the LLC isolation CAT can provide, but found some *weird result*.
I run the benchmark task on core 0 and the noisy neighbor that access
20MB array on core 1;
These two cores are configured to have *different* cache areas: core 0
has  8MB cache, core 1 has 4MB cache;
These two cores are in two isolated cpuset. No other tasks runs on
these two cores.
If I run the benchmark alone, the latency is around 5.5M cycles;
but if run the benchmark along with the noisy neighbor, the latency
*decreases* to 4.9M cycles.

I double checked that the Turbo Boost is disabled by checking the MSR
value with the following command:
    rdmsr -pi 0x1a0 -f 38:38
    1=disabled
    0=enabled
    it returns 1.
I also disabled the cache prefetch in BIOS.

Now I'm very confused. How come the latency decreases when a noisy
neighbor is running. It seems that the noisy neighbor may help some
hardware/software prefetcher to prefetch the data for the benchmark.
But right now, I couldn't think out any other prefetchers that may
cause this...
The benchmark and the noisy neighbor are independent and don't share
the array data.

If you have any insight or comment, could you please let me know?

Thanks and best regards,

Meng

-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/