From mboxrd@z Thu Jan 1 00:00:00 1970 From: Meng Xu Subject: Re: Question about the CAT and CMT in Xen Date: Tue, 1 Sep 2015 22:55:55 -0400 Message-ID: References: <55E49C11.40303@citrix.com> <20150901054716.GA19417@pengc-linux.bj.intel.com> <55E5A263.5050708@citrix.com> <55E5B675.7020607@citrix.com> <20150902023156.GD19417@pengc-linux.bj.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20150902023156.GD19417@pengc-linux.bj.intel.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Chao Peng Cc: Andrew Cooper , "xen-devel@lists.xen.org" , "Autee, Priya V" List-Id: xen-devel@lists.xenproject.org 2015-09-01 22:31 GMT-04:00 Chao Peng : > On Tue, Sep 01, 2015 at 09:51:57PM -0400, Meng Xu wrote: >> Hi Andrew and Chao, >> >> [Important things go first] It turns out my machine (Intel E5-2618L >> v3) does have CAT capability! >> Xen gives the false alarm that my machine does not have it. This >> should be a bug, IMO. :-) > > Even some Haswell Servers do support CAT, but it's usually > model-specific and the feature is not enumerated in a standard way as > listed in Intel SDM. Xen now however only takes care of the standard > enumeration so your case is not detected. Ah. I prefer not to upgrade the firmware... Last time when I upgrade the firmware, I screw up the whole system. :-( Is it possible to check if CAT is supported on a machine with another way? Right now, I think you are checking some specific bits returned by CPUID. > > This could be done by adding cpu model check in Xen code or even use > updated firmware, which is I prefered. > >> >> 2015-09-01 10:42 GMT-04:00 Meng Xu : >> > 2015-09-01 10:30 GMT-04:00 Andrew Cooper : >> >> On 01/09/15 15:20, Meng Xu wrote: >> >>> 2015-09-01 9:04 GMT-04:00 Andrew Cooper : >> >>>> On 01/09/15 13:55, Meng Xu wrote: >> >>>>> 2015-09-01 1:47 GMT-04:00 Chao Peng : >> >>>>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote: >> >>>>>>> I looked into the xen/arch/x86/psr.c and found that the function >> >>>>>>> cat_cpu_init() just returned without initializing the variable >> >>>>>>> "cat_socket_enable". >> >>>>>>> >> >>>>>>> Both !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level < >> >>>>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function >> >>>>>>> cat_cpu_init(). >> >> I'm thinking this check could be wrong for Intel E5-2618L v3. It >> should work on Chao's machine but not on mine. There should be a >> better way to check this probably. :-) >> >> --- >> >> I used another way to check the CAT capability, as suggested by Priya >> (cc.ed) from Intel. >> I did the following steps as Priya suggested: >> 1. Download msr-tools utility on your linux distribution to perform >> msr read write operations./ if you already have it installed modprobe >> msr >> 2. rdmsr 0xc91 >> which returns 0xfffff >> 3. wrmsr -p 1 0xc91 0xf >> which does not return anything >> 4. wrmsr -p 1 0xc8f 0x100000000 >> which does not return anything >> 5. rdmsr 0xc91 >> which returns 0xf >> >> This shows that the CPU does have the MSRs that are used for CAT. >> >> I also run the CAT tools on Linux provided by Intel, which can be >> downloaded at https://01.org/packet-processing/cache-monitoring-technology-memory-bandwidth-monitoring-cache-allocation-technology. >> It shows me that I can set up different cache partitions for different COS. >> Basically, the pre-configured cache setting provided by the CAT tools >> work perfectly on my machine. :-) >> >> ------ >> OK. That just check the registers are there and tools do not return >> error. It may still not work, right? :-) >> Well, I also did some performance evaluation by running a small simple >> benchmark I wrote. >> >> The benchmark task sequentially access a 6MB array; >> >> I run the benchmark on core 0 in the following scenarios: >> >> Scenario 1): Core 0 is allocated for 8MB cache with CAT, the latency >> of accessing the 6MB array is around 5.5M cycles; >> Scenario 2): Core 0 is allocated for 4MB cache with CAT, the latency >> of accessing the 6MB array is around 16.9M cycles. >> The slowdown in scenario 2) is 16.9M / 5.5M ~=3x. >> >> ------ISSUES------- >> I tried to run some noisy neighbors on another core to see how good >> the LLC isolation CAT can provide, but found some *weird result*. >> I run the benchmark task on core 0 and the noisy neighbor that access >> 20MB array on core 1; >> These two cores are configured to have *different* cache areas: core 0 >> has 8MB cache, core 1 has 4MB cache; >> These two cores are in two isolated cpuset. No other tasks runs on >> these two cores. >> If I run the benchmark alone, the latency is around 5.5M cycles; >> but if run the benchmark along with the noisy neighbor, the latency >> *decreases* to 4.9M cycles. >> >> I double checked that the Turbo Boost is disabled by checking the MSR >> value with the following command: >> rdmsr -pi 0x1a0 -f 38:38 >> 1=disabled >> 0=enabled >> it returns 1. >> I also disabled the cache prefetch in BIOS. >> >> Now I'm very confused. How come the latency decreases when a noisy >> neighbor is running. It seems that the noisy neighbor may help some >> hardware/software prefetcher to prefetch the data for the benchmark. >> But right now, I couldn't think out any other prefetchers that may >> cause this... >> The benchmark and the noisy neighbor are independent and don't share >> the array data. > > Did you reboot the machine between your two tests and are the two cores > you used in the same socket? I didn't reboot between these two test. I'm using the same socket 0. Thank you very much! Meng ----------- Meng Xu PhD Student in Computer and Information Science University of Pennsylvania http://www.cis.upenn.edu/~mengxu/