Question about the CAT and CMT in Xen

Question about the CAT and CMT in Xen

[Meng Xu]

[-- Attachment #1.1: Type: text/plain, Size: 2037 bytes --]

Hi Chao,

I'm Meng Xu from the University of Pennsylvania.

We purchased a computer that has Intel(R) Xeon(R) CPU E5-2618L v3 @ 2.30GHz
inside.
According to
http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
Intel E5-2618L v3 should support both CAT and CMT.

I'm playing the latest Xen on the machine to evaluate how CAT and CMT work
​.​


The commit point I used is in the staging branch, commit
7b99717f62caeac08eea224a177cd28f047ac4b5, which was push on Aug. 17th, 2015.
I checked that the branch has merged your code.

After I boot up the system, I tried the following commands and it shows
some errors.

​***​
My question is
​***​
:
1) Is there any specific BIOS configuration I need to configure?
2) The CMT is not enabled by default. Do I need to issue any command to
enable it?  I didn't find any document about how to enable it, according to
http://xenbits.xen.org/docs/unstable-staging/misc/xl-psr.html. :-(
3) I'm not 100% sure if this machine we purchased supports the CAT. Right
now, it shows "No such device" on the machine.
​ But it may probably because the hypervisor forgot to initialize it?​


Below is the details of what I tried:

# xl psr-hwinfo
Cache Monitoring Technology (CMT):
Enabled         : 0
Cache Allocation Technology (CAT):
libxl: error: libxl_psr.c:96:libxl__psr_cat_log_err_msg: CAT is not
supported in this system: No such device
Failed to get cat info

#xl psr-cmt-attach 0
libxl: error: libxl_psr.c:70:libxl__psr_cmt_log_err_msg: CMT is not
supported in this system: No such device

Please let me know if you need any further information to confirm if the
machine support the CAT or if it's because there is some issue with the
software layer. :-)

Thank you very much for your time and help in this question!

Best regards,

Meng


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

[-- Attachment #1.2: Type: text/html, Size: 2691 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Question about the CAT and CMT in Xen

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 2226 bytes --]

On 31/08/15 18:42, Meng Xu wrote:
> Hi Chao,
>
> I'm Meng Xu from the University of Pennsylvania.
>
> We purchased a computer that has Intel(R) Xeon(R) CPU E5-2618L v3 @
> 2.30GHz inside.
> According to
> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
> Intel E5-2618L v3 should support both CAT and CMT.
>
> I'm playing the latest Xen on the machine to evaluate how CAT and CMT
> work
> ​.​
>
>
> The commit point I used is in the staging branch, commit
> 7b99717f62caeac08eea224a177cd28f047ac4b5, which was push on Aug. 17th,
> 2015.
> I checked that the branch has merged your code.
>
> After I boot up the system, I tried the following commands and it
> shows some errors.
>
> ​***​
> My question is
> ​***​
> :
> 1) Is there any specific BIOS configuration I need to configure?
> 2) The CMT is not enabled by default. Do I need to issue any command
> to enable it?  I didn't find any document about how to enable it,
> according to
> http://xenbits.xen.org/docs/unstable-staging/misc/xl-psr.html. :-(
> 3) I'm not 100% sure if this machine we purchased supports the CAT.
> Right now, it shows "No such device" on the machine.
> ​ But it may probably because the hypervisor forgot to initialize it?​
>
>
> Below is the details of what I tried:
>
> # xl psr-hwinfo
> Cache Monitoring Technology (CMT):
> Enabled         : 0
> Cache Allocation Technology (CAT):
> libxl: error: libxl_psr.c:96:libxl__psr_cat_log_err_msg: CAT is not
> supported in this system: No such device
> Failed to get cat info
>
> #xl psr-cmt-attach 0
> libxl: error: libxl_psr.c:70:libxl__psr_cmt_log_err_msg: CMT is not
> supported in this system: No such device
>
> Please let me know if you need any further information to confirm if
> the machine support the CAT or if it's because there is some issue
> with the software layer. :-)
>
> Thank you very much for your time and help in this question!
>
> Best regards,
>
> Meng

To get started, use "psr=cat,cmt" on the Xen command line.  Neither are
enabled by default as they can potentially allocate quite a lot of
resources behind the scenes.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 3838 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Question about the CAT and CMT in Xen

[Meng Xu]

Hi Andrew and Chao,

2015-08-31 14:25 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>
> On 31/08/15 18:42, Meng Xu wrote:
>
> Hi Chao,
>
> I'm Meng Xu from the University of Pennsylvania.
>
> We purchased a computer that has Intel(R) Xeon(R) CPU E5-2618L v3 @ 2.30GHz inside.
> According to http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html, Intel E5-2618L v3 should support both CAT and CMT.
>
> I'm playing the latest Xen on the machine to evaluate how CAT and CMT work
> .
>
>
> The commit point I used is in the staging branch, commit 7b99717f62caeac08eea224a177cd28f047ac4b5, which was push on Aug. 17th, 2015.
> I checked that the branch has merged your code.
>
> After I boot up the system, I tried the following commands and it shows some errors.
>
> ***
> My question is
> ***
> :
> 1) Is there any specific BIOS configuration I need to configure?
> 2) The CMT is not enabled by default. Do I need to issue any command to enable it?  I didn't find any document about how to enable it, according to http://xenbits.xen.org/docs/unstable-staging/misc/xl-psr.html. :-(
> 3) I'm not 100% sure if this machine we purchased supports the CAT. Right now, it shows "No such device" on the machine.
> But it may probably because the hypervisor forgot to initialize it?
>
>
> Below is the details of what I tried:
>
> # xl psr-hwinfo
> Cache Monitoring Technology (CMT):
> Enabled         : 0
> Cache Allocation Technology (CAT):
> libxl: error: libxl_psr.c:96:libxl__psr_cat_log_err_msg: CAT is not supported in this system: No such device
> Failed to get cat info
>
> #xl psr-cmt-attach 0
> libxl: error: libxl_psr.c:70:libxl__psr_cmt_log_err_msg: CMT is not supported in this system: No such device
>
> Please let me know if you need any further information to confirm if the machine support the CAT or if it's because there is some issue with the software layer. :-)
>
> Thank you very much for your time and help in this question!
>
> Best regards,
>
> Meng
>
>
> To get started, use "psr=cat,cmt" on the Xen command line.  Neither are enabled by default as they can potentially allocate quite a lot of resources behind the scenes.


Thank you very much for your suggestion! Yes, after adding the
psr=cat:1,cmt:1, the CMT works. Command "
xl psr-hwinfo" shows the total RMID as 31. It means CMT is enabled/working.

However, the CAT is still not enabled on the socket. It have the same
error message:
Cache Allocation Technology (CAT):
libxl: error: libxl_psr.c:96:libxl__psr_cat_log_err_msg: CAT is not
enabled on the socket: No such file or directory
Failed to get cat info


I looked into the xen/arch/x86/psr.c and found that the function
cat_cpu_init() just returned without initializing the variable
"cat_socket_enable".

Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
cat_cpu_init().

OK. I understand that  the cpuid info shows that the CPU does not
support CAT. However, according to the table at
http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
 Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.

I'm not sure which part is incorrect: the hardware or the software?
(Hope Chao could give some insight about this.)

Best regards,

Meng



-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: Question about the CAT and CMT in Xen

[Andrew Cooper]

On 31/08/15 21:09, Meng Xu wrote:
> Hi Andrew and Chao,
>
> 2015-08-31 14:25 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>> On 31/08/15 18:42, Meng Xu wrote:
>>
>> Hi Chao,
>>
>> I'm Meng Xu from the University of Pennsylvania.
>>
>> We purchased a computer that has Intel(R) Xeon(R) CPU E5-2618L v3 @ 2.30GHz inside.
>> According to http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html, Intel E5-2618L v3 should support both CAT and CMT.
>>
>> I'm playing the latest Xen on the machine to evaluate how CAT and CMT work
>> .
>>
>>
>> The commit point I used is in the staging branch, commit 7b99717f62caeac08eea224a177cd28f047ac4b5, which was push on Aug. 17th, 2015.
>> I checked that the branch has merged your code.
>>
>> After I boot up the system, I tried the following commands and it shows some errors.
>>
>> ***
>> My question is
>> ***
>> :
>> 1) Is there any specific BIOS configuration I need to configure?
>> 2) The CMT is not enabled by default. Do I need to issue any command to enable it?  I didn't find any document about how to enable it, according to http://xenbits.xen.org/docs/unstable-staging/misc/xl-psr.html. :-(
>> 3) I'm not 100% sure if this machine we purchased supports the CAT. Right now, it shows "No such device" on the machine.
>> But it may probably because the hypervisor forgot to initialize it?
>>
>>
>> Below is the details of what I tried:
>>
>> # xl psr-hwinfo
>> Cache Monitoring Technology (CMT):
>> Enabled         : 0
>> Cache Allocation Technology (CAT):
>> libxl: error: libxl_psr.c:96:libxl__psr_cat_log_err_msg: CAT is not supported in this system: No such device
>> Failed to get cat info
>>
>> #xl psr-cmt-attach 0
>> libxl: error: libxl_psr.c:70:libxl__psr_cmt_log_err_msg: CMT is not supported in this system: No such device
>>
>> Please let me know if you need any further information to confirm if the machine support the CAT or if it's because there is some issue with the software layer. :-)
>>
>> Thank you very much for your time and help in this question!
>>
>> Best regards,
>>
>> Meng
>>
>>
>> To get started, use "psr=cat,cmt" on the Xen command line.  Neither are enabled by default as they can potentially allocate quite a lot of resources behind the scenes.
>
> Thank you very much for your suggestion! Yes, after adding the
> psr=cat:1,cmt:1, the CMT works. Command "
> xl psr-hwinfo" shows the total RMID as 31. It means CMT is enabled/working.
>
> However, the CAT is still not enabled on the socket. It have the same
> error message:
> Cache Allocation Technology (CAT):
> libxl: error: libxl_psr.c:96:libxl__psr_cat_log_err_msg: CAT is not
> enabled on the socket: No such file or directory
> Failed to get cat info
>
>
> I looked into the xen/arch/x86/psr.c and found that the function
> cat_cpu_init() just returned without initializing the variable
> "cat_socket_enable".
>
> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
> cat_cpu_init().
>
> OK. I understand that  the cpuid info shows that the CPU does not
> support CAT. However, according to the table at
> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
>  Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.
>
> I'm not sure which part is incorrect: the hardware or the software?
> (Hope Chao could give some insight about this.)
>
> Best regards,

IIRC, CAT support required a firmware update on the Broadwell hardware
XenServer now has.

See if there is a BIOS update available from your vendor.

~Andrew

Re: Question about the CAT and CMT in Xen

[Chao Peng]

On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
> I looked into the xen/arch/x86/psr.c and found that the function
> cat_cpu_init() just returned without initializing the variable
> "cat_socket_enable".
> 
> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
> cat_cpu_init().
> 
> OK. I understand that  the cpuid info shows that the CPU does not
> support CAT. However, according to the table at
> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
>  Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.
> 
> I'm not sure which part is incorrect: the hardware or the software?
> (Hope Chao could give some insight about this.)
> 

Hmmm, from cpuid info it looks like this model does not support CAT. I'm
not sure which microarchitecture it is. But if it is broadwell, hardware
should support that, what you need is some fireware update, just as Andrew
has said. But otherwise it is haswell, then the hardware probably doesn't
support that. In that case the above link probably is also wrong. I can check
that internally.

Chao

Re: Question about the CAT and CMT in Xen

[Meng Xu]

2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
>> I looked into the xen/arch/x86/psr.c and found that the function
>> cat_cpu_init() just returned without initializing the variable
>> "cat_socket_enable".
>>
>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
>> cat_cpu_init().
>>
>> OK. I understand that  the cpuid info shows that the CPU does not
>> support CAT. However, according to the table at
>> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
>>  Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.
>>
>> I'm not sure which part is incorrect: the hardware or the software?
>> (Hope Chao could give some insight about this.)
>>
>
> Hmmm, from cpuid info it looks like this model does not support CAT. I'm
> not sure which microarchitecture it is.

According to http://www.cpu-world.com/CPUs/Xeon/Intel-Xeon%20E5-2618L%20v3.html,
2618L v3 is Haswell. :-(

> But if it is broadwell, hardware
> should support that, what you need is some fireware update, just as Andrew
> has said. But otherwise it is haswell, then the hardware probably doesn't
> support that. In that case the above link probably is also wrong. I can check
> that internally.

Ah, could you please let me know if the link[1] is wrong?  If it does
not support CAT, I'm screwed. :'-(

[1]  http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html

>
> Chao



-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: Question about the CAT and CMT in Xen

[Andrew Cooper]

On 01/09/15 13:55, Meng Xu wrote:
> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
>>> I looked into the xen/arch/x86/psr.c and found that the function
>>> cat_cpu_init() just returned without initializing the variable
>>> "cat_socket_enable".
>>>
>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
>>> cat_cpu_init().
>>>
>>> OK. I understand that  the cpuid info shows that the CPU does not
>>> support CAT. However, according to the table at
>>> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
>>>  Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.
>>>
>>> I'm not sure which part is incorrect: the hardware or the software?
>>> (Hope Chao could give some insight about this.)
>>>
>> Hmmm, from cpuid info it looks like this model does not support CAT. I'm
>> not sure which microarchitecture it is.
> According to http://www.cpu-world.com/CPUs/Xeon/Intel-Xeon%20E5-2618L%20v3.html,
> 2618L v3 is Haswell. :-(

Wikipedia agrees.

Haswell only has plain L3 cache usage information.

It is Broadwell which adds memory bandwidth monitoring, and Cache
Allocation.

~Andrew

Re: Question about the CAT and CMT in Xen

[Meng Xu]

2015-09-01 9:04 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
> On 01/09/15 13:55, Meng Xu wrote:
>> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
>>>> I looked into the xen/arch/x86/psr.c and found that the function
>>>> cat_cpu_init() just returned without initializing the variable
>>>> "cat_socket_enable".
>>>>
>>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
>>>> cat_cpu_init().
>>>>
>>>> OK. I understand that  the cpuid info shows that the CPU does not
>>>> support CAT. However, according to the table at
>>>> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
>>>>  Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.
>>>>
>>>> I'm not sure which part is incorrect: the hardware or the software?
>>>> (Hope Chao could give some insight about this.)
>>>>
>>> Hmmm, from cpuid info it looks like this model does not support CAT. I'm
>>> not sure which microarchitecture it is.
>> According to http://www.cpu-world.com/CPUs/Xeon/Intel-Xeon%20E5-2618L%20v3.html,
>> 2618L v3 is Haswell. :-(
>
> Wikipedia agrees.
>
> Haswell only has plain L3 cache usage information.
>
> It is Broadwell which adds memory bandwidth monitoring, and Cache
> Allocation.

Ah, then the Intel's website [1] that lists the CAT-enabled processor
is quite misleading. I checked several processors (2618Lv3, 2608Lv3,
2658v3) listed  in [1] that is claimed to have CAT, and all of them
belong to Haswell... I think "v3" actually represents Haswell.

Andrew, is it possible to know which processor XenServer is currently
using that is confirmed to have the CAT mechanism?

Thank you very much!

Best,

[1] http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html

Meng

-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: Question about the CAT and CMT in Xen

[Andrew Cooper]

On 01/09/15 15:20, Meng Xu wrote:
> 2015-09-01 9:04 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>> On 01/09/15 13:55, Meng Xu wrote:
>>> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
>>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
>>>>> I looked into the xen/arch/x86/psr.c and found that the function
>>>>> cat_cpu_init() just returned without initializing the variable
>>>>> "cat_socket_enable".
>>>>>
>>>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
>>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
>>>>> cat_cpu_init().
>>>>>
>>>>> OK. I understand that  the cpuid info shows that the CPU does not
>>>>> support CAT. However, according to the table at
>>>>> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
>>>>>  Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.
>>>>>
>>>>> I'm not sure which part is incorrect: the hardware or the software?
>>>>> (Hope Chao could give some insight about this.)
>>>>>
>>>> Hmmm, from cpuid info it looks like this model does not support CAT. I'm
>>>> not sure which microarchitecture it is.
>>> According to http://www.cpu-world.com/CPUs/Xeon/Intel-Xeon%20E5-2618L%20v3.html,
>>> 2618L v3 is Haswell. :-(
>> Wikipedia agrees.
>>
>> Haswell only has plain L3 cache usage information.
>>
>> It is Broadwell which adds memory bandwidth monitoring, and Cache
>> Allocation.
> Ah, then the Intel's website [1] that lists the CAT-enabled processor
> is quite misleading. I checked several processors (2618Lv3, 2608Lv3,
> 2658v3) listed  in [1] that is claimed to have CAT, and all of them
> belong to Haswell... I think "v3" actually represents Haswell.
>
> Andrew, is it possible to know which processor XenServer is currently
> using that is confirmed to have the CAT mechanism?

I cant be much help, unfortunately,

>From /proc/cpuinfo
processor    : 14
vendor_id    : GenuineIntel
cpu family    : 6
model        : 79
model name    : Genuine Intel(R) CPU 0000 @ 2.20GHz

This is a 2-socket Broadwell-EP SDP and is probably a firmware revision
out of date, but I can confirm that I have had a bit of a play with CMT
and CAT on it.

~Andrew

Re: Question about the CAT and CMT in Xen

[Meng Xu]

2015-09-01 10:30 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
> On 01/09/15 15:20, Meng Xu wrote:
>> 2015-09-01 9:04 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>>> On 01/09/15 13:55, Meng Xu wrote:
>>>> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
>>>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
>>>>>> I looked into the xen/arch/x86/psr.c and found that the function
>>>>>> cat_cpu_init() just returned without initializing the variable
>>>>>> "cat_socket_enable".
>>>>>>
>>>>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
>>>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
>>>>>> cat_cpu_init().
>>>>>>
>>>>>> OK. I understand that  the cpuid info shows that the CPU does not
>>>>>> support CAT. However, according to the table at
>>>>>> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
>>>>>>  Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.
>>>>>>
>>>>>> I'm not sure which part is incorrect: the hardware or the software?
>>>>>> (Hope Chao could give some insight about this.)
>>>>>>
>>>>> Hmmm, from cpuid info it looks like this model does not support CAT. I'm
>>>>> not sure which microarchitecture it is.
>>>> According to http://www.cpu-world.com/CPUs/Xeon/Intel-Xeon%20E5-2618L%20v3.html,
>>>> 2618L v3 is Haswell. :-(
>>> Wikipedia agrees.
>>>
>>> Haswell only has plain L3 cache usage information.
>>>
>>> It is Broadwell which adds memory bandwidth monitoring, and Cache
>>> Allocation.
>> Ah, then the Intel's website [1] that lists the CAT-enabled processor
>> is quite misleading. I checked several processors (2618Lv3, 2608Lv3,
>> 2658v3) listed  in [1] that is claimed to have CAT, and all of them
>> belong to Haswell... I think "v3" actually represents Haswell.
>>
>> Andrew, is it possible to know which processor XenServer is currently
>> using that is confirmed to have the CAT mechanism?
>
> I cant be much help, unfortunately,

Thank you very much for your trying to help! :-)

Hope Chao could give more information about which Intel processors
support the CAT.

Best,

Meng



-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: Question about the CAT and CMT in Xen

[Chao Peng]

On Tue, Sep 01, 2015 at 10:42:57AM -0400, Meng Xu wrote:
> 2015-09-01 10:30 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
> > On 01/09/15 15:20, Meng Xu wrote:
> >> 2015-09-01 9:04 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
> >>> On 01/09/15 13:55, Meng Xu wrote:
> >>>> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
> >>>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
> >>>>>> I looked into the xen/arch/x86/psr.c and found that the function
> >>>>>> cat_cpu_init() just returned without initializing the variable
> >>>>>> "cat_socket_enable".
> >>>>>>
> >>>>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
> >>>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
> >>>>>> cat_cpu_init().
> >>>>>>
> >>>>>> OK. I understand that  the cpuid info shows that the CPU does not
> >>>>>> support CAT. However, according to the table at
> >>>>>> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
> >>>>>>  Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.
> >>>>>>
> >>>>>> I'm not sure which part is incorrect: the hardware or the software?
> >>>>>> (Hope Chao could give some insight about this.)
> >>>>>>
> >>>>> Hmmm, from cpuid info it looks like this model does not support CAT. I'm
> >>>>> not sure which microarchitecture it is.
> >>>> According to http://www.cpu-world.com/CPUs/Xeon/Intel-Xeon%20E5-2618L%20v3.html,
> >>>> 2618L v3 is Haswell. :-(
> >>> Wikipedia agrees.
> >>>
> >>> Haswell only has plain L3 cache usage information.
> >>>
> >>> It is Broadwell which adds memory bandwidth monitoring, and Cache
> >>> Allocation.
> >> Ah, then the Intel's website [1] that lists the CAT-enabled processor
> >> is quite misleading. I checked several processors (2618Lv3, 2608Lv3,
> >> 2658v3) listed  in [1] that is claimed to have CAT, and all of them
> >> belong to Haswell... I think "v3" actually represents Haswell.
> >>
> >> Andrew, is it possible to know which processor XenServer is currently
> >> using that is confirmed to have the CAT mechanism?
> >
> > I cant be much help, unfortunately,
> 
> Thank you very much for your trying to help! :-)
> 
> Hope Chao could give more information about which Intel processors
> support the CAT.

Until now what I can tell you is: All Broadwell Server support CAT and some
models of Haswell Server may also support it.

And the good news is the information @ Intel's website [1] is accurate
so your purchased "E5-2618L v3" should have it. But you definitely need
to ask your vendor for the firmware update.

Chao

Re: Question about the CAT and CMT in Xen

[Meng Xu]

Hi Andrew and Chao,

[Important things go first] It turns out my machine (Intel E5-2618L
v3) does have CAT capability!
Xen gives the false alarm that my machine does not have it. This
should be a bug, IMO. :-)

2015-09-01 10:42 GMT-04:00 Meng Xu <xumengpanda@gmail.com>:
> 2015-09-01 10:30 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>> On 01/09/15 15:20, Meng Xu wrote:
>>> 2015-09-01 9:04 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>>>> On 01/09/15 13:55, Meng Xu wrote:
>>>>> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
>>>>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
>>>>>>> I looked into the xen/arch/x86/psr.c and found that the function
>>>>>>> cat_cpu_init() just returned without initializing the variable
>>>>>>> "cat_socket_enable".
>>>>>>>
>>>>>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
>>>>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
>>>>>>> cat_cpu_init().

I'm thinking this check could be wrong for Intel E5-2618L v3. It
should work on Chao's machine but not on mine. There should be a
better way to check this probably. :-)

---

I used another way to check the CAT capability, as suggested by Priya
(cc.ed) from Intel.
I did the following steps as Priya suggested:
1. Download msr-tools utility on your linux distribution to perform
msr read write operations./ if you already have it installed modprobe
msr
2. rdmsr 0xc91
which returns 0xfffff
3. wrmsr -p 1 0xc91 0xf
which does not return anything
4. wrmsr -p 1 0xc8f 0x100000000
which does not return anything
5. rdmsr 0xc91
which returns 0xf

This shows that the CPU does have the MSRs that are used for CAT.

I also run the CAT tools on Linux provided by Intel, which can be
downloaded at https://01.org/packet-processing/cache-monitoring-technology-memory-bandwidth-monitoring-cache-allocation-technology.
It shows me that I can set up different cache partitions for different COS.
Basically, the pre-configured cache setting provided by the CAT tools
work perfectly on my machine. :-)

------
OK. That just check the registers are there and tools do not return
error. It may still not work, right? :-)
Well, I also did some performance evaluation by running a small simple
benchmark I wrote.

The benchmark task sequentially access a 6MB array;

I run the benchmark on core 0 in the following scenarios:

Scenario 1): Core 0 is allocated for 8MB cache with CAT, the latency
of accessing the 6MB array is around 5.5M cycles;
Scenario 2): Core 0 is allocated for 4MB cache with CAT, the latency
of accessing the 6MB array is around 16.9M cycles.
The slowdown in scenario 2) is 16.9M / 5.5M ~=3x.

------ISSUES-------
I tried to run some noisy neighbors on another core to see how good
the LLC isolation CAT can provide, but found some *weird result*.
I run the benchmark task on core 0 and the noisy neighbor that access
20MB array on core 1;
These two cores are configured to have *different* cache areas: core 0
has  8MB cache, core 1 has 4MB cache;
These two cores are in two isolated cpuset. No other tasks runs on
these two cores.
If I run the benchmark alone, the latency is around 5.5M cycles;
but if run the benchmark along with the noisy neighbor, the latency
*decreases* to 4.9M cycles.

I double checked that the Turbo Boost is disabled by checking the MSR
value with the following command:
    rdmsr -pi 0x1a0 -f 38:38
    1=disabled
    0=enabled
    it returns 1.
I also disabled the cache prefetch in BIOS.

Now I'm very confused. How come the latency decreases when a noisy
neighbor is running. It seems that the noisy neighbor may help some
hardware/software prefetcher to prefetch the data for the benchmark.
But right now, I couldn't think out any other prefetchers that may
cause this...
The benchmark and the noisy neighbor are independent and don't share
the array data.

If you have any insight or comment, could you please let me know?

Thanks and best regards,

Meng

-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: Question about the CAT and CMT in Xen

[Meng Xu]

2015-09-01 21:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
> On Tue, Sep 01, 2015 at 10:42:57AM -0400, Meng Xu wrote:
>> 2015-09-01 10:30 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>> > On 01/09/15 15:20, Meng Xu wrote:
>> >> 2015-09-01 9:04 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>> >>> On 01/09/15 13:55, Meng Xu wrote:
>> >>>> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
>> >>>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
>> >>>>>> I looked into the xen/arch/x86/psr.c and found that the function
>> >>>>>> cat_cpu_init() just returned without initializing the variable
>> >>>>>> "cat_socket_enable".
>> >>>>>>
>> >>>>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
>> >>>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
>> >>>>>> cat_cpu_init().
>> >>>>>>
>> >>>>>> OK. I understand that  the cpuid info shows that the CPU does not
>> >>>>>> support CAT. However, according to the table at
>> >>>>>> http://www.intel.com/content/www/us/en/communications/cache-monitoring-cache-allocation-technologies.html,
>> >>>>>>  Intel(R) Xeon(R) CPU E5-2618L v3 should support CAT.
>> >>>>>>
>> >>>>>> I'm not sure which part is incorrect: the hardware or the software?
>> >>>>>> (Hope Chao could give some insight about this.)
>> >>>>>>
>> >>>>> Hmmm, from cpuid info it looks like this model does not support CAT. I'm
>> >>>>> not sure which microarchitecture it is.
>> >>>> According to http://www.cpu-world.com/CPUs/Xeon/Intel-Xeon%20E5-2618L%20v3.html,
>> >>>> 2618L v3 is Haswell. :-(
>> >>> Wikipedia agrees.
>> >>>
>> >>> Haswell only has plain L3 cache usage information.
>> >>>
>> >>> It is Broadwell which adds memory bandwidth monitoring, and Cache
>> >>> Allocation.
>> >> Ah, then the Intel's website [1] that lists the CAT-enabled processor
>> >> is quite misleading. I checked several processors (2618Lv3, 2608Lv3,
>> >> 2658v3) listed  in [1] that is claimed to have CAT, and all of them
>> >> belong to Haswell... I think "v3" actually represents Haswell.
>> >>
>> >> Andrew, is it possible to know which processor XenServer is currently
>> >> using that is confirmed to have the CAT mechanism?
>> >
>> > I cant be much help, unfortunately,
>>
>> Thank you very much for your trying to help! :-)
>>
>> Hope Chao could give more information about which Intel processors
>> support the CAT.
>
> Until now what I can tell you is: All Broadwell Server support CAT and some
> models of Haswell Server may also support it.
>
> And the good news is the information @ Intel's website [1] is accurate
> so your purchased "E5-2618L v3" should have it. But you definitely need
> to ask your vendor for the firmware update.

Ah, I didn't do the firmware update and it seems working on Linux with
MSR tools and the CAT tool provided on 01.org.

Now I suspect it is because of some bug in the CAT code of Xen. ;-)

I just sent another email to explain why I think the tool on Linux works. :-P

Thanks,

Meng

-- 


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: Question about the CAT and CMT in Xen

[Chao Peng]

On Tue, Sep 01, 2015 at 09:51:57PM -0400, Meng Xu wrote:
> Hi Andrew and Chao,
> 
> [Important things go first] It turns out my machine (Intel E5-2618L
> v3) does have CAT capability!
> Xen gives the false alarm that my machine does not have it. This
> should be a bug, IMO. :-)

Even some Haswell Servers do support CAT, but it's usually
model-specific and the feature is not enumerated in a standard way as
listed in Intel SDM. Xen now however only takes care of the standard
enumeration so your case is not detected.

This could be done by adding cpu model check in Xen code or even use
updated firmware, which is I prefered.

> 
> 2015-09-01 10:42 GMT-04:00 Meng Xu <xumengpanda@gmail.com>:
> > 2015-09-01 10:30 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
> >> On 01/09/15 15:20, Meng Xu wrote:
> >>> 2015-09-01 9:04 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
> >>>> On 01/09/15 13:55, Meng Xu wrote:
> >>>>> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
> >>>>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
> >>>>>>> I looked into the xen/arch/x86/psr.c and found that the function
> >>>>>>> cat_cpu_init() just returned without initializing the variable
> >>>>>>> "cat_socket_enable".
> >>>>>>>
> >>>>>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
> >>>>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
> >>>>>>> cat_cpu_init().
> 
> I'm thinking this check could be wrong for Intel E5-2618L v3. It
> should work on Chao's machine but not on mine. There should be a
> better way to check this probably. :-)
> 
> ---
> 
> I used another way to check the CAT capability, as suggested by Priya
> (cc.ed) from Intel.
> I did the following steps as Priya suggested:
> 1. Download msr-tools utility on your linux distribution to perform
> msr read write operations./ if you already have it installed modprobe
> msr
> 2. rdmsr 0xc91
> which returns 0xfffff
> 3. wrmsr -p 1 0xc91 0xf
> which does not return anything
> 4. wrmsr -p 1 0xc8f 0x100000000
> which does not return anything
> 5. rdmsr 0xc91
> which returns 0xf
> 
> This shows that the CPU does have the MSRs that are used for CAT.
> 
> I also run the CAT tools on Linux provided by Intel, which can be
> downloaded at https://01.org/packet-processing/cache-monitoring-technology-memory-bandwidth-monitoring-cache-allocation-technology.
> It shows me that I can set up different cache partitions for different COS.
> Basically, the pre-configured cache setting provided by the CAT tools
> work perfectly on my machine. :-)
> 
> ------
> OK. That just check the registers are there and tools do not return
> error. It may still not work, right? :-)
> Well, I also did some performance evaluation by running a small simple
> benchmark I wrote.
> 
> The benchmark task sequentially access a 6MB array;
> 
> I run the benchmark on core 0 in the following scenarios:
> 
> Scenario 1): Core 0 is allocated for 8MB cache with CAT, the latency
> of accessing the 6MB array is around 5.5M cycles;
> Scenario 2): Core 0 is allocated for 4MB cache with CAT, the latency
> of accessing the 6MB array is around 16.9M cycles.
> The slowdown in scenario 2) is 16.9M / 5.5M ~=3x.
> 
> ------ISSUES-------
> I tried to run some noisy neighbors on another core to see how good
> the LLC isolation CAT can provide, but found some *weird result*.
> I run the benchmark task on core 0 and the noisy neighbor that access
> 20MB array on core 1;
> These two cores are configured to have *different* cache areas: core 0
> has  8MB cache, core 1 has 4MB cache;
> These two cores are in two isolated cpuset. No other tasks runs on
> these two cores.
> If I run the benchmark alone, the latency is around 5.5M cycles;
> but if run the benchmark along with the noisy neighbor, the latency
> *decreases* to 4.9M cycles.
> 
> I double checked that the Turbo Boost is disabled by checking the MSR
> value with the following command:
>     rdmsr -pi 0x1a0 -f 38:38
>     1=disabled
>     0=enabled
>     it returns 1.
> I also disabled the cache prefetch in BIOS.
> 
> Now I'm very confused. How come the latency decreases when a noisy
> neighbor is running. It seems that the noisy neighbor may help some
> hardware/software prefetcher to prefetch the data for the benchmark.
> But right now, I couldn't think out any other prefetchers that may
> cause this...
> The benchmark and the noisy neighbor are independent and don't share
> the array data.

Did you reboot the machine between your two tests and are the two cores
you used in the same socket?

Thanks,
Chao

Re: Question about the CAT and CMT in Xen

[Meng Xu]

2015-09-01 22:31 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
> On Tue, Sep 01, 2015 at 09:51:57PM -0400, Meng Xu wrote:
>> Hi Andrew and Chao,
>>
>> [Important things go first] It turns out my machine (Intel E5-2618L
>> v3) does have CAT capability!
>> Xen gives the false alarm that my machine does not have it. This
>> should be a bug, IMO. :-)
>
> Even some Haswell Servers do support CAT, but it's usually
> model-specific and the feature is not enumerated in a standard way as
> listed in Intel SDM. Xen now however only takes care of the standard
> enumeration so your case is not detected.

Ah. I prefer not to upgrade the firmware... Last time when I upgrade
the firmware, I screw up the whole system. :-(

Is it possible to check if CAT is supported on a machine with another
way? Right now, I think you are checking some specific bits returned
by CPUID.

>
> This could be done by adding cpu model check in Xen code or even use
> updated firmware, which is I prefered.
>
>>
>> 2015-09-01 10:42 GMT-04:00 Meng Xu <xumengpanda@gmail.com>:
>> > 2015-09-01 10:30 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>> >> On 01/09/15 15:20, Meng Xu wrote:
>> >>> 2015-09-01 9:04 GMT-04:00 Andrew Cooper <andrew.cooper3@citrix.com>:
>> >>>> On 01/09/15 13:55, Meng Xu wrote:
>> >>>>> 2015-09-01 1:47 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
>> >>>>>> On Mon, Aug 31, 2015 at 04:09:31PM -0400, Meng Xu wrote:
>> >>>>>>> I looked into the xen/arch/x86/psr.c and found that the function
>> >>>>>>> cat_cpu_init() just returned without initializing the variable
>> >>>>>>> "cat_socket_enable".
>> >>>>>>>
>> >>>>>>> Both  !cpu_has(c, X86_FEATURE_CAT) and c->cpuid_level <
>> >>>>>>> PSR_CPUID_LEVEL_CAT are evaluated as 1 inside the function
>> >>>>>>> cat_cpu_init().
>>
>> I'm thinking this check could be wrong for Intel E5-2618L v3. It
>> should work on Chao's machine but not on mine. There should be a
>> better way to check this probably. :-)
>>
>> ---
>>
>> I used another way to check the CAT capability, as suggested by Priya
>> (cc.ed) from Intel.
>> I did the following steps as Priya suggested:
>> 1. Download msr-tools utility on your linux distribution to perform
>> msr read write operations./ if you already have it installed modprobe
>> msr
>> 2. rdmsr 0xc91
>> which returns 0xfffff
>> 3. wrmsr -p 1 0xc91 0xf
>> which does not return anything
>> 4. wrmsr -p 1 0xc8f 0x100000000
>> which does not return anything
>> 5. rdmsr 0xc91
>> which returns 0xf
>>
>> This shows that the CPU does have the MSRs that are used for CAT.
>>
>> I also run the CAT tools on Linux provided by Intel, which can be
>> downloaded at https://01.org/packet-processing/cache-monitoring-technology-memory-bandwidth-monitoring-cache-allocation-technology.
>> It shows me that I can set up different cache partitions for different COS.
>> Basically, the pre-configured cache setting provided by the CAT tools
>> work perfectly on my machine. :-)
>>
>> ------
>> OK. That just check the registers are there and tools do not return
>> error. It may still not work, right? :-)
>> Well, I also did some performance evaluation by running a small simple
>> benchmark I wrote.
>>
>> The benchmark task sequentially access a 6MB array;
>>
>> I run the benchmark on core 0 in the following scenarios:
>>
>> Scenario 1): Core 0 is allocated for 8MB cache with CAT, the latency
>> of accessing the 6MB array is around 5.5M cycles;
>> Scenario 2): Core 0 is allocated for 4MB cache with CAT, the latency
>> of accessing the 6MB array is around 16.9M cycles.
>> The slowdown in scenario 2) is 16.9M / 5.5M ~=3x.
>>
>> ------ISSUES-------
>> I tried to run some noisy neighbors on another core to see how good
>> the LLC isolation CAT can provide, but found some *weird result*.
>> I run the benchmark task on core 0 and the noisy neighbor that access
>> 20MB array on core 1;
>> These two cores are configured to have *different* cache areas: core 0
>> has  8MB cache, core 1 has 4MB cache;
>> These two cores are in two isolated cpuset. No other tasks runs on
>> these two cores.
>> If I run the benchmark alone, the latency is around 5.5M cycles;
>> but if run the benchmark along with the noisy neighbor, the latency
>> *decreases* to 4.9M cycles.
>>
>> I double checked that the Turbo Boost is disabled by checking the MSR
>> value with the following command:
>>     rdmsr -pi 0x1a0 -f 38:38
>>     1=disabled
>>     0=enabled
>>     it returns 1.
>> I also disabled the cache prefetch in BIOS.
>>
>> Now I'm very confused. How come the latency decreases when a noisy
>> neighbor is running. It seems that the noisy neighbor may help some
>> hardware/software prefetcher to prefetch the data for the benchmark.
>> But right now, I couldn't think out any other prefetchers that may
>> cause this...
>> The benchmark and the noisy neighbor are independent and don't share
>> the array data.
>
> Did you reboot the machine between your two tests and are the two cores
> you used in the same socket?

I didn't reboot between these two test.

I'm using the same socket 0.

Thank you very much!

Meng


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: Question about the CAT and CMT in Xen

[Jan Beulich]

>>> Meng Xu <xumengpanda@gmail.com> 09/02/15 4:57 AM >>>
>2015-09-01 22:31 GMT-04:00 Chao Peng <chao.p.peng@linux.intel.com>:
>> On Tue, Sep 01, 2015 at 09:51:57PM -0400, Meng Xu wrote:
>>> Hi Andrew and Chao,
>>>
>>> [Important things go first] It turns out my machine (Intel E5-2618L
>>> v3) does have CAT capability!
>>> Xen gives the false alarm that my machine does not have it. This
>>> should be a bug, IMO. :-)
>>
>> Even some Haswell Servers do support CAT, but it's usually
>> model-specific and the feature is not enumerated in a standard way as
>> listed in Intel SDM. Xen now however only takes care of the standard
>> enumeration so your case is not detected.
>
>Ah. I prefer not to upgrade the firmware... Last time when I upgrade
>the firmware, I screw up the whole system. :-(

I would very much expect this to be an exception, not the norm.

>Is it possible to check if CAT is supported on a machine with another
>way? Right now, I think you are checking some specific bits returned
>by CPUID.

That's the one and only way it should be done in production software.
For your own debugging/development purposes you could of course
extend that check.

Jan