Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers

[Like Xu <like.xu@linux.intel.com>]

On 2021/4/15 10:49, Liuxiangdong wrote:
> 
> 
> On 2021/4/15 9:38, Xu, Like wrote:
>> On 2021/4/14 22:49, Liuxiangdong wrote:
>>> Hi Like,
>>>
>>> On 2021/4/9 16:46, Like Xu wrote:
>>>> Hi Liuxiangdong,
>>>>
>>>> On 2021/4/9 16:33, Liuxiangdong (Aven, Cloud Infrastructure Service 
>>>> Product Dept.) wrote:
>>>>> Do you have any comments or ideas about it ?
>>>>>
>>>>> https://lore.kernel.org/kvm/606E5EF6.2060402@huawei.com/
>>>>
>>>> My expectation is that there may be many fewer PEBS samples
>>>> on Skylake without any soft lockup.
>>>>
>>>> You may need to confirm the statement
>>>>
>>>> "All that matters is that the EPT pages don't get
>>>> unmapped ever while PEBS is active"
>>>>
>>>> is true in the kernel level.
>>>>
>>>> Try "-overcommit mem-lock=on" for your qemu.
>>>>
>>>
>>> Sorry, in fact, I don't quite understand
>>> "My expectation is that there may be many fewer PEBS samples on Skylake 
>>> without any soft lockup. "
>>
>> For testcase: perf record -e instructions:pp ./workload
>>
>> We can get 2242 samples on the ICX guest, but
>> only 17 samples or less on the Skylake guest.
>>
>> In my testcase on Skylake, neither the host nor the guest triggered the 
>> soft lock.
>>
> 
> Thanks for your explanation！
> Could you please show your complete qemu command and qemu version used on 
> Skylake?
> I hope I can test it again according to your qemu cmd and version.

A new version is released and you may have a try.

qemu command: "-enable-kvm -cpu host,migratable=no"
qemu base commit: db55d2c9239d445cb7f1fa8ede8e42bd339058f4
kvm base commit: f96be2deac9bca3ef5a2b0b66b71fcef8bad586d

diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 63c55f45ca92..727f55400eaf 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -5618,6 +5618,7 @@ __init int intel_pmu_init(void)
  	case INTEL_FAM6_KABYLAKE:
  	case INTEL_FAM6_COMETLAKE_L:
  	case INTEL_FAM6_COMETLAKE:
+		x86_pmu.pebs_vmx = 1;
  		x86_add_quirk(intel_pebs_isolation_quirk);
  		x86_pmu.late_ack = true;
  		memcpy(hw_cache_event_ids, skl_hw_cache_event_ids, 
sizeof(hw_cache_event_ids));
diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
index 100a749251b8..9e37e3dbe3ae 100644
--- a/arch/x86/kvm/pmu.c
+++ b/arch/x86/kvm/pmu.c
@@ -150,9 +150,8 @@ static void pmc_reprogram_counter(struct kvm_pmc *pmc, 
u32 type,
  		 * the accuracy of the PEBS profiling result, because the "event IP"
  		 * in the PEBS record is calibrated on the guest side.
  		 */
-		attr.precise_ip = 1;
-		if (x86_match_cpu(vmx_icl_pebs_cpu) && pmc->idx == 32)
-			attr.precise_ip = 3;
+		attr.precise_ip = x86_match_cpu(vmx_icl_pebs_cpu) ?
+			((pmc->idx == 32) ? 3 : 1) : ((pmc->idx == 1) ? 3 : 1);
  	}

  	event = perf_event_create_kernel_counter(&attr, -1, current,

> 
> 
>>>
>>> And, I have used "-overcommit mem-lock=on"  when soft lockup happens.
>>
>> I misunderstood the use of "mem-lock=on". It is not the same as the
>> guest mem pin and I believe more kernel patches are needed.
>>
>>>
>>>
>>> Now, I have tried to configure 1G-hugepages for 2G-mem vm. Each of guest 
>>> numa nodes has 1G mem.
>>> When I use pebs(perf record -e cycles:pp) in guest, there are successful 
>>> pebs samples just for a while and
>>> then I cannot get pebs samples. Host doesn't soft lockup in this process.
>>
>> In the worst case, no samples are expected.
>>
>>>
>>> Are there something wrong on skylake for we can only get a few samples? 
>>> IRQ?  Or using hugepage is not effecitve?
>>
>> The few samples comes from hardware limitation.
>> The Skylake doesn't have this "EPT-Friendly PEBS" capabilityand
>> some PEBS records will be lost when used by guests.
>>
>>>
>>> Thanks!
>>>
>>>>>
>>>>>
>>>>> On 2021/4/6 13:14, Xu, Like wrote:
>>>>>> Hi Xiangdong,
>>>>>>
>>>>>> On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service 
>>>>>> Product Dept.) wrote:
>>>>>>> Hi，like.
>>>>>>> Some questions about this new pebs patches set：
>>>>>>> https://lore.kernel.org/kvm/20210329054137.120994-2-like.xu@linux.intel.com/ 
>>>>>>>
>>>>>>>
>>>>>>> The new hardware facility supporting guest PEBS is only available
>>>>>>> on Intel Ice Lake Server platforms for now.
>>>>>>
>>>>>> Yes, we have documented this "EPT-friendly PEBS" capability in the SDM
>>>>>> 18.3.10.1 Processor Event Based Sampling (PEBS) Facility
>>>>>>
>>>>>> And again, this patch set doesn't officially support guest PEBS on 
>>>>>> the Skylake.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> AFAIK， Icelake supports adaptive PEBS and extended PEBS which 
>>>>>>> Skylake doesn't.
>>>>>>> But we can still use IA32_PEBS_ENABLE MSR to indicate 
>>>>>>> general-purpose counter in Skylake.
>>>>>>
>>>>>> For Skylake, only the PMC0-PMC3 are valid for PEBS and you may
>>>>>> mask the other unsupported bits in the pmu->pebs_enable_mask.
>>>>>>
>>>>>>> Is there anything else that only Icelake supports in this patches set?
>>>>>>
>>>>>> The PDIR counter on the Ice Lake is the fixed counter 0
>>>>>> while the PDIR counter on the Sky Lake is the gp counter 1.
>>>>>>
>>>>>> You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Besides, we have tried this patches set in Icelake.  We can use 
>>>>>>> pebs(eg: "perf record -e cycles:pp")
>>>>>>> when guest is kernel-5.11, but can't when kernel-4.18. Is there a 
>>>>>>> minimum guest kernel version requirement?
>>>>>>
>>>>>> The Ice Lake CPU model has been added since v5.4.
>>>>>>
>>>>>> You may double check whether the stable tree(s) code has
>>>>>> INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Xiangdong Liu
>>>>>>
>>>>>
>>>>
>>>
>>
>