From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76D1FC2B9F4 for ; Tue, 22 Jun 2021 09:38:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5F7BB61352 for ; Tue, 22 Jun 2021 09:38:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229663AbhFVJlD (ORCPT ); Tue, 22 Jun 2021 05:41:03 -0400 Received: from mga04.intel.com ([192.55.52.120]:56666 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229490AbhFVJlC (ORCPT ); Tue, 22 Jun 2021 05:41:02 -0400 IronPort-SDR: 7+6yYYodvtbKJ3DFVkb8RsJ0OPLjPAhwJrHmfC26R/+HjW8PNQu07T7wJiDakJuT04DL7vEt7w 9IzpNqIP/lOQ== X-IronPort-AV: E=McAfee;i="6200,9189,10022"; a="205195349" X-IronPort-AV: E=Sophos;i="5.83,291,1616482800"; d="scan'208";a="205195349" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2021 02:38:40 -0700 IronPort-SDR: YjghSkQQhfpqXchU5FmETDRBEKjg++NYbLVqNiZ3bX2P3p3aIdtsxEvlzAZ3khA29zw8feGTq5 hSRTxjz0q+eg== X-IronPort-AV: E=Sophos;i="5.83,291,1616482800"; d="scan'208";a="641599201" Received: from vmm_a4_icx.sh.intel.com (HELO localhost.localdomain) ([10.239.53.245]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2021 02:38:37 -0700 From: Zhu Lingshan To: lingshan.zhu@live.com Cc: Like Xu , Will Deacon , Marc Zyngier , Guo Ren , Nick Hu , Paul Walmsley , Boris Ostrovsky , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-csky@vger.kernel.org, linux-riscv@lists.infradead.org, xen-devel@lists.xenproject.org, Peter Zijlstra , Zhu Lingshan Subject: [PATCH V7 01/18] perf/core: Use static_call to optimize perf_guest_info_callbacks Date: Tue, 22 Jun 2021 17:38:06 +0800 Message-Id: <20210622093823.8215-2-lingshan.zhu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210622093823.8215-1-lingshan.zhu@intel.com> References: <20210622093823.8215-1-lingshan.zhu@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-csky@vger.kernel.org From: Like Xu For "struct perf_guest_info_callbacks", the two fields "is_in_guest" and "is_user_mode" are replaced with a new multiplexed member named "state", and the "get_guest_ip" field will be renamed to "get_ip". For arm64, xen and kvm/x86, the application of DEFINE_STATIC_CALL_RET0 could make all that perf_guest_cbs stuff suck less. For arm, csky, nds32, and riscv, just applied some renamed refactoring. Cc: Will Deacon Cc: Marc Zyngier Cc: Guo Ren Cc: Nick Hu Cc: Paul Walmsley Cc: Boris Ostrovsky Cc: linux-arm-kernel@lists.infradead.org Cc: kvmarm@lists.cs.columbia.edu Cc: linux-csky@vger.kernel.org Cc: linux-riscv@lists.infradead.org Cc: xen-devel@lists.xenproject.org Suggested-by: Peter Zijlstra (Intel) Original-by: Peter Zijlstra (Intel) Signed-off-by: Like Xu Signed-off-by: Zhu Lingshan --- arch/arm/kernel/perf_callchain.c | 16 ++++++++----- arch/arm64/kernel/perf_callchain.c | 29 ++++++++++++++++++----- arch/arm64/kvm/perf.c | 22 ++++++++--------- arch/csky/kernel/perf_callchain.c | 4 ++-- arch/nds32/kernel/perf_event_cpu.c | 16 ++++++++----- arch/riscv/kernel/perf_callchain.c | 4 ++-- arch/x86/events/core.c | 38 ++++++++++++++++++++++++------ arch/x86/events/intel/core.c | 7 +++--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/pmu.c | 2 +- arch/x86/kvm/x86.c | 37 ++++++++++++++++------------- arch/x86/xen/pmu.c | 33 +++++++++++--------------- include/linux/perf_event.h | 12 ++++++---- kernel/events/core.c | 9 +++++++ 14 files changed, 144 insertions(+), 87 deletions(-) diff --git a/arch/arm/kernel/perf_callchain.c b/arch/arm/kernel/perf_callchain.c index 3b69a76d341e..1ce30f86d6c7 100644 --- a/arch/arm/kernel/perf_callchain.c +++ b/arch/arm/kernel/perf_callchain.c @@ -64,7 +64,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs { struct frame_tail __user *tail; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -100,7 +100,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re { struct stackframe fr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -111,8 +111,8 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (perf_guest_cbs && perf_guest_cbs->state()) + return perf_guest_cbs->get_ip(); return instruction_pointer(regs); } @@ -120,9 +120,13 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int state = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (perf_guest_cbs) + state = perf_guest_cbs->state(); + + if (perf_guest_cbs && state) { + if (state & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/arm64/kernel/perf_callchain.c b/arch/arm64/kernel/perf_callchain.c index 88ff471b0bce..5df2bd5d12ba 100644 --- a/arch/arm64/kernel/perf_callchain.c +++ b/arch/arm64/kernel/perf_callchain.c @@ -5,6 +5,7 @@ * Copyright (C) 2015 ARM Limited */ #include +#include #include #include @@ -99,10 +100,25 @@ compat_user_backtrace(struct compat_frame_tail __user *tail, } #endif /* CONFIG_COMPAT */ +DEFINE_STATIC_CALL_RET0(arm64_guest_state, *(perf_guest_cbs->state)); +DEFINE_STATIC_CALL_RET0(arm64_guest_get_ip, *(perf_guest_cbs->get_ip)); + +void arch_perf_update_guest_cbs(void) +{ + static_call_update(arm64_guest_state, (void *)&__static_call_return0); + static_call_update(arm64_guest_get_ip, (void *)&__static_call_return0); + + if (perf_guest_cbs && perf_guest_cbs->state) + static_call_update(arm64_guest_state, perf_guest_cbs->state); + + if (perf_guest_cbs && perf_guest_cbs->get_ip) + static_call_update(arm64_guest_get_ip, perf_guest_cbs->get_ip); +} + void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(arm64_guest_state)()) { /* We don't support guest os callchain now */ return; } @@ -149,7 +165,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, { struct stackframe frame; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(arm64_guest_state)()) { /* We don't support guest os callchain now */ return; } @@ -160,8 +176,8 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (static_call(arm64_guest_state)()) + return static_call(arm64_guest_get_ip)(); return instruction_pointer(regs); } @@ -169,9 +185,10 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int guest = static_call(arm64_guest_state)(); - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (guest) { + if (guest & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/arm64/kvm/perf.c b/arch/arm64/kvm/perf.c index 151c31fb9860..8a3387e58f42 100644 --- a/arch/arm64/kvm/perf.c +++ b/arch/arm64/kvm/perf.c @@ -13,21 +13,20 @@ DEFINE_STATIC_KEY_FALSE(kvm_arm_pmu_available); -static int kvm_is_in_guest(void) -{ - return kvm_get_running_vcpu() != NULL; -} - -static int kvm_is_user_mode(void) +static unsigned int kvm_guest_state(void) { struct kvm_vcpu *vcpu; + unsigned int state = 0; + + if (kvm_get_running_vcpu()) + state |= PERF_GUEST_ACTIVE; vcpu = kvm_get_running_vcpu(); - if (vcpu) - return !vcpu_mode_priv(vcpu); + if (vcpu && !vcpu_mode_priv(vcpu)) + state |= PERF_GUEST_USER; - return 0; + return state; } static unsigned long kvm_get_guest_ip(void) @@ -43,9 +42,8 @@ static unsigned long kvm_get_guest_ip(void) } static struct perf_guest_info_callbacks kvm_guest_cbs = { - .is_in_guest = kvm_is_in_guest, - .is_user_mode = kvm_is_user_mode, - .get_guest_ip = kvm_get_guest_ip, + .state = kvm_guest_state, + .get_ip = kvm_get_guest_ip, }; int kvm_perf_init(void) diff --git a/arch/csky/kernel/perf_callchain.c b/arch/csky/kernel/perf_callchain.c index ab55e98ee8f6..3e42239dd1b2 100644 --- a/arch/csky/kernel/perf_callchain.c +++ b/arch/csky/kernel/perf_callchain.c @@ -89,7 +89,7 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry, unsigned long fp = 0; /* C-SKY does not support virtualization. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) + if (perf_guest_cbs && perf_guest_cbs->state()) return; fp = regs->regs[4]; @@ -113,7 +113,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct stackframe fr; /* C-SKY does not support virtualization. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { pr_warn("C-SKY does not support perf in guest mode!"); return; } diff --git a/arch/nds32/kernel/perf_event_cpu.c b/arch/nds32/kernel/perf_event_cpu.c index 0ce6f9f307e6..1dc32ba842ce 100644 --- a/arch/nds32/kernel/perf_event_cpu.c +++ b/arch/nds32/kernel/perf_event_cpu.c @@ -1371,7 +1371,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, leaf_fp = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -1481,7 +1481,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, { struct stackframe fr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -1494,8 +1494,8 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, unsigned long perf_instruction_pointer(struct pt_regs *regs) { /* However, NDS32 does not support virtualization */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (perf_guest_cbs && perf_guest_cbs->state()) + return perf_guest_cbs->get_ip(); return instruction_pointer(regs); } @@ -1503,10 +1503,14 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int state = 0; + + if (perf_guest_cbs) + state = perf_guest_cbs->state(); /* However, NDS32 does not support virtualization */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (perf_guest_cbs && state) { + if (state & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/riscv/kernel/perf_callchain.c b/arch/riscv/kernel/perf_callchain.c index 0bb1854dce83..ea63f70cae5d 100644 --- a/arch/riscv/kernel/perf_callchain.c +++ b/arch/riscv/kernel/perf_callchain.c @@ -59,7 +59,7 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry, unsigned long fp = 0; /* RISC-V does not support perf in guest mode. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) + if (perf_guest_cbs && perf_guest_cbs->state()) return; fp = regs->s0; @@ -79,7 +79,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) { /* RISC-V does not support perf in guest mode. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { pr_warn("RISC-V does not support perf in guest mode!"); return; } diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index 8f71dd72ef95..c71af4cfba9b 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -90,6 +90,27 @@ DEFINE_STATIC_CALL_NULL(x86_pmu_pebs_aliases, *x86_pmu.pebs_aliases); */ DEFINE_STATIC_CALL_RET0(x86_pmu_guest_get_msrs, *x86_pmu.guest_get_msrs); +DEFINE_STATIC_CALL_RET0(x86_guest_state, *(perf_guest_cbs->state)); +DEFINE_STATIC_CALL_RET0(x86_guest_get_ip, *(perf_guest_cbs->get_ip)); +DEFINE_STATIC_CALL_RET0(x86_guest_handle_intel_pt_intr, *(perf_guest_cbs->handle_intel_pt_intr)); + +void arch_perf_update_guest_cbs(void) +{ + static_call_update(x86_guest_state, (void *)&__static_call_return0); + static_call_update(x86_guest_get_ip, (void *)&__static_call_return0); + static_call_update(x86_guest_handle_intel_pt_intr, (void *)&__static_call_return0); + + if (perf_guest_cbs && perf_guest_cbs->state) + static_call_update(x86_guest_state, perf_guest_cbs->state); + + if (perf_guest_cbs && perf_guest_cbs->get_ip) + static_call_update(x86_guest_get_ip, perf_guest_cbs->get_ip); + + if (perf_guest_cbs && perf_guest_cbs->handle_intel_pt_intr) + static_call_update(x86_guest_handle_intel_pt_intr, + perf_guest_cbs->handle_intel_pt_intr); +} + u64 __read_mostly hw_cache_event_ids [PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_OP_MAX] @@ -2738,7 +2759,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re struct unwind_state state; unsigned long addr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(x86_guest_state)()) { /* TODO: We don't support guest os callchain now */ return; } @@ -2841,7 +2862,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs struct stack_frame frame; const struct stack_frame __user *fp; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(x86_guest_state)()) { /* TODO: We don't support guest os callchain now */ return; } @@ -2918,18 +2939,21 @@ static unsigned long code_segment_base(struct pt_regs *regs) unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + unsigned long ip = static_call(x86_guest_get_ip)(); + + if (likely(!ip)) + ip = regs->ip + code_segment_base(regs); - return regs->ip + code_segment_base(regs); + return ip; } unsigned long perf_misc_flags(struct pt_regs *regs) { + unsigned int guest = static_call(x86_guest_state)(); int misc = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (guest) { + if (guest & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index e28892270c58..430f5743f3ca 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2780,6 +2780,8 @@ static void intel_pmu_reset(void) local_irq_restore(flags); } +DECLARE_STATIC_CALL(x86_guest_handle_intel_pt_intr, *(perf_guest_cbs->handle_intel_pt_intr)); + static int handle_pmi_common(struct pt_regs *regs, u64 status) { struct perf_sample_data data; @@ -2850,10 +2852,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) */ if (__test_and_clear_bit(GLOBAL_STATUS_TRACE_TOPAPMI_BIT, (unsigned long *)&status)) { handled++; - if (unlikely(perf_guest_cbs && perf_guest_cbs->is_in_guest() && - perf_guest_cbs->handle_intel_pt_intr)) - perf_guest_cbs->handle_intel_pt_intr(); - else + if (!static_call(x86_guest_handle_intel_pt_intr)()) intel_pt_interrupt(); } diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 9c7ced0e3171..f752fcf56d76 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1813,7 +1813,7 @@ int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu); int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err); void __kvm_request_immediate_exit(struct kvm_vcpu *vcpu); -int kvm_is_in_guest(void); +unsigned int kvm_guest_state(void); void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, u32 size); diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 827886c12c16..2dcbd1b30004 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -87,7 +87,7 @@ static void kvm_perf_overflow_intr(struct perf_event *perf_event, * woken up. So we should wake it, but this is impossible from * NMI context. Do it from irq work instead. */ - if (!kvm_is_in_guest()) + if (!kvm_guest_state()) irq_work_queue(&pmc_to_pmu(pmc)->irq_work); else kvm_make_request(KVM_REQ_PMI, pmc->vcpu); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b594275d49b5..9cb1c02d348c 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8035,44 +8035,47 @@ static void kvm_timer_init(void) DEFINE_PER_CPU(struct kvm_vcpu *, current_vcpu); EXPORT_PER_CPU_SYMBOL_GPL(current_vcpu); -int kvm_is_in_guest(void) +unsigned int kvm_guest_state(void) { - return __this_cpu_read(current_vcpu) != NULL; -} - -static int kvm_is_user_mode(void) -{ - int user_mode = 3; + struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); + unsigned int state = 0; - if (__this_cpu_read(current_vcpu)) - user_mode = static_call(kvm_x86_get_cpl)(__this_cpu_read(current_vcpu)); + if (vcpu) { + state |= PERF_GUEST_ACTIVE; + if (static_call(kvm_x86_get_cpl)(vcpu)) + state |= PERF_GUEST_USER; + } - return user_mode != 0; + return state; } -static unsigned long kvm_get_guest_ip(void) +static unsigned long kvm_guest_get_ip(void) { + struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); unsigned long ip = 0; - if (__this_cpu_read(current_vcpu)) - ip = kvm_rip_read(__this_cpu_read(current_vcpu)); + if (vcpu) + ip = kvm_rip_read(vcpu); return ip; } -static void kvm_handle_intel_pt_intr(void) +static unsigned int kvm_handle_intel_pt_intr(void) { struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); + if (!vcpu) + return 0; + kvm_make_request(KVM_REQ_PMI, vcpu); __set_bit(MSR_CORE_PERF_GLOBAL_OVF_CTRL_TRACE_TOPA_PMI_BIT, (unsigned long *)&vcpu->arch.pmu.global_status); + return 1; } static struct perf_guest_info_callbacks kvm_guest_cbs = { - .is_in_guest = kvm_is_in_guest, - .is_user_mode = kvm_is_user_mode, - .get_guest_ip = kvm_get_guest_ip, + .state = kvm_guest_state, + .get_ip = kvm_guest_get_ip, .handle_intel_pt_intr = kvm_handle_intel_pt_intr, }; diff --git a/arch/x86/xen/pmu.c b/arch/x86/xen/pmu.c index e13b0b49fcdf..7352cf002b87 100644 --- a/arch/x86/xen/pmu.c +++ b/arch/x86/xen/pmu.c @@ -413,34 +413,30 @@ int pmu_apic_update(uint32_t val) } /* perf callbacks */ -static int xen_is_in_guest(void) +static unsigned int xen_guest_state(void) { const struct xen_pmu_data *xenpmu_data = get_xenpmu_data(); + unsigned int state = 0; if (!xenpmu_data) { pr_warn_once("%s: pmudata not initialized\n", __func__); - return 0; + return state; } if (!xen_initial_domain() || (xenpmu_data->domain_id >= DOMID_SELF)) - return 0; - - return 1; -} + return state; -static int xen_is_user_mode(void) -{ - const struct xen_pmu_data *xenpmu_data = get_xenpmu_data(); + state |= PERF_GUEST_ACTIVE; - if (!xenpmu_data) { - pr_warn_once("%s: pmudata not initialized\n", __func__); - return 0; + if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_PV) { + if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_USER) + state |= PERF_GUEST_USER; + } else { + if (!!(xenpmu_data->pmu.r.regs.cpl & 3)) + state |= PERF_GUEST_USER; } - if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_PV) - return (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_USER); - else - return !!(xenpmu_data->pmu.r.regs.cpl & 3); + return state; } static unsigned long xen_get_guest_ip(void) @@ -456,9 +452,8 @@ static unsigned long xen_get_guest_ip(void) } static struct perf_guest_info_callbacks xen_guest_cbs = { - .is_in_guest = xen_is_in_guest, - .is_user_mode = xen_is_user_mode, - .get_guest_ip = xen_get_guest_ip, + .state = xen_guest_state, + .get_ip = xen_get_guest_ip, }; /* Convert registers from Xen's format to Linux' */ diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index f5a6a2f069ed..8065e5f093f4 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -26,11 +26,13 @@ # include #endif +#define PERF_GUEST_ACTIVE 0x01 +#define PERF_GUEST_USER 0x02 + struct perf_guest_info_callbacks { - int (*is_in_guest)(void); - int (*is_user_mode)(void); - unsigned long (*get_guest_ip)(void); - void (*handle_intel_pt_intr)(void); + unsigned int (*state)(void); + unsigned long (*get_ip)(void); + unsigned int (*handle_intel_pt_intr)(void); }; #ifdef CONFIG_HAVE_HW_BREAKPOINT @@ -1237,6 +1239,8 @@ extern void perf_event_bpf_event(struct bpf_prog *prog, u16 flags); extern struct perf_guest_info_callbacks *perf_guest_cbs; +extern void __weak arch_perf_update_guest_cbs(void); + extern int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks); extern int perf_unregister_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks); diff --git a/kernel/events/core.c b/kernel/events/core.c index 6fee4a7e88d7..101fa7d0bfda 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6479,9 +6479,18 @@ static void perf_pending_event(struct irq_work *entry) */ struct perf_guest_info_callbacks *perf_guest_cbs; +/* explicitly use __weak to fix duplicate symbol error */ +void __weak arch_perf_update_guest_cbs(void) +{ +} + int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *cbs) { + if (WARN_ON_ONCE(perf_guest_cbs)) + return -EBUSY; + perf_guest_cbs = cbs; + arch_perf_update_guest_cbs(); return 0; } EXPORT_SYMBOL_GPL(perf_register_guest_info_callbacks); -- 2.27.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83440C2B9F4 for ; Tue, 22 Jun 2021 09:39:23 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4608A601FE for ; Tue, 22 Jun 2021 09:39:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4608A601FE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xzb65VPazQVZ9LmsmnEu2EAwOUmm1gFTFAG+VeiBuE0=; b=L6Sg9E6JR4YuiJ 9W7BNZ6ye0dHkt6dZFDwnueWbWRf/Nrbq1L/KCMcSLH9CZqoCpsYjkC1vDSyxKR2ITtaqI7OMyTvS rQK2O5t9EhaC3tE3OKjnnra/eadrAZggu3t01zNFCvS9KI/NmmUCraStm9e/crFe2uXDjGdELEIEq aI4jigmlmxmyisDsm8aWEzWD4OBfE0gUghypV65/wGxhdVeHQfjbn6xpTVFWdpkA/PsebeYPZmS1R zM4XM+JNiq9wdyDzj8jAP//iS8Wcut9sE3URepGDdaAefbvSzT5WIhH/9fNMhdruOrAyn62BD0Grz l/s/bG6YYjbjzHoP5rVQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lvcs4-006TPO-5u; Tue, 22 Jun 2021 09:39:04 +0000 Received: from mga09.intel.com ([134.134.136.24]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lvcrp-006TLd-87; Tue, 22 Jun 2021 09:38:52 +0000 IronPort-SDR: LpcK+MFjKVDBqMkuuZJmuE9dx42ha6A0fARAKuJPFjIfFCGu9sEemDNWVOmPiuTlUFc/hPTRaV yEOtgvgBMZXg== X-IronPort-AV: E=McAfee;i="6200,9189,10022"; a="206964820" X-IronPort-AV: E=Sophos;i="5.83,291,1616482800"; d="scan'208";a="206964820" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2021 02:38:40 -0700 IronPort-SDR: YjghSkQQhfpqXchU5FmETDRBEKjg++NYbLVqNiZ3bX2P3p3aIdtsxEvlzAZ3khA29zw8feGTq5 hSRTxjz0q+eg== X-IronPort-AV: E=Sophos;i="5.83,291,1616482800"; d="scan'208";a="641599201" Received: from vmm_a4_icx.sh.intel.com (HELO localhost.localdomain) ([10.239.53.245]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2021 02:38:37 -0700 From: Zhu Lingshan To: lingshan.zhu@live.com Cc: Like Xu , Will Deacon , Marc Zyngier , Guo Ren , Nick Hu , Paul Walmsley , Boris Ostrovsky , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-csky@vger.kernel.org, linux-riscv@lists.infradead.org, xen-devel@lists.xenproject.org, Peter Zijlstra , Zhu Lingshan Subject: [PATCH V7 01/18] perf/core: Use static_call to optimize perf_guest_info_callbacks Date: Tue, 22 Jun 2021 17:38:06 +0800 Message-Id: <20210622093823.8215-2-lingshan.zhu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210622093823.8215-1-lingshan.zhu@intel.com> References: <20210622093823.8215-1-lingshan.zhu@intel.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210622_023849_504043_52BC8817 X-CRM114-Status: GOOD ( 19.11 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Like Xu For "struct perf_guest_info_callbacks", the two fields "is_in_guest" and "is_user_mode" are replaced with a new multiplexed member named "state", and the "get_guest_ip" field will be renamed to "get_ip". For arm64, xen and kvm/x86, the application of DEFINE_STATIC_CALL_RET0 could make all that perf_guest_cbs stuff suck less. For arm, csky, nds32, and riscv, just applied some renamed refactoring. Cc: Will Deacon Cc: Marc Zyngier Cc: Guo Ren Cc: Nick Hu Cc: Paul Walmsley Cc: Boris Ostrovsky Cc: linux-arm-kernel@lists.infradead.org Cc: kvmarm@lists.cs.columbia.edu Cc: linux-csky@vger.kernel.org Cc: linux-riscv@lists.infradead.org Cc: xen-devel@lists.xenproject.org Suggested-by: Peter Zijlstra (Intel) Original-by: Peter Zijlstra (Intel) Signed-off-by: Like Xu Signed-off-by: Zhu Lingshan --- arch/arm/kernel/perf_callchain.c | 16 ++++++++----- arch/arm64/kernel/perf_callchain.c | 29 ++++++++++++++++++----- arch/arm64/kvm/perf.c | 22 ++++++++--------- arch/csky/kernel/perf_callchain.c | 4 ++-- arch/nds32/kernel/perf_event_cpu.c | 16 ++++++++----- arch/riscv/kernel/perf_callchain.c | 4 ++-- arch/x86/events/core.c | 38 ++++++++++++++++++++++++------ arch/x86/events/intel/core.c | 7 +++--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/pmu.c | 2 +- arch/x86/kvm/x86.c | 37 ++++++++++++++++------------- arch/x86/xen/pmu.c | 33 +++++++++++--------------- include/linux/perf_event.h | 12 ++++++---- kernel/events/core.c | 9 +++++++ 14 files changed, 144 insertions(+), 87 deletions(-) diff --git a/arch/arm/kernel/perf_callchain.c b/arch/arm/kernel/perf_callchain.c index 3b69a76d341e..1ce30f86d6c7 100644 --- a/arch/arm/kernel/perf_callchain.c +++ b/arch/arm/kernel/perf_callchain.c @@ -64,7 +64,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs { struct frame_tail __user *tail; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -100,7 +100,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re { struct stackframe fr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -111,8 +111,8 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (perf_guest_cbs && perf_guest_cbs->state()) + return perf_guest_cbs->get_ip(); return instruction_pointer(regs); } @@ -120,9 +120,13 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int state = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (perf_guest_cbs) + state = perf_guest_cbs->state(); + + if (perf_guest_cbs && state) { + if (state & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/arm64/kernel/perf_callchain.c b/arch/arm64/kernel/perf_callchain.c index 88ff471b0bce..5df2bd5d12ba 100644 --- a/arch/arm64/kernel/perf_callchain.c +++ b/arch/arm64/kernel/perf_callchain.c @@ -5,6 +5,7 @@ * Copyright (C) 2015 ARM Limited */ #include +#include #include #include @@ -99,10 +100,25 @@ compat_user_backtrace(struct compat_frame_tail __user *tail, } #endif /* CONFIG_COMPAT */ +DEFINE_STATIC_CALL_RET0(arm64_guest_state, *(perf_guest_cbs->state)); +DEFINE_STATIC_CALL_RET0(arm64_guest_get_ip, *(perf_guest_cbs->get_ip)); + +void arch_perf_update_guest_cbs(void) +{ + static_call_update(arm64_guest_state, (void *)&__static_call_return0); + static_call_update(arm64_guest_get_ip, (void *)&__static_call_return0); + + if (perf_guest_cbs && perf_guest_cbs->state) + static_call_update(arm64_guest_state, perf_guest_cbs->state); + + if (perf_guest_cbs && perf_guest_cbs->get_ip) + static_call_update(arm64_guest_get_ip, perf_guest_cbs->get_ip); +} + void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(arm64_guest_state)()) { /* We don't support guest os callchain now */ return; } @@ -149,7 +165,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, { struct stackframe frame; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(arm64_guest_state)()) { /* We don't support guest os callchain now */ return; } @@ -160,8 +176,8 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (static_call(arm64_guest_state)()) + return static_call(arm64_guest_get_ip)(); return instruction_pointer(regs); } @@ -169,9 +185,10 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int guest = static_call(arm64_guest_state)(); - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (guest) { + if (guest & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/arm64/kvm/perf.c b/arch/arm64/kvm/perf.c index 151c31fb9860..8a3387e58f42 100644 --- a/arch/arm64/kvm/perf.c +++ b/arch/arm64/kvm/perf.c @@ -13,21 +13,20 @@ DEFINE_STATIC_KEY_FALSE(kvm_arm_pmu_available); -static int kvm_is_in_guest(void) -{ - return kvm_get_running_vcpu() != NULL; -} - -static int kvm_is_user_mode(void) +static unsigned int kvm_guest_state(void) { struct kvm_vcpu *vcpu; + unsigned int state = 0; + + if (kvm_get_running_vcpu()) + state |= PERF_GUEST_ACTIVE; vcpu = kvm_get_running_vcpu(); - if (vcpu) - return !vcpu_mode_priv(vcpu); + if (vcpu && !vcpu_mode_priv(vcpu)) + state |= PERF_GUEST_USER; - return 0; + return state; } static unsigned long kvm_get_guest_ip(void) @@ -43,9 +42,8 @@ static unsigned long kvm_get_guest_ip(void) } static struct perf_guest_info_callbacks kvm_guest_cbs = { - .is_in_guest = kvm_is_in_guest, - .is_user_mode = kvm_is_user_mode, - .get_guest_ip = kvm_get_guest_ip, + .state = kvm_guest_state, + .get_ip = kvm_get_guest_ip, }; int kvm_perf_init(void) diff --git a/arch/csky/kernel/perf_callchain.c b/arch/csky/kernel/perf_callchain.c index ab55e98ee8f6..3e42239dd1b2 100644 --- a/arch/csky/kernel/perf_callchain.c +++ b/arch/csky/kernel/perf_callchain.c @@ -89,7 +89,7 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry, unsigned long fp = 0; /* C-SKY does not support virtualization. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) + if (perf_guest_cbs && perf_guest_cbs->state()) return; fp = regs->regs[4]; @@ -113,7 +113,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct stackframe fr; /* C-SKY does not support virtualization. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { pr_warn("C-SKY does not support perf in guest mode!"); return; } diff --git a/arch/nds32/kernel/perf_event_cpu.c b/arch/nds32/kernel/perf_event_cpu.c index 0ce6f9f307e6..1dc32ba842ce 100644 --- a/arch/nds32/kernel/perf_event_cpu.c +++ b/arch/nds32/kernel/perf_event_cpu.c @@ -1371,7 +1371,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, leaf_fp = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -1481,7 +1481,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, { struct stackframe fr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -1494,8 +1494,8 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, unsigned long perf_instruction_pointer(struct pt_regs *regs) { /* However, NDS32 does not support virtualization */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (perf_guest_cbs && perf_guest_cbs->state()) + return perf_guest_cbs->get_ip(); return instruction_pointer(regs); } @@ -1503,10 +1503,14 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int state = 0; + + if (perf_guest_cbs) + state = perf_guest_cbs->state(); /* However, NDS32 does not support virtualization */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (perf_guest_cbs && state) { + if (state & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/riscv/kernel/perf_callchain.c b/arch/riscv/kernel/perf_callchain.c index 0bb1854dce83..ea63f70cae5d 100644 --- a/arch/riscv/kernel/perf_callchain.c +++ b/arch/riscv/kernel/perf_callchain.c @@ -59,7 +59,7 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry, unsigned long fp = 0; /* RISC-V does not support perf in guest mode. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) + if (perf_guest_cbs && perf_guest_cbs->state()) return; fp = regs->s0; @@ -79,7 +79,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) { /* RISC-V does not support perf in guest mode. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { pr_warn("RISC-V does not support perf in guest mode!"); return; } diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index 8f71dd72ef95..c71af4cfba9b 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -90,6 +90,27 @@ DEFINE_STATIC_CALL_NULL(x86_pmu_pebs_aliases, *x86_pmu.pebs_aliases); */ DEFINE_STATIC_CALL_RET0(x86_pmu_guest_get_msrs, *x86_pmu.guest_get_msrs); +DEFINE_STATIC_CALL_RET0(x86_guest_state, *(perf_guest_cbs->state)); +DEFINE_STATIC_CALL_RET0(x86_guest_get_ip, *(perf_guest_cbs->get_ip)); +DEFINE_STATIC_CALL_RET0(x86_guest_handle_intel_pt_intr, *(perf_guest_cbs->handle_intel_pt_intr)); + +void arch_perf_update_guest_cbs(void) +{ + static_call_update(x86_guest_state, (void *)&__static_call_return0); + static_call_update(x86_guest_get_ip, (void *)&__static_call_return0); + static_call_update(x86_guest_handle_intel_pt_intr, (void *)&__static_call_return0); + + if (perf_guest_cbs && perf_guest_cbs->state) + static_call_update(x86_guest_state, perf_guest_cbs->state); + + if (perf_guest_cbs && perf_guest_cbs->get_ip) + static_call_update(x86_guest_get_ip, perf_guest_cbs->get_ip); + + if (perf_guest_cbs && perf_guest_cbs->handle_intel_pt_intr) + static_call_update(x86_guest_handle_intel_pt_intr, + perf_guest_cbs->handle_intel_pt_intr); +} + u64 __read_mostly hw_cache_event_ids [PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_OP_MAX] @@ -2738,7 +2759,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re struct unwind_state state; unsigned long addr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(x86_guest_state)()) { /* TODO: We don't support guest os callchain now */ return; } @@ -2841,7 +2862,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs struct stack_frame frame; const struct stack_frame __user *fp; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(x86_guest_state)()) { /* TODO: We don't support guest os callchain now */ return; } @@ -2918,18 +2939,21 @@ static unsigned long code_segment_base(struct pt_regs *regs) unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + unsigned long ip = static_call(x86_guest_get_ip)(); + + if (likely(!ip)) + ip = regs->ip + code_segment_base(regs); - return regs->ip + code_segment_base(regs); + return ip; } unsigned long perf_misc_flags(struct pt_regs *regs) { + unsigned int guest = static_call(x86_guest_state)(); int misc = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (guest) { + if (guest & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index e28892270c58..430f5743f3ca 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2780,6 +2780,8 @@ static void intel_pmu_reset(void) local_irq_restore(flags); } +DECLARE_STATIC_CALL(x86_guest_handle_intel_pt_intr, *(perf_guest_cbs->handle_intel_pt_intr)); + static int handle_pmi_common(struct pt_regs *regs, u64 status) { struct perf_sample_data data; @@ -2850,10 +2852,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) */ if (__test_and_clear_bit(GLOBAL_STATUS_TRACE_TOPAPMI_BIT, (unsigned long *)&status)) { handled++; - if (unlikely(perf_guest_cbs && perf_guest_cbs->is_in_guest() && - perf_guest_cbs->handle_intel_pt_intr)) - perf_guest_cbs->handle_intel_pt_intr(); - else + if (!static_call(x86_guest_handle_intel_pt_intr)()) intel_pt_interrupt(); } diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 9c7ced0e3171..f752fcf56d76 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1813,7 +1813,7 @@ int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu); int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err); void __kvm_request_immediate_exit(struct kvm_vcpu *vcpu); -int kvm_is_in_guest(void); +unsigned int kvm_guest_state(void); void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, u32 size); diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 827886c12c16..2dcbd1b30004 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -87,7 +87,7 @@ static void kvm_perf_overflow_intr(struct perf_event *perf_event, * woken up. So we should wake it, but this is impossible from * NMI context. Do it from irq work instead. */ - if (!kvm_is_in_guest()) + if (!kvm_guest_state()) irq_work_queue(&pmc_to_pmu(pmc)->irq_work); else kvm_make_request(KVM_REQ_PMI, pmc->vcpu); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b594275d49b5..9cb1c02d348c 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8035,44 +8035,47 @@ static void kvm_timer_init(void) DEFINE_PER_CPU(struct kvm_vcpu *, current_vcpu); EXPORT_PER_CPU_SYMBOL_GPL(current_vcpu); -int kvm_is_in_guest(void) +unsigned int kvm_guest_state(void) { - return __this_cpu_read(current_vcpu) != NULL; -} - -static int kvm_is_user_mode(void) -{ - int user_mode = 3; + struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); + unsigned int state = 0; - if (__this_cpu_read(current_vcpu)) - user_mode = static_call(kvm_x86_get_cpl)(__this_cpu_read(current_vcpu)); + if (vcpu) { + state |= PERF_GUEST_ACTIVE; + if (static_call(kvm_x86_get_cpl)(vcpu)) + state |= PERF_GUEST_USER; + } - return user_mode != 0; + return state; } -static unsigned long kvm_get_guest_ip(void) +static unsigned long kvm_guest_get_ip(void) { + struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); unsigned long ip = 0; - if (__this_cpu_read(current_vcpu)) - ip = kvm_rip_read(__this_cpu_read(current_vcpu)); + if (vcpu) + ip = kvm_rip_read(vcpu); return ip; } -static void kvm_handle_intel_pt_intr(void) +static unsigned int kvm_handle_intel_pt_intr(void) { struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); + if (!vcpu) + return 0; + kvm_make_request(KVM_REQ_PMI, vcpu); __set_bit(MSR_CORE_PERF_GLOBAL_OVF_CTRL_TRACE_TOPA_PMI_BIT, (unsigned long *)&vcpu->arch.pmu.global_status); + return 1; } static struct perf_guest_info_callbacks kvm_guest_cbs = { - .is_in_guest = kvm_is_in_guest, - .is_user_mode = kvm_is_user_mode, - .get_guest_ip = kvm_get_guest_ip, + .state = kvm_guest_state, + .get_ip = kvm_guest_get_ip, .handle_intel_pt_intr = kvm_handle_intel_pt_intr, }; diff --git a/arch/x86/xen/pmu.c b/arch/x86/xen/pmu.c index e13b0b49fcdf..7352cf002b87 100644 --- a/arch/x86/xen/pmu.c +++ b/arch/x86/xen/pmu.c @@ -413,34 +413,30 @@ int pmu_apic_update(uint32_t val) } /* perf callbacks */ -static int xen_is_in_guest(void) +static unsigned int xen_guest_state(void) { const struct xen_pmu_data *xenpmu_data = get_xenpmu_data(); + unsigned int state = 0; if (!xenpmu_data) { pr_warn_once("%s: pmudata not initialized\n", __func__); - return 0; + return state; } if (!xen_initial_domain() || (xenpmu_data->domain_id >= DOMID_SELF)) - return 0; - - return 1; -} + return state; -static int xen_is_user_mode(void) -{ - const struct xen_pmu_data *xenpmu_data = get_xenpmu_data(); + state |= PERF_GUEST_ACTIVE; - if (!xenpmu_data) { - pr_warn_once("%s: pmudata not initialized\n", __func__); - return 0; + if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_PV) { + if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_USER) + state |= PERF_GUEST_USER; + } else { + if (!!(xenpmu_data->pmu.r.regs.cpl & 3)) + state |= PERF_GUEST_USER; } - if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_PV) - return (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_USER); - else - return !!(xenpmu_data->pmu.r.regs.cpl & 3); + return state; } static unsigned long xen_get_guest_ip(void) @@ -456,9 +452,8 @@ static unsigned long xen_get_guest_ip(void) } static struct perf_guest_info_callbacks xen_guest_cbs = { - .is_in_guest = xen_is_in_guest, - .is_user_mode = xen_is_user_mode, - .get_guest_ip = xen_get_guest_ip, + .state = xen_guest_state, + .get_ip = xen_get_guest_ip, }; /* Convert registers from Xen's format to Linux' */ diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index f5a6a2f069ed..8065e5f093f4 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -26,11 +26,13 @@ # include #endif +#define PERF_GUEST_ACTIVE 0x01 +#define PERF_GUEST_USER 0x02 + struct perf_guest_info_callbacks { - int (*is_in_guest)(void); - int (*is_user_mode)(void); - unsigned long (*get_guest_ip)(void); - void (*handle_intel_pt_intr)(void); + unsigned int (*state)(void); + unsigned long (*get_ip)(void); + unsigned int (*handle_intel_pt_intr)(void); }; #ifdef CONFIG_HAVE_HW_BREAKPOINT @@ -1237,6 +1239,8 @@ extern void perf_event_bpf_event(struct bpf_prog *prog, u16 flags); extern struct perf_guest_info_callbacks *perf_guest_cbs; +extern void __weak arch_perf_update_guest_cbs(void); + extern int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks); extern int perf_unregister_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks); diff --git a/kernel/events/core.c b/kernel/events/core.c index 6fee4a7e88d7..101fa7d0bfda 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6479,9 +6479,18 @@ static void perf_pending_event(struct irq_work *entry) */ struct perf_guest_info_callbacks *perf_guest_cbs; +/* explicitly use __weak to fix duplicate symbol error */ +void __weak arch_perf_update_guest_cbs(void) +{ +} + int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *cbs) { + if (WARN_ON_ONCE(perf_guest_cbs)) + return -EBUSY; + perf_guest_cbs = cbs; + arch_perf_update_guest_cbs(); return 0; } EXPORT_SYMBOL_GPL(perf_register_guest_info_callbacks); -- 2.27.0 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0223BC49EA4 for ; Tue, 22 Jun 2021 15:20:51 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 71C11613B4 for ; Tue, 22 Jun 2021 15:20:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 71C11613B4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id D14F6407ED; Tue, 22 Jun 2021 11:20:50 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g15bXfdfXFMU; Tue, 22 Jun 2021 11:20:47 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 602674A173; Tue, 22 Jun 2021 11:20:47 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id CA6D3405A6 for ; Tue, 22 Jun 2021 05:38:51 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xWmLIgvdGypK for ; Tue, 22 Jun 2021 05:38:49 -0400 (EDT) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 50F8B40573 for ; Tue, 22 Jun 2021 05:38:49 -0400 (EDT) IronPort-SDR: PUoDmaAr4yAM7BSh/6fRWB/jxMQVkZQbQ3ymN35HBvwUekdlO2L9gr/2aEzH7z3tBMsN0wjwRF QCD8dWU8y0jA== X-IronPort-AV: E=McAfee;i="6200,9189,10022"; a="187402088" X-IronPort-AV: E=Sophos;i="5.83,291,1616482800"; d="scan'208";a="187402088" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2021 02:38:40 -0700 IronPort-SDR: YjghSkQQhfpqXchU5FmETDRBEKjg++NYbLVqNiZ3bX2P3p3aIdtsxEvlzAZ3khA29zw8feGTq5 hSRTxjz0q+eg== X-IronPort-AV: E=Sophos;i="5.83,291,1616482800"; d="scan'208";a="641599201" Received: from vmm_a4_icx.sh.intel.com (HELO localhost.localdomain) ([10.239.53.245]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2021 02:38:37 -0700 From: Zhu Lingshan To: lingshan.zhu@live.com Subject: [PATCH V7 01/18] perf/core: Use static_call to optimize perf_guest_info_callbacks Date: Tue, 22 Jun 2021 17:38:06 +0800 Message-Id: <20210622093823.8215-2-lingshan.zhu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210622093823.8215-1-lingshan.zhu@intel.com> References: <20210622093823.8215-1-lingshan.zhu@intel.com> MIME-Version: 1.0 X-Mailman-Approved-At: Tue, 22 Jun 2021 11:20:46 -0400 Cc: Zhu Lingshan , Like Xu , Nick Hu , Peter Zijlstra , Marc Zyngier , linux-csky@vger.kernel.org, linux-riscv@lists.infradead.org, Guo Ren , Paul Walmsley , xen-devel@lists.xenproject.org, Boris Ostrovsky , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu From: Like Xu For "struct perf_guest_info_callbacks", the two fields "is_in_guest" and "is_user_mode" are replaced with a new multiplexed member named "state", and the "get_guest_ip" field will be renamed to "get_ip". For arm64, xen and kvm/x86, the application of DEFINE_STATIC_CALL_RET0 could make all that perf_guest_cbs stuff suck less. For arm, csky, nds32, and riscv, just applied some renamed refactoring. Cc: Will Deacon Cc: Marc Zyngier Cc: Guo Ren Cc: Nick Hu Cc: Paul Walmsley Cc: Boris Ostrovsky Cc: linux-arm-kernel@lists.infradead.org Cc: kvmarm@lists.cs.columbia.edu Cc: linux-csky@vger.kernel.org Cc: linux-riscv@lists.infradead.org Cc: xen-devel@lists.xenproject.org Suggested-by: Peter Zijlstra (Intel) Original-by: Peter Zijlstra (Intel) Signed-off-by: Like Xu Signed-off-by: Zhu Lingshan --- arch/arm/kernel/perf_callchain.c | 16 ++++++++----- arch/arm64/kernel/perf_callchain.c | 29 ++++++++++++++++++----- arch/arm64/kvm/perf.c | 22 ++++++++--------- arch/csky/kernel/perf_callchain.c | 4 ++-- arch/nds32/kernel/perf_event_cpu.c | 16 ++++++++----- arch/riscv/kernel/perf_callchain.c | 4 ++-- arch/x86/events/core.c | 38 ++++++++++++++++++++++++------ arch/x86/events/intel/core.c | 7 +++--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/pmu.c | 2 +- arch/x86/kvm/x86.c | 37 ++++++++++++++++------------- arch/x86/xen/pmu.c | 33 +++++++++++--------------- include/linux/perf_event.h | 12 ++++++---- kernel/events/core.c | 9 +++++++ 14 files changed, 144 insertions(+), 87 deletions(-) diff --git a/arch/arm/kernel/perf_callchain.c b/arch/arm/kernel/perf_callchain.c index 3b69a76d341e..1ce30f86d6c7 100644 --- a/arch/arm/kernel/perf_callchain.c +++ b/arch/arm/kernel/perf_callchain.c @@ -64,7 +64,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs { struct frame_tail __user *tail; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -100,7 +100,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re { struct stackframe fr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -111,8 +111,8 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (perf_guest_cbs && perf_guest_cbs->state()) + return perf_guest_cbs->get_ip(); return instruction_pointer(regs); } @@ -120,9 +120,13 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int state = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (perf_guest_cbs) + state = perf_guest_cbs->state(); + + if (perf_guest_cbs && state) { + if (state & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/arm64/kernel/perf_callchain.c b/arch/arm64/kernel/perf_callchain.c index 88ff471b0bce..5df2bd5d12ba 100644 --- a/arch/arm64/kernel/perf_callchain.c +++ b/arch/arm64/kernel/perf_callchain.c @@ -5,6 +5,7 @@ * Copyright (C) 2015 ARM Limited */ #include +#include #include #include @@ -99,10 +100,25 @@ compat_user_backtrace(struct compat_frame_tail __user *tail, } #endif /* CONFIG_COMPAT */ +DEFINE_STATIC_CALL_RET0(arm64_guest_state, *(perf_guest_cbs->state)); +DEFINE_STATIC_CALL_RET0(arm64_guest_get_ip, *(perf_guest_cbs->get_ip)); + +void arch_perf_update_guest_cbs(void) +{ + static_call_update(arm64_guest_state, (void *)&__static_call_return0); + static_call_update(arm64_guest_get_ip, (void *)&__static_call_return0); + + if (perf_guest_cbs && perf_guest_cbs->state) + static_call_update(arm64_guest_state, perf_guest_cbs->state); + + if (perf_guest_cbs && perf_guest_cbs->get_ip) + static_call_update(arm64_guest_get_ip, perf_guest_cbs->get_ip); +} + void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(arm64_guest_state)()) { /* We don't support guest os callchain now */ return; } @@ -149,7 +165,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, { struct stackframe frame; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(arm64_guest_state)()) { /* We don't support guest os callchain now */ return; } @@ -160,8 +176,8 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (static_call(arm64_guest_state)()) + return static_call(arm64_guest_get_ip)(); return instruction_pointer(regs); } @@ -169,9 +185,10 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int guest = static_call(arm64_guest_state)(); - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (guest) { + if (guest & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/arm64/kvm/perf.c b/arch/arm64/kvm/perf.c index 151c31fb9860..8a3387e58f42 100644 --- a/arch/arm64/kvm/perf.c +++ b/arch/arm64/kvm/perf.c @@ -13,21 +13,20 @@ DEFINE_STATIC_KEY_FALSE(kvm_arm_pmu_available); -static int kvm_is_in_guest(void) -{ - return kvm_get_running_vcpu() != NULL; -} - -static int kvm_is_user_mode(void) +static unsigned int kvm_guest_state(void) { struct kvm_vcpu *vcpu; + unsigned int state = 0; + + if (kvm_get_running_vcpu()) + state |= PERF_GUEST_ACTIVE; vcpu = kvm_get_running_vcpu(); - if (vcpu) - return !vcpu_mode_priv(vcpu); + if (vcpu && !vcpu_mode_priv(vcpu)) + state |= PERF_GUEST_USER; - return 0; + return state; } static unsigned long kvm_get_guest_ip(void) @@ -43,9 +42,8 @@ static unsigned long kvm_get_guest_ip(void) } static struct perf_guest_info_callbacks kvm_guest_cbs = { - .is_in_guest = kvm_is_in_guest, - .is_user_mode = kvm_is_user_mode, - .get_guest_ip = kvm_get_guest_ip, + .state = kvm_guest_state, + .get_ip = kvm_get_guest_ip, }; int kvm_perf_init(void) diff --git a/arch/csky/kernel/perf_callchain.c b/arch/csky/kernel/perf_callchain.c index ab55e98ee8f6..3e42239dd1b2 100644 --- a/arch/csky/kernel/perf_callchain.c +++ b/arch/csky/kernel/perf_callchain.c @@ -89,7 +89,7 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry, unsigned long fp = 0; /* C-SKY does not support virtualization. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) + if (perf_guest_cbs && perf_guest_cbs->state()) return; fp = regs->regs[4]; @@ -113,7 +113,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct stackframe fr; /* C-SKY does not support virtualization. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { pr_warn("C-SKY does not support perf in guest mode!"); return; } diff --git a/arch/nds32/kernel/perf_event_cpu.c b/arch/nds32/kernel/perf_event_cpu.c index 0ce6f9f307e6..1dc32ba842ce 100644 --- a/arch/nds32/kernel/perf_event_cpu.c +++ b/arch/nds32/kernel/perf_event_cpu.c @@ -1371,7 +1371,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, leaf_fp = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -1481,7 +1481,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, { struct stackframe fr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -1494,8 +1494,8 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, unsigned long perf_instruction_pointer(struct pt_regs *regs) { /* However, NDS32 does not support virtualization */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (perf_guest_cbs && perf_guest_cbs->state()) + return perf_guest_cbs->get_ip(); return instruction_pointer(regs); } @@ -1503,10 +1503,14 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int state = 0; + + if (perf_guest_cbs) + state = perf_guest_cbs->state(); /* However, NDS32 does not support virtualization */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (perf_guest_cbs && state) { + if (state & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/riscv/kernel/perf_callchain.c b/arch/riscv/kernel/perf_callchain.c index 0bb1854dce83..ea63f70cae5d 100644 --- a/arch/riscv/kernel/perf_callchain.c +++ b/arch/riscv/kernel/perf_callchain.c @@ -59,7 +59,7 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry, unsigned long fp = 0; /* RISC-V does not support perf in guest mode. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) + if (perf_guest_cbs && perf_guest_cbs->state()) return; fp = regs->s0; @@ -79,7 +79,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) { /* RISC-V does not support perf in guest mode. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { pr_warn("RISC-V does not support perf in guest mode!"); return; } diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index 8f71dd72ef95..c71af4cfba9b 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -90,6 +90,27 @@ DEFINE_STATIC_CALL_NULL(x86_pmu_pebs_aliases, *x86_pmu.pebs_aliases); */ DEFINE_STATIC_CALL_RET0(x86_pmu_guest_get_msrs, *x86_pmu.guest_get_msrs); +DEFINE_STATIC_CALL_RET0(x86_guest_state, *(perf_guest_cbs->state)); +DEFINE_STATIC_CALL_RET0(x86_guest_get_ip, *(perf_guest_cbs->get_ip)); +DEFINE_STATIC_CALL_RET0(x86_guest_handle_intel_pt_intr, *(perf_guest_cbs->handle_intel_pt_intr)); + +void arch_perf_update_guest_cbs(void) +{ + static_call_update(x86_guest_state, (void *)&__static_call_return0); + static_call_update(x86_guest_get_ip, (void *)&__static_call_return0); + static_call_update(x86_guest_handle_intel_pt_intr, (void *)&__static_call_return0); + + if (perf_guest_cbs && perf_guest_cbs->state) + static_call_update(x86_guest_state, perf_guest_cbs->state); + + if (perf_guest_cbs && perf_guest_cbs->get_ip) + static_call_update(x86_guest_get_ip, perf_guest_cbs->get_ip); + + if (perf_guest_cbs && perf_guest_cbs->handle_intel_pt_intr) + static_call_update(x86_guest_handle_intel_pt_intr, + perf_guest_cbs->handle_intel_pt_intr); +} + u64 __read_mostly hw_cache_event_ids [PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_OP_MAX] @@ -2738,7 +2759,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re struct unwind_state state; unsigned long addr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(x86_guest_state)()) { /* TODO: We don't support guest os callchain now */ return; } @@ -2841,7 +2862,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs struct stack_frame frame; const struct stack_frame __user *fp; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(x86_guest_state)()) { /* TODO: We don't support guest os callchain now */ return; } @@ -2918,18 +2939,21 @@ static unsigned long code_segment_base(struct pt_regs *regs) unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + unsigned long ip = static_call(x86_guest_get_ip)(); + + if (likely(!ip)) + ip = regs->ip + code_segment_base(regs); - return regs->ip + code_segment_base(regs); + return ip; } unsigned long perf_misc_flags(struct pt_regs *regs) { + unsigned int guest = static_call(x86_guest_state)(); int misc = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (guest) { + if (guest & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index e28892270c58..430f5743f3ca 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2780,6 +2780,8 @@ static void intel_pmu_reset(void) local_irq_restore(flags); } +DECLARE_STATIC_CALL(x86_guest_handle_intel_pt_intr, *(perf_guest_cbs->handle_intel_pt_intr)); + static int handle_pmi_common(struct pt_regs *regs, u64 status) { struct perf_sample_data data; @@ -2850,10 +2852,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) */ if (__test_and_clear_bit(GLOBAL_STATUS_TRACE_TOPAPMI_BIT, (unsigned long *)&status)) { handled++; - if (unlikely(perf_guest_cbs && perf_guest_cbs->is_in_guest() && - perf_guest_cbs->handle_intel_pt_intr)) - perf_guest_cbs->handle_intel_pt_intr(); - else + if (!static_call(x86_guest_handle_intel_pt_intr)()) intel_pt_interrupt(); } diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 9c7ced0e3171..f752fcf56d76 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1813,7 +1813,7 @@ int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu); int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err); void __kvm_request_immediate_exit(struct kvm_vcpu *vcpu); -int kvm_is_in_guest(void); +unsigned int kvm_guest_state(void); void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, u32 size); diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 827886c12c16..2dcbd1b30004 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -87,7 +87,7 @@ static void kvm_perf_overflow_intr(struct perf_event *perf_event, * woken up. So we should wake it, but this is impossible from * NMI context. Do it from irq work instead. */ - if (!kvm_is_in_guest()) + if (!kvm_guest_state()) irq_work_queue(&pmc_to_pmu(pmc)->irq_work); else kvm_make_request(KVM_REQ_PMI, pmc->vcpu); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b594275d49b5..9cb1c02d348c 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8035,44 +8035,47 @@ static void kvm_timer_init(void) DEFINE_PER_CPU(struct kvm_vcpu *, current_vcpu); EXPORT_PER_CPU_SYMBOL_GPL(current_vcpu); -int kvm_is_in_guest(void) +unsigned int kvm_guest_state(void) { - return __this_cpu_read(current_vcpu) != NULL; -} - -static int kvm_is_user_mode(void) -{ - int user_mode = 3; + struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); + unsigned int state = 0; - if (__this_cpu_read(current_vcpu)) - user_mode = static_call(kvm_x86_get_cpl)(__this_cpu_read(current_vcpu)); + if (vcpu) { + state |= PERF_GUEST_ACTIVE; + if (static_call(kvm_x86_get_cpl)(vcpu)) + state |= PERF_GUEST_USER; + } - return user_mode != 0; + return state; } -static unsigned long kvm_get_guest_ip(void) +static unsigned long kvm_guest_get_ip(void) { + struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); unsigned long ip = 0; - if (__this_cpu_read(current_vcpu)) - ip = kvm_rip_read(__this_cpu_read(current_vcpu)); + if (vcpu) + ip = kvm_rip_read(vcpu); return ip; } -static void kvm_handle_intel_pt_intr(void) +static unsigned int kvm_handle_intel_pt_intr(void) { struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); + if (!vcpu) + return 0; + kvm_make_request(KVM_REQ_PMI, vcpu); __set_bit(MSR_CORE_PERF_GLOBAL_OVF_CTRL_TRACE_TOPA_PMI_BIT, (unsigned long *)&vcpu->arch.pmu.global_status); + return 1; } static struct perf_guest_info_callbacks kvm_guest_cbs = { - .is_in_guest = kvm_is_in_guest, - .is_user_mode = kvm_is_user_mode, - .get_guest_ip = kvm_get_guest_ip, + .state = kvm_guest_state, + .get_ip = kvm_guest_get_ip, .handle_intel_pt_intr = kvm_handle_intel_pt_intr, }; diff --git a/arch/x86/xen/pmu.c b/arch/x86/xen/pmu.c index e13b0b49fcdf..7352cf002b87 100644 --- a/arch/x86/xen/pmu.c +++ b/arch/x86/xen/pmu.c @@ -413,34 +413,30 @@ int pmu_apic_update(uint32_t val) } /* perf callbacks */ -static int xen_is_in_guest(void) +static unsigned int xen_guest_state(void) { const struct xen_pmu_data *xenpmu_data = get_xenpmu_data(); + unsigned int state = 0; if (!xenpmu_data) { pr_warn_once("%s: pmudata not initialized\n", __func__); - return 0; + return state; } if (!xen_initial_domain() || (xenpmu_data->domain_id >= DOMID_SELF)) - return 0; - - return 1; -} + return state; -static int xen_is_user_mode(void) -{ - const struct xen_pmu_data *xenpmu_data = get_xenpmu_data(); + state |= PERF_GUEST_ACTIVE; - if (!xenpmu_data) { - pr_warn_once("%s: pmudata not initialized\n", __func__); - return 0; + if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_PV) { + if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_USER) + state |= PERF_GUEST_USER; + } else { + if (!!(xenpmu_data->pmu.r.regs.cpl & 3)) + state |= PERF_GUEST_USER; } - if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_PV) - return (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_USER); - else - return !!(xenpmu_data->pmu.r.regs.cpl & 3); + return state; } static unsigned long xen_get_guest_ip(void) @@ -456,9 +452,8 @@ static unsigned long xen_get_guest_ip(void) } static struct perf_guest_info_callbacks xen_guest_cbs = { - .is_in_guest = xen_is_in_guest, - .is_user_mode = xen_is_user_mode, - .get_guest_ip = xen_get_guest_ip, + .state = xen_guest_state, + .get_ip = xen_get_guest_ip, }; /* Convert registers from Xen's format to Linux' */ diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index f5a6a2f069ed..8065e5f093f4 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -26,11 +26,13 @@ # include #endif +#define PERF_GUEST_ACTIVE 0x01 +#define PERF_GUEST_USER 0x02 + struct perf_guest_info_callbacks { - int (*is_in_guest)(void); - int (*is_user_mode)(void); - unsigned long (*get_guest_ip)(void); - void (*handle_intel_pt_intr)(void); + unsigned int (*state)(void); + unsigned long (*get_ip)(void); + unsigned int (*handle_intel_pt_intr)(void); }; #ifdef CONFIG_HAVE_HW_BREAKPOINT @@ -1237,6 +1239,8 @@ extern void perf_event_bpf_event(struct bpf_prog *prog, u16 flags); extern struct perf_guest_info_callbacks *perf_guest_cbs; +extern void __weak arch_perf_update_guest_cbs(void); + extern int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks); extern int perf_unregister_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks); diff --git a/kernel/events/core.c b/kernel/events/core.c index 6fee4a7e88d7..101fa7d0bfda 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6479,9 +6479,18 @@ static void perf_pending_event(struct irq_work *entry) */ struct perf_guest_info_callbacks *perf_guest_cbs; +/* explicitly use __weak to fix duplicate symbol error */ +void __weak arch_perf_update_guest_cbs(void) +{ +} + int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *cbs) { + if (WARN_ON_ONCE(perf_guest_cbs)) + return -EBUSY; + perf_guest_cbs = cbs; + arch_perf_update_guest_cbs(); return 0; } EXPORT_SYMBOL_GPL(perf_register_guest_info_callbacks); -- 2.27.0 _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B2A2C2B9F4 for ; Tue, 22 Jun 2021 09:40:39 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4C0C361353 for ; Tue, 22 Jun 2021 09:40:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4C0C361353 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xXTwKLVB68SxkwKPYjXL3DHxgdILXL75wQY3DRV4/wc=; b=4x6E4Ova4Johl+ tJLPYeHz9+At5MRXE5mq1N/d/VPrUOgc05Rr/p55PlM8/IM0Hk37GMPURhcSSn+gu1ijP5xH3Hmx/ Yc5CIXgxdjBid2CbX913Pko5DQkAxV1pSqUV8+dmI7VdVPrVVxNNMJbpsrgFCBumg8JpPVG+uVeQ3 pBdIQKoU4JJX01U4ixr2Yj2uxiBPQbNPJsod1KpzsFiBQ3sSrFC5sjBh2rS8+GQeTRvN4x1qVdZjj Py07ekSiOZy5NKSAr/UBiDg2Ue7YXv0zrIv9Z1RymExc5QwpMYpL30j3k3zqd3BlP0v8XrPnbE3T6 SEoIpU6rUXr+w0ctUDjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lvcru-006TMy-O4; Tue, 22 Jun 2021 09:38:54 +0000 Received: from mga09.intel.com ([134.134.136.24]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lvcrp-006TLd-87; Tue, 22 Jun 2021 09:38:52 +0000 IronPort-SDR: LpcK+MFjKVDBqMkuuZJmuE9dx42ha6A0fARAKuJPFjIfFCGu9sEemDNWVOmPiuTlUFc/hPTRaV yEOtgvgBMZXg== X-IronPort-AV: E=McAfee;i="6200,9189,10022"; a="206964820" X-IronPort-AV: E=Sophos;i="5.83,291,1616482800"; d="scan'208";a="206964820" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2021 02:38:40 -0700 IronPort-SDR: YjghSkQQhfpqXchU5FmETDRBEKjg++NYbLVqNiZ3bX2P3p3aIdtsxEvlzAZ3khA29zw8feGTq5 hSRTxjz0q+eg== X-IronPort-AV: E=Sophos;i="5.83,291,1616482800"; d="scan'208";a="641599201" Received: from vmm_a4_icx.sh.intel.com (HELO localhost.localdomain) ([10.239.53.245]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2021 02:38:37 -0700 From: Zhu Lingshan To: lingshan.zhu@live.com Cc: Like Xu , Will Deacon , Marc Zyngier , Guo Ren , Nick Hu , Paul Walmsley , Boris Ostrovsky , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-csky@vger.kernel.org, linux-riscv@lists.infradead.org, xen-devel@lists.xenproject.org, Peter Zijlstra , Zhu Lingshan Subject: [PATCH V7 01/18] perf/core: Use static_call to optimize perf_guest_info_callbacks Date: Tue, 22 Jun 2021 17:38:06 +0800 Message-Id: <20210622093823.8215-2-lingshan.zhu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210622093823.8215-1-lingshan.zhu@intel.com> References: <20210622093823.8215-1-lingshan.zhu@intel.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210622_023849_504043_52BC8817 X-CRM114-Status: GOOD ( 19.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Like Xu For "struct perf_guest_info_callbacks", the two fields "is_in_guest" and "is_user_mode" are replaced with a new multiplexed member named "state", and the "get_guest_ip" field will be renamed to "get_ip". For arm64, xen and kvm/x86, the application of DEFINE_STATIC_CALL_RET0 could make all that perf_guest_cbs stuff suck less. For arm, csky, nds32, and riscv, just applied some renamed refactoring. Cc: Will Deacon Cc: Marc Zyngier Cc: Guo Ren Cc: Nick Hu Cc: Paul Walmsley Cc: Boris Ostrovsky Cc: linux-arm-kernel@lists.infradead.org Cc: kvmarm@lists.cs.columbia.edu Cc: linux-csky@vger.kernel.org Cc: linux-riscv@lists.infradead.org Cc: xen-devel@lists.xenproject.org Suggested-by: Peter Zijlstra (Intel) Original-by: Peter Zijlstra (Intel) Signed-off-by: Like Xu Signed-off-by: Zhu Lingshan --- arch/arm/kernel/perf_callchain.c | 16 ++++++++----- arch/arm64/kernel/perf_callchain.c | 29 ++++++++++++++++++----- arch/arm64/kvm/perf.c | 22 ++++++++--------- arch/csky/kernel/perf_callchain.c | 4 ++-- arch/nds32/kernel/perf_event_cpu.c | 16 ++++++++----- arch/riscv/kernel/perf_callchain.c | 4 ++-- arch/x86/events/core.c | 38 ++++++++++++++++++++++++------ arch/x86/events/intel/core.c | 7 +++--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/pmu.c | 2 +- arch/x86/kvm/x86.c | 37 ++++++++++++++++------------- arch/x86/xen/pmu.c | 33 +++++++++++--------------- include/linux/perf_event.h | 12 ++++++---- kernel/events/core.c | 9 +++++++ 14 files changed, 144 insertions(+), 87 deletions(-) diff --git a/arch/arm/kernel/perf_callchain.c b/arch/arm/kernel/perf_callchain.c index 3b69a76d341e..1ce30f86d6c7 100644 --- a/arch/arm/kernel/perf_callchain.c +++ b/arch/arm/kernel/perf_callchain.c @@ -64,7 +64,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs { struct frame_tail __user *tail; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -100,7 +100,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re { struct stackframe fr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -111,8 +111,8 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (perf_guest_cbs && perf_guest_cbs->state()) + return perf_guest_cbs->get_ip(); return instruction_pointer(regs); } @@ -120,9 +120,13 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int state = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (perf_guest_cbs) + state = perf_guest_cbs->state(); + + if (perf_guest_cbs && state) { + if (state & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/arm64/kernel/perf_callchain.c b/arch/arm64/kernel/perf_callchain.c index 88ff471b0bce..5df2bd5d12ba 100644 --- a/arch/arm64/kernel/perf_callchain.c +++ b/arch/arm64/kernel/perf_callchain.c @@ -5,6 +5,7 @@ * Copyright (C) 2015 ARM Limited */ #include +#include #include #include @@ -99,10 +100,25 @@ compat_user_backtrace(struct compat_frame_tail __user *tail, } #endif /* CONFIG_COMPAT */ +DEFINE_STATIC_CALL_RET0(arm64_guest_state, *(perf_guest_cbs->state)); +DEFINE_STATIC_CALL_RET0(arm64_guest_get_ip, *(perf_guest_cbs->get_ip)); + +void arch_perf_update_guest_cbs(void) +{ + static_call_update(arm64_guest_state, (void *)&__static_call_return0); + static_call_update(arm64_guest_get_ip, (void *)&__static_call_return0); + + if (perf_guest_cbs && perf_guest_cbs->state) + static_call_update(arm64_guest_state, perf_guest_cbs->state); + + if (perf_guest_cbs && perf_guest_cbs->get_ip) + static_call_update(arm64_guest_get_ip, perf_guest_cbs->get_ip); +} + void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(arm64_guest_state)()) { /* We don't support guest os callchain now */ return; } @@ -149,7 +165,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, { struct stackframe frame; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(arm64_guest_state)()) { /* We don't support guest os callchain now */ return; } @@ -160,8 +176,8 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (static_call(arm64_guest_state)()) + return static_call(arm64_guest_get_ip)(); return instruction_pointer(regs); } @@ -169,9 +185,10 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int guest = static_call(arm64_guest_state)(); - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (guest) { + if (guest & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/arm64/kvm/perf.c b/arch/arm64/kvm/perf.c index 151c31fb9860..8a3387e58f42 100644 --- a/arch/arm64/kvm/perf.c +++ b/arch/arm64/kvm/perf.c @@ -13,21 +13,20 @@ DEFINE_STATIC_KEY_FALSE(kvm_arm_pmu_available); -static int kvm_is_in_guest(void) -{ - return kvm_get_running_vcpu() != NULL; -} - -static int kvm_is_user_mode(void) +static unsigned int kvm_guest_state(void) { struct kvm_vcpu *vcpu; + unsigned int state = 0; + + if (kvm_get_running_vcpu()) + state |= PERF_GUEST_ACTIVE; vcpu = kvm_get_running_vcpu(); - if (vcpu) - return !vcpu_mode_priv(vcpu); + if (vcpu && !vcpu_mode_priv(vcpu)) + state |= PERF_GUEST_USER; - return 0; + return state; } static unsigned long kvm_get_guest_ip(void) @@ -43,9 +42,8 @@ static unsigned long kvm_get_guest_ip(void) } static struct perf_guest_info_callbacks kvm_guest_cbs = { - .is_in_guest = kvm_is_in_guest, - .is_user_mode = kvm_is_user_mode, - .get_guest_ip = kvm_get_guest_ip, + .state = kvm_guest_state, + .get_ip = kvm_get_guest_ip, }; int kvm_perf_init(void) diff --git a/arch/csky/kernel/perf_callchain.c b/arch/csky/kernel/perf_callchain.c index ab55e98ee8f6..3e42239dd1b2 100644 --- a/arch/csky/kernel/perf_callchain.c +++ b/arch/csky/kernel/perf_callchain.c @@ -89,7 +89,7 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry, unsigned long fp = 0; /* C-SKY does not support virtualization. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) + if (perf_guest_cbs && perf_guest_cbs->state()) return; fp = regs->regs[4]; @@ -113,7 +113,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct stackframe fr; /* C-SKY does not support virtualization. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { pr_warn("C-SKY does not support perf in guest mode!"); return; } diff --git a/arch/nds32/kernel/perf_event_cpu.c b/arch/nds32/kernel/perf_event_cpu.c index 0ce6f9f307e6..1dc32ba842ce 100644 --- a/arch/nds32/kernel/perf_event_cpu.c +++ b/arch/nds32/kernel/perf_event_cpu.c @@ -1371,7 +1371,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, leaf_fp = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -1481,7 +1481,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, { struct stackframe fr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { /* We don't support guest os callchain now */ return; } @@ -1494,8 +1494,8 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, unsigned long perf_instruction_pointer(struct pt_regs *regs) { /* However, NDS32 does not support virtualization */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + if (perf_guest_cbs && perf_guest_cbs->state()) + return perf_guest_cbs->get_ip(); return instruction_pointer(regs); } @@ -1503,10 +1503,14 @@ unsigned long perf_instruction_pointer(struct pt_regs *regs) unsigned long perf_misc_flags(struct pt_regs *regs) { int misc = 0; + unsigned int state = 0; + + if (perf_guest_cbs) + state = perf_guest_cbs->state(); /* However, NDS32 does not support virtualization */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (perf_guest_cbs && state) { + if (state & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/riscv/kernel/perf_callchain.c b/arch/riscv/kernel/perf_callchain.c index 0bb1854dce83..ea63f70cae5d 100644 --- a/arch/riscv/kernel/perf_callchain.c +++ b/arch/riscv/kernel/perf_callchain.c @@ -59,7 +59,7 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry, unsigned long fp = 0; /* RISC-V does not support perf in guest mode. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) + if (perf_guest_cbs && perf_guest_cbs->state()) return; fp = regs->s0; @@ -79,7 +79,7 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) { /* RISC-V does not support perf in guest mode. */ - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (perf_guest_cbs && perf_guest_cbs->state()) { pr_warn("RISC-V does not support perf in guest mode!"); return; } diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index 8f71dd72ef95..c71af4cfba9b 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -90,6 +90,27 @@ DEFINE_STATIC_CALL_NULL(x86_pmu_pebs_aliases, *x86_pmu.pebs_aliases); */ DEFINE_STATIC_CALL_RET0(x86_pmu_guest_get_msrs, *x86_pmu.guest_get_msrs); +DEFINE_STATIC_CALL_RET0(x86_guest_state, *(perf_guest_cbs->state)); +DEFINE_STATIC_CALL_RET0(x86_guest_get_ip, *(perf_guest_cbs->get_ip)); +DEFINE_STATIC_CALL_RET0(x86_guest_handle_intel_pt_intr, *(perf_guest_cbs->handle_intel_pt_intr)); + +void arch_perf_update_guest_cbs(void) +{ + static_call_update(x86_guest_state, (void *)&__static_call_return0); + static_call_update(x86_guest_get_ip, (void *)&__static_call_return0); + static_call_update(x86_guest_handle_intel_pt_intr, (void *)&__static_call_return0); + + if (perf_guest_cbs && perf_guest_cbs->state) + static_call_update(x86_guest_state, perf_guest_cbs->state); + + if (perf_guest_cbs && perf_guest_cbs->get_ip) + static_call_update(x86_guest_get_ip, perf_guest_cbs->get_ip); + + if (perf_guest_cbs && perf_guest_cbs->handle_intel_pt_intr) + static_call_update(x86_guest_handle_intel_pt_intr, + perf_guest_cbs->handle_intel_pt_intr); +} + u64 __read_mostly hw_cache_event_ids [PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_OP_MAX] @@ -2738,7 +2759,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re struct unwind_state state; unsigned long addr; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(x86_guest_state)()) { /* TODO: We don't support guest os callchain now */ return; } @@ -2841,7 +2862,7 @@ perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs struct stack_frame frame; const struct stack_frame __user *fp; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { + if (static_call(x86_guest_state)()) { /* TODO: We don't support guest os callchain now */ return; } @@ -2918,18 +2939,21 @@ static unsigned long code_segment_base(struct pt_regs *regs) unsigned long perf_instruction_pointer(struct pt_regs *regs) { - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) - return perf_guest_cbs->get_guest_ip(); + unsigned long ip = static_call(x86_guest_get_ip)(); + + if (likely(!ip)) + ip = regs->ip + code_segment_base(regs); - return regs->ip + code_segment_base(regs); + return ip; } unsigned long perf_misc_flags(struct pt_regs *regs) { + unsigned int guest = static_call(x86_guest_state)(); int misc = 0; - if (perf_guest_cbs && perf_guest_cbs->is_in_guest()) { - if (perf_guest_cbs->is_user_mode()) + if (guest) { + if (guest & PERF_GUEST_USER) misc |= PERF_RECORD_MISC_GUEST_USER; else misc |= PERF_RECORD_MISC_GUEST_KERNEL; diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index e28892270c58..430f5743f3ca 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2780,6 +2780,8 @@ static void intel_pmu_reset(void) local_irq_restore(flags); } +DECLARE_STATIC_CALL(x86_guest_handle_intel_pt_intr, *(perf_guest_cbs->handle_intel_pt_intr)); + static int handle_pmi_common(struct pt_regs *regs, u64 status) { struct perf_sample_data data; @@ -2850,10 +2852,7 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status) */ if (__test_and_clear_bit(GLOBAL_STATUS_TRACE_TOPAPMI_BIT, (unsigned long *)&status)) { handled++; - if (unlikely(perf_guest_cbs && perf_guest_cbs->is_in_guest() && - perf_guest_cbs->handle_intel_pt_intr)) - perf_guest_cbs->handle_intel_pt_intr(); - else + if (!static_call(x86_guest_handle_intel_pt_intr)()) intel_pt_interrupt(); } diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 9c7ced0e3171..f752fcf56d76 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1813,7 +1813,7 @@ int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu); int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err); void __kvm_request_immediate_exit(struct kvm_vcpu *vcpu); -int kvm_is_in_guest(void); +unsigned int kvm_guest_state(void); void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, u32 size); diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 827886c12c16..2dcbd1b30004 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -87,7 +87,7 @@ static void kvm_perf_overflow_intr(struct perf_event *perf_event, * woken up. So we should wake it, but this is impossible from * NMI context. Do it from irq work instead. */ - if (!kvm_is_in_guest()) + if (!kvm_guest_state()) irq_work_queue(&pmc_to_pmu(pmc)->irq_work); else kvm_make_request(KVM_REQ_PMI, pmc->vcpu); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b594275d49b5..9cb1c02d348c 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8035,44 +8035,47 @@ static void kvm_timer_init(void) DEFINE_PER_CPU(struct kvm_vcpu *, current_vcpu); EXPORT_PER_CPU_SYMBOL_GPL(current_vcpu); -int kvm_is_in_guest(void) +unsigned int kvm_guest_state(void) { - return __this_cpu_read(current_vcpu) != NULL; -} - -static int kvm_is_user_mode(void) -{ - int user_mode = 3; + struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); + unsigned int state = 0; - if (__this_cpu_read(current_vcpu)) - user_mode = static_call(kvm_x86_get_cpl)(__this_cpu_read(current_vcpu)); + if (vcpu) { + state |= PERF_GUEST_ACTIVE; + if (static_call(kvm_x86_get_cpl)(vcpu)) + state |= PERF_GUEST_USER; + } - return user_mode != 0; + return state; } -static unsigned long kvm_get_guest_ip(void) +static unsigned long kvm_guest_get_ip(void) { + struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); unsigned long ip = 0; - if (__this_cpu_read(current_vcpu)) - ip = kvm_rip_read(__this_cpu_read(current_vcpu)); + if (vcpu) + ip = kvm_rip_read(vcpu); return ip; } -static void kvm_handle_intel_pt_intr(void) +static unsigned int kvm_handle_intel_pt_intr(void) { struct kvm_vcpu *vcpu = __this_cpu_read(current_vcpu); + if (!vcpu) + return 0; + kvm_make_request(KVM_REQ_PMI, vcpu); __set_bit(MSR_CORE_PERF_GLOBAL_OVF_CTRL_TRACE_TOPA_PMI_BIT, (unsigned long *)&vcpu->arch.pmu.global_status); + return 1; } static struct perf_guest_info_callbacks kvm_guest_cbs = { - .is_in_guest = kvm_is_in_guest, - .is_user_mode = kvm_is_user_mode, - .get_guest_ip = kvm_get_guest_ip, + .state = kvm_guest_state, + .get_ip = kvm_guest_get_ip, .handle_intel_pt_intr = kvm_handle_intel_pt_intr, }; diff --git a/arch/x86/xen/pmu.c b/arch/x86/xen/pmu.c index e13b0b49fcdf..7352cf002b87 100644 --- a/arch/x86/xen/pmu.c +++ b/arch/x86/xen/pmu.c @@ -413,34 +413,30 @@ int pmu_apic_update(uint32_t val) } /* perf callbacks */ -static int xen_is_in_guest(void) +static unsigned int xen_guest_state(void) { const struct xen_pmu_data *xenpmu_data = get_xenpmu_data(); + unsigned int state = 0; if (!xenpmu_data) { pr_warn_once("%s: pmudata not initialized\n", __func__); - return 0; + return state; } if (!xen_initial_domain() || (xenpmu_data->domain_id >= DOMID_SELF)) - return 0; - - return 1; -} + return state; -static int xen_is_user_mode(void) -{ - const struct xen_pmu_data *xenpmu_data = get_xenpmu_data(); + state |= PERF_GUEST_ACTIVE; - if (!xenpmu_data) { - pr_warn_once("%s: pmudata not initialized\n", __func__); - return 0; + if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_PV) { + if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_USER) + state |= PERF_GUEST_USER; + } else { + if (!!(xenpmu_data->pmu.r.regs.cpl & 3)) + state |= PERF_GUEST_USER; } - if (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_PV) - return (xenpmu_data->pmu.pmu_flags & PMU_SAMPLE_USER); - else - return !!(xenpmu_data->pmu.r.regs.cpl & 3); + return state; } static unsigned long xen_get_guest_ip(void) @@ -456,9 +452,8 @@ static unsigned long xen_get_guest_ip(void) } static struct perf_guest_info_callbacks xen_guest_cbs = { - .is_in_guest = xen_is_in_guest, - .is_user_mode = xen_is_user_mode, - .get_guest_ip = xen_get_guest_ip, + .state = xen_guest_state, + .get_ip = xen_get_guest_ip, }; /* Convert registers from Xen's format to Linux' */ diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index f5a6a2f069ed..8065e5f093f4 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -26,11 +26,13 @@ # include #endif +#define PERF_GUEST_ACTIVE 0x01 +#define PERF_GUEST_USER 0x02 + struct perf_guest_info_callbacks { - int (*is_in_guest)(void); - int (*is_user_mode)(void); - unsigned long (*get_guest_ip)(void); - void (*handle_intel_pt_intr)(void); + unsigned int (*state)(void); + unsigned long (*get_ip)(void); + unsigned int (*handle_intel_pt_intr)(void); }; #ifdef CONFIG_HAVE_HW_BREAKPOINT @@ -1237,6 +1239,8 @@ extern void perf_event_bpf_event(struct bpf_prog *prog, u16 flags); extern struct perf_guest_info_callbacks *perf_guest_cbs; +extern void __weak arch_perf_update_guest_cbs(void); + extern int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks); extern int perf_unregister_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks); diff --git a/kernel/events/core.c b/kernel/events/core.c index 6fee4a7e88d7..101fa7d0bfda 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6479,9 +6479,18 @@ static void perf_pending_event(struct irq_work *entry) */ struct perf_guest_info_callbacks *perf_guest_cbs; +/* explicitly use __weak to fix duplicate symbol error */ +void __weak arch_perf_update_guest_cbs(void) +{ +} + int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *cbs) { + if (WARN_ON_ONCE(perf_guest_cbs)) + return -EBUSY; + perf_guest_cbs = cbs; + arch_perf_update_guest_cbs(); return 0; } EXPORT_SYMBOL_GPL(perf_register_guest_info_callbacks); -- 2.27.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel