From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CAFCC433F5 for ; Thu, 27 Jan 2022 16:18:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id B395849ED9; Thu, 27 Jan 2022 11:18:02 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D1-ThxhEcbHb; Thu, 27 Jan 2022 11:18:01 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 43E4049F10; Thu, 27 Jan 2022 11:18:01 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 4E9B04A11C for ; Thu, 27 Jan 2022 11:18:00 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wFl8ejG9XwJY for ; Thu, 27 Jan 2022 11:17:58 -0500 (EST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mm01.cs.columbia.edu (Postfix) with ESMTP id D9C1449ED9 for ; Thu, 27 Jan 2022 11:17:58 -0500 (EST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0A7A61063; Thu, 27 Jan 2022 08:17:58 -0800 (PST) Received: from monolith.localdoman (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 241263F766; Thu, 27 Jan 2022 08:17:56 -0800 (PST) From: Alexandru Elisei To: maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, will@kernel.org, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu Subject: [PATCH v4 0/6] KVM: arm64: Improve PMU support on heterogeneous systems Date: Thu, 27 Jan 2022 16:17:53 +0000 Message-Id: <20220127161759.53553-1-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Cc: tglx@linutronix.de, mingo@redhat.com X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu (CC'ing Peter Maydell in case this might be of interest to qemu) The series can be found on a branch at [1], and the kvmtool support at [2]. Detailed explanation of the issue and symptoms that the patches attempt to correct can be found in the cover letter for v1 [3]. A summary of the problem is that on heterogeneous systems KVM will always use the same PMU for creating the VCPU events for *all* VCPUs regardless of the physical CPU on which the VCPU is running, leading to events suddenly stopping and resuming in the guest as the VCPU thread gets migrated across different CPUs. This series proposes to fix this behaviour by allowing the user to specify which physical PMU is used when creating the VCPU events needed for guest PMU emulation. When the PMU is set, KVM will refuse to the VCPU on a physical which is not part of the supported CPUs for the specified PMU. The restriction is that all VCPUs must use the same PMU to avoid emulating an asymmetric platform. The default behaviour stays the same - without userspace setting the PMU, events will stop counting if the VCPU is scheduled on the wrong CPU. Changes since v3 [4]: - Gathered Reviewed-by, thank you! - Rebased on top of v5.17-rc1, on top of which I added the USB Type-C fixes to get my rockpro64 to boot [6]. Shouldn't affect KVM in any way. - Patches #1 ("KVM: arm64: Do not change the PMU event filter after a VCPU has run") and #3 ("KVM: arm64: Keep a per-VM pointer to the default PMU") are new. - Use list_is_singular() in #4 ("KVM: arm64: Keep a list of probed PMUs"). - Return -EBUSY if a VCPU has run or if a PMU event filter has been set. - Use the PMU that the user specified to replace the default PMU at the VM level. Changes since v2 [5]: - Rebased on top of v5.16-rc5 - Check that all VCPUs have the same PMU set (or none at all). - Use the VCPU's PMUVer value when calculating the event mask, if a PMU is set for that VCPU. - Clear the unsupported CPU flag in vcpu_put(). - Move the handling of the unsupported CPU flag in kvm_vcpu_exit_request(). - Free the cpumask of supported CPUs if kvm_arch_vcpu_create() fails. Changes since v1 [3]: - Rebased on top of v5.16-rc4 - Implemented review comments: protect iterating through the list of PMUs with a mutex, documentation changes, initialize vcpu-arch.supported_cpus to cpu_possible_mask, changed vcpu->arch.cpu_not_supported to a VCPU flag, set exit reason to KVM_EXIT_FAIL_ENTRY and populate fail_entry when the VCPU is run on a CPU not in the PMU's supported cpumask. Many thanks for the review! [1] https://gitlab.arm.com/linux-arm/linux-ae/-/tree/pmu-big-little-fix-v4 [2] https://gitlab.arm.com/linux-arm/kvmtool-ae/-/tree/pmu-big-little-fix-v2 [3] https://www.spinics.net/lists/arm-kernel/msg933579.html [4] https://www.spinics.net/lists/kvm-arm/msg51079.html [5] https://www.spinics.net/lists/kvm-arm/msg50944.html [6] https://www.spinics.net/lists/linux-usb/msg221604.html Alexandru Elisei (4): perf: Fix wrong name in comment for struct perf_cpu_context KVM: arm64: Keep a list of probed PMUs KVM: arm64: Add KVM_ARM_VCPU_PMU_V3_SET_PMU attribute KVM: arm64: Refuse to run VCPU if the PMU doesn't match the physical CPU Marc Zyngier (2): KVM: arm64: Do not change the PMU event filter after a VCPU has run KVM: arm64: Keep a per-VM pointer to the default PMU Documentation/virt/kvm/devices/vcpu.rst | 34 +++++- arch/arm64/include/asm/kvm_host.h | 15 ++- arch/arm64/include/uapi/asm/kvm.h | 4 + arch/arm64/kvm/arm.c | 21 ++++ arch/arm64/kvm/pmu-emul.c | 141 ++++++++++++++++++------ include/kvm/arm_pmu.h | 5 + include/linux/perf_event.h | 2 +- tools/arch/arm64/include/uapi/asm/kvm.h | 1 + 8 files changed, 189 insertions(+), 34 deletions(-) -- 2.34.1 _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D7EFC433EF for ; Thu, 27 Jan 2022 16:19:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=YTlFApy+OZaUEy8/0aZS0KiArDnaRDuZrA5BrHNf4nc=; b=1XpjZscd+J9RB2 HQR+hsDTEJWZHzvW5eUE8MHVAbcRTJCXXhczhdwrlk9bbdMlhgwJ05idNjoPEJzSY1Jlr2k5P7LMm JiOS2xB3NzZ5J1TZkntwdGuuJv5i+oGRq8yIb673VLLLJQS48m4/vmiDIoEIoYsi38v+/WDvwNcWB A6r3Pwdidudp6oUUWWijD6Md7CyOIFSP96fVbpcVBBK6m//VZOj92fVxOC8Avxpu/TTVjwP8F78ys uW5kdgA8WSYaqRh42CsK/6We8F7UOFGoKtMPN0dQfa7gWD4YbNKc1TVCYdcvXco5fFDvtqBZPhIyI /0us484pkAAMAt+c6dPg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nD7TJ-00GQZM-7i; Thu, 27 Jan 2022 16:18:05 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nD7TF-00GQYY-2u for linux-arm-kernel@lists.infradead.org; Thu, 27 Jan 2022 16:18:03 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0A7A61063; Thu, 27 Jan 2022 08:17:58 -0800 (PST) Received: from monolith.localdoman (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 241263F766; Thu, 27 Jan 2022 08:17:56 -0800 (PST) From: Alexandru Elisei To: maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, will@kernel.org, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu Cc: tglx@linutronix.de, mingo@redhat.com, peter.maydell@linaro.org, reijiw@google.com Subject: [PATCH v4 0/6] KVM: arm64: Improve PMU support on heterogeneous systems Date: Thu, 27 Jan 2022 16:17:53 +0000 Message-Id: <20220127161759.53553-1-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220127_081801_265968_8CDCE007 X-CRM114-Status: GOOD ( 18.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org (CC'ing Peter Maydell in case this might be of interest to qemu) The series can be found on a branch at [1], and the kvmtool support at [2]. Detailed explanation of the issue and symptoms that the patches attempt to correct can be found in the cover letter for v1 [3]. A summary of the problem is that on heterogeneous systems KVM will always use the same PMU for creating the VCPU events for *all* VCPUs regardless of the physical CPU on which the VCPU is running, leading to events suddenly stopping and resuming in the guest as the VCPU thread gets migrated across different CPUs. This series proposes to fix this behaviour by allowing the user to specify which physical PMU is used when creating the VCPU events needed for guest PMU emulation. When the PMU is set, KVM will refuse to the VCPU on a physical which is not part of the supported CPUs for the specified PMU. The restriction is that all VCPUs must use the same PMU to avoid emulating an asymmetric platform. The default behaviour stays the same - without userspace setting the PMU, events will stop counting if the VCPU is scheduled on the wrong CPU. Changes since v3 [4]: - Gathered Reviewed-by, thank you! - Rebased on top of v5.17-rc1, on top of which I added the USB Type-C fixes to get my rockpro64 to boot [6]. Shouldn't affect KVM in any way. - Patches #1 ("KVM: arm64: Do not change the PMU event filter after a VCPU has run") and #3 ("KVM: arm64: Keep a per-VM pointer to the default PMU") are new. - Use list_is_singular() in #4 ("KVM: arm64: Keep a list of probed PMUs"). - Return -EBUSY if a VCPU has run or if a PMU event filter has been set. - Use the PMU that the user specified to replace the default PMU at the VM level. Changes since v2 [5]: - Rebased on top of v5.16-rc5 - Check that all VCPUs have the same PMU set (or none at all). - Use the VCPU's PMUVer value when calculating the event mask, if a PMU is set for that VCPU. - Clear the unsupported CPU flag in vcpu_put(). - Move the handling of the unsupported CPU flag in kvm_vcpu_exit_request(). - Free the cpumask of supported CPUs if kvm_arch_vcpu_create() fails. Changes since v1 [3]: - Rebased on top of v5.16-rc4 - Implemented review comments: protect iterating through the list of PMUs with a mutex, documentation changes, initialize vcpu-arch.supported_cpus to cpu_possible_mask, changed vcpu->arch.cpu_not_supported to a VCPU flag, set exit reason to KVM_EXIT_FAIL_ENTRY and populate fail_entry when the VCPU is run on a CPU not in the PMU's supported cpumask. Many thanks for the review! [1] https://gitlab.arm.com/linux-arm/linux-ae/-/tree/pmu-big-little-fix-v4 [2] https://gitlab.arm.com/linux-arm/kvmtool-ae/-/tree/pmu-big-little-fix-v2 [3] https://www.spinics.net/lists/arm-kernel/msg933579.html [4] https://www.spinics.net/lists/kvm-arm/msg51079.html [5] https://www.spinics.net/lists/kvm-arm/msg50944.html [6] https://www.spinics.net/lists/linux-usb/msg221604.html Alexandru Elisei (4): perf: Fix wrong name in comment for struct perf_cpu_context KVM: arm64: Keep a list of probed PMUs KVM: arm64: Add KVM_ARM_VCPU_PMU_V3_SET_PMU attribute KVM: arm64: Refuse to run VCPU if the PMU doesn't match the physical CPU Marc Zyngier (2): KVM: arm64: Do not change the PMU event filter after a VCPU has run KVM: arm64: Keep a per-VM pointer to the default PMU Documentation/virt/kvm/devices/vcpu.rst | 34 +++++- arch/arm64/include/asm/kvm_host.h | 15 ++- arch/arm64/include/uapi/asm/kvm.h | 4 + arch/arm64/kvm/arm.c | 21 ++++ arch/arm64/kvm/pmu-emul.c | 141 ++++++++++++++++++------ include/kvm/arm_pmu.h | 5 + include/linux/perf_event.h | 2 +- tools/arch/arm64/include/uapi/asm/kvm.h | 1 + 8 files changed, 189 insertions(+), 34 deletions(-) -- 2.34.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel