From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoffer Dall Subject: Re: [PATCH v10 4/5] arm64: arm_pmu: Add support for exclude_host/exclude_guest attributes Date: Wed, 6 Mar 2019 09:42:59 +0100 Message-ID: <20190306084259.GE551@e113682-lin.lund.arm.com> References: <1547482308-29839-1-git-send-email-andrew.murray@arm.com> <1547482308-29839-5-git-send-email-andrew.murray@arm.com> <20190218215307.GA28113@e113682-lin.lund.arm.com> <20190220161539.GA2055@e119886-lin.cambridge.arm.com> <20190226124459.GB551@e113682-lin.lund.arm.com> <20190304111424.GC40984@e119886-lin.cambridge.arm.com> <20190305114551.GG40984@e119886-lin.cambridge.arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 8821F4A432 for ; Wed, 6 Mar 2019 03:43:03 -0500 (EST) Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kmYYWfBRKROY for ; Wed, 6 Mar 2019 03:43:02 -0500 (EST) Received: from foss.arm.com (usa-sjc-mx-foss1.foss.arm.com [217.140.101.70]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 194DD4A41A for ; Wed, 6 Mar 2019 03:43:02 -0500 (EST) Content-Disposition: inline In-Reply-To: <20190305114551.GG40984@e119886-lin.cambridge.arm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu To: Andrew Murray Cc: Marc Zyngier , Catalin Marinas , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org List-Id: kvmarm@lists.cs.columbia.edu On Tue, Mar 05, 2019 at 11:45:52AM +0000, Andrew Murray wrote: > On Mon, Mar 04, 2019 at 11:14:24AM +0000, Andrew Murray wrote: > > On Tue, Feb 26, 2019 at 01:44:59PM +0100, Christoffer Dall wrote: > > > On Wed, Feb 20, 2019 at 04:15:40PM +0000, Andrew Murray wrote: > > > > On Mon, Feb 18, 2019 at 10:53:07PM +0100, Christoffer Dall wrote: > > > > > On Mon, Jan 14, 2019 at 04:11:47PM +0000, Andrew Murray wrote: > > > > > > Add support for the :G and :H attributes in perf by handling the > > > > > > exclude_host/exclude_guest event attributes. > > > > > > > > > > > > We notify KVM of counters that we wish to be enabled or disabled on > > > > > > guest entry/exit and thus defer from starting or stopping :G events > > > > > > as per the events exclude_host attribute. > > > > > > > > > > > > With both VHE and non-VHE we switch the counters between host/guest > > > > > > at EL2. We are able to eliminate counters counting host events on > > > > > > the boundaries of guest entry/exit when using :G by filtering out > > > > > > EL2 for exclude_host. However when using :H unless exclude_hv is set > > > > > > on non-VHE then there is a small blackout window at the guest > > > > > > entry/exit where host events are not captured. > > > > > > > > > > > > Signed-off-by: Andrew Murray > > > > > > Reviewed-by: Suzuki K Poulose > > > > > > --- > > > > > > > > > > > Let me see if I get this right: > > > > > > > > > > exclude_user: VHE: Don't count EL0 > > > > > Non-VHE: Don't count EL0 > > > > > > > > > > exclude_kernel: VHE: Don't count EL2 and don't count EL1 > > > > > Non-VHE: Don't count EL1 > > > > > > > > > > exclude_hv: VHE: No effect > > > > > Non-VHE: Don't count EL2 > > > > > > > > > > exclude_host: VHE: Don't count EL2 + enable/disable on guest entry/exit > > > > > Non-VHE: disable on guest entry/disable on guest entry/exit > > > > > > > > > > And the logic I extract is that _user applies across both guest and > > > > > host, as does _kernel (regardless of the mode the kernel on the current > > > > > system runs in, might be only EL1, might be EL1 and EL2), and _hv is > > > > > specific to non-VHE systems to measure events in a specific piece of KVM > > > > > code that runs at EL2. > > > > > > > > > > As I expressed before, that doesn't seem to be the intent behind the > > > > > exclude_hv flag, but I'm not sure how other architectures actually > > > > > implement things today, and even if it's a curiosity of the Arm > > > > > architecture and has value to non-VHE hypervisor hackers, and we don't > > > > > really have to care about uniformity with the other architectures, then > > > > > fine. > > > > > > > > > > It has taken me a while to make sense of this code change, so I really > > > > > wish we can find a suitable place to document the semantics clearly for > > > > > perf users on arm64. > > > > > > > > > > Now, another thing comes to mind: Do we really need to enable and > > > > > disable anything on a VHE system on entry/exit to/from a guest? Can we > > > > > instead do the following: > > > > > > > > > > exclude_host: Disable EL2 counting > > > > > Disable EL0 counting > > > > > Enable EL0 counting on vcpu_load > > > > > (unless exclude_user is also set) > > > > > Disable EL0 counting on vcpu_put > > > > > > > > > > exclude_guest: Disable EL1 counting > > > > > Disable EL0 counting on vcpu_load > > > > > Enable EL0 counting on vcpu_put > > > > > (unless exclude_user is also set) > > > > > > > > > > If that works, we can avoid the overhead in the critical path on VHE > > > > > systems and actually have slightly more accurate counting, leaving the > > > > > entry/exit operations to be specific to non-VHE. > > > > > > > > This all makes sense. > > > > > > > > At present on VHE, for host only events, there is a small blackout window > > > > at the guest entry/exit - this is where we turn off/on host counting before > > > > entering/exiting the guest. (This blackout window also exists on !VHE unless > > > > exclude_hv is set). > > > > > > > > To mitigate against this the PMU switching code was brought as close to the > > > > guest entry/exit as possible - but as you point out at this point in > > > > kvm_arch_vcpu_ioctl_run we're running without interrupts/preemption and so > > > > on the critical path. I believe we add about 11 instructions when there are > > > > no PMU counters enabled and about 23 when they are enabled. I suppose it > > > > would be possible to use static keys to reduce the overhead when counters > > > > are not enabled... > > > > > > > > Your suggestion provides an optimal solution, however it adds some > > > > complexity - here is a rough attempt at implementing it... > > > > > > > > > > > > > Do you think this is worth developing further? > > > > > > > > > > Yes, I think it's worth going to fairly great lengths to keep the > > > critical path on VHE systems lean, and I suspect if we take the easy way > > > to add functionality first, it will only be harder to factor things out > > > later. > > It looks like we can also apply this same approach on !VHE. For the sole > use-case where a user wishes to count only for a host or guest EL0, then we > switch the counter at EL1 in vcpu_load instead of the switch code. > ok, even better then. Looking forward to seeing the patches. Thanks, Christoffer From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_PASS,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8677C43381 for ; Wed, 6 Mar 2019 08:43:18 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7670820675 for ; Wed, 6 Mar 2019 08:43:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="olcvuYI7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7670820675 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=zmVxCoud8kWtObgwb5Ci7lBUu8x377YT/HuMDOiYkXE=; b=olcvuYI7u1dMBj 76jncQl9MtrZ82X9u5IdWdrRLsVyaLSV8Ad3AEBk5VZiqForziNGYJTHu+ugLCsJlV5efk507ZZKA 6D5RaZ2/P2h7JcBaiKZ8Jzn/UGYkShEwUvrkBGlwWGRl9dS1Xb1JUYSJB52yB6yedOtTqfdaAwLU8 +ZcsZ/1xFxfKA71biGCzfAYlbkKuOCEVHGJE1GxDHq6dWwfhc56NBA2KEz8XLpqW2dNH0iaXcPbqJ xfLy4uQGc/EWRf7DlDH/LRRP4XzRjtV6/bDd0MOzEEu0/VbCpBYvIlqQyuXz0qN6E0f5whJFKDvTu wnvzAYgWPxbafRUjsT4A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h1S8p-0004n6-PH; Wed, 06 Mar 2019 08:43:07 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h1S8m-0004mj-Hj for linux-arm-kernel@lists.infradead.org; Wed, 06 Mar 2019 08:43:06 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 546B4A78; Wed, 6 Mar 2019 00:43:01 -0800 (PST) Received: from localhost (e113682-lin.copenhagen.arm.com [10.32.144.41]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E28203F71D; Wed, 6 Mar 2019 00:43:00 -0800 (PST) Date: Wed, 6 Mar 2019 09:42:59 +0100 From: Christoffer Dall To: Andrew Murray Subject: Re: [PATCH v10 4/5] arm64: arm_pmu: Add support for exclude_host/exclude_guest attributes Message-ID: <20190306084259.GE551@e113682-lin.lund.arm.com> References: <1547482308-29839-1-git-send-email-andrew.murray@arm.com> <1547482308-29839-5-git-send-email-andrew.murray@arm.com> <20190218215307.GA28113@e113682-lin.lund.arm.com> <20190220161539.GA2055@e119886-lin.cambridge.arm.com> <20190226124459.GB551@e113682-lin.lund.arm.com> <20190304111424.GC40984@e119886-lin.cambridge.arm.com> <20190305114551.GG40984@e119886-lin.cambridge.arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190305114551.GG40984@e119886-lin.cambridge.arm.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190306_004304_601575_BEB00B6D X-CRM114-Status: GOOD ( 37.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Marc Zyngier , Catalin Marinas , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Mar 05, 2019 at 11:45:52AM +0000, Andrew Murray wrote: > On Mon, Mar 04, 2019 at 11:14:24AM +0000, Andrew Murray wrote: > > On Tue, Feb 26, 2019 at 01:44:59PM +0100, Christoffer Dall wrote: > > > On Wed, Feb 20, 2019 at 04:15:40PM +0000, Andrew Murray wrote: > > > > On Mon, Feb 18, 2019 at 10:53:07PM +0100, Christoffer Dall wrote: > > > > > On Mon, Jan 14, 2019 at 04:11:47PM +0000, Andrew Murray wrote: > > > > > > Add support for the :G and :H attributes in perf by handling the > > > > > > exclude_host/exclude_guest event attributes. > > > > > > > > > > > > We notify KVM of counters that we wish to be enabled or disabled on > > > > > > guest entry/exit and thus defer from starting or stopping :G events > > > > > > as per the events exclude_host attribute. > > > > > > > > > > > > With both VHE and non-VHE we switch the counters between host/guest > > > > > > at EL2. We are able to eliminate counters counting host events on > > > > > > the boundaries of guest entry/exit when using :G by filtering out > > > > > > EL2 for exclude_host. However when using :H unless exclude_hv is set > > > > > > on non-VHE then there is a small blackout window at the guest > > > > > > entry/exit where host events are not captured. > > > > > > > > > > > > Signed-off-by: Andrew Murray > > > > > > Reviewed-by: Suzuki K Poulose > > > > > > --- > > > > > > > > > > > Let me see if I get this right: > > > > > > > > > > exclude_user: VHE: Don't count EL0 > > > > > Non-VHE: Don't count EL0 > > > > > > > > > > exclude_kernel: VHE: Don't count EL2 and don't count EL1 > > > > > Non-VHE: Don't count EL1 > > > > > > > > > > exclude_hv: VHE: No effect > > > > > Non-VHE: Don't count EL2 > > > > > > > > > > exclude_host: VHE: Don't count EL2 + enable/disable on guest entry/exit > > > > > Non-VHE: disable on guest entry/disable on guest entry/exit > > > > > > > > > > And the logic I extract is that _user applies across both guest and > > > > > host, as does _kernel (regardless of the mode the kernel on the current > > > > > system runs in, might be only EL1, might be EL1 and EL2), and _hv is > > > > > specific to non-VHE systems to measure events in a specific piece of KVM > > > > > code that runs at EL2. > > > > > > > > > > As I expressed before, that doesn't seem to be the intent behind the > > > > > exclude_hv flag, but I'm not sure how other architectures actually > > > > > implement things today, and even if it's a curiosity of the Arm > > > > > architecture and has value to non-VHE hypervisor hackers, and we don't > > > > > really have to care about uniformity with the other architectures, then > > > > > fine. > > > > > > > > > > It has taken me a while to make sense of this code change, so I really > > > > > wish we can find a suitable place to document the semantics clearly for > > > > > perf users on arm64. > > > > > > > > > > Now, another thing comes to mind: Do we really need to enable and > > > > > disable anything on a VHE system on entry/exit to/from a guest? Can we > > > > > instead do the following: > > > > > > > > > > exclude_host: Disable EL2 counting > > > > > Disable EL0 counting > > > > > Enable EL0 counting on vcpu_load > > > > > (unless exclude_user is also set) > > > > > Disable EL0 counting on vcpu_put > > > > > > > > > > exclude_guest: Disable EL1 counting > > > > > Disable EL0 counting on vcpu_load > > > > > Enable EL0 counting on vcpu_put > > > > > (unless exclude_user is also set) > > > > > > > > > > If that works, we can avoid the overhead in the critical path on VHE > > > > > systems and actually have slightly more accurate counting, leaving the > > > > > entry/exit operations to be specific to non-VHE. > > > > > > > > This all makes sense. > > > > > > > > At present on VHE, for host only events, there is a small blackout window > > > > at the guest entry/exit - this is where we turn off/on host counting before > > > > entering/exiting the guest. (This blackout window also exists on !VHE unless > > > > exclude_hv is set). > > > > > > > > To mitigate against this the PMU switching code was brought as close to the > > > > guest entry/exit as possible - but as you point out at this point in > > > > kvm_arch_vcpu_ioctl_run we're running without interrupts/preemption and so > > > > on the critical path. I believe we add about 11 instructions when there are > > > > no PMU counters enabled and about 23 when they are enabled. I suppose it > > > > would be possible to use static keys to reduce the overhead when counters > > > > are not enabled... > > > > > > > > Your suggestion provides an optimal solution, however it adds some > > > > complexity - here is a rough attempt at implementing it... > > > > > > > > > > > > > Do you think this is worth developing further? > > > > > > > > > > Yes, I think it's worth going to fairly great lengths to keep the > > > critical path on VHE systems lean, and I suspect if we take the easy way > > > to add functionality first, it will only be harder to factor things out > > > later. > > It looks like we can also apply this same approach on !VHE. For the sole > use-case where a user wishes to count only for a host or guest EL0, then we > switch the counter at EL1 in vcpu_load instead of the switch code. > ok, even better then. Looking forward to seeing the patches. Thanks, Christoffer _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel