From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.6 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFF30C43387 for ; Tue, 18 Dec 2018 14:38:55 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9528421871 for ; Tue, 18 Dec 2018 14:38:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="RDUW8Saz" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9528421871 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=T2A8y85Khf8BLk/FfutbHIc9fYK3buI3v6G8x8a4Yqg=; b=RDUW8Sazg0cawM XZSZV3Edzqid/oi6QkjWrGk0bW/Niu0nBBAOOqIaJ9aZlbcbU/c9pe5EGV8ae3V+R6+TVzSVPYk/k P46VoH14xj8ueoAzmNRyMXffVDeqaMd7XlHa+SHKt8ROn2vw1SVJawi5Fi0llNg7H7GCoNq30VEKE Q6GbvJqfAHHHhIea+N8FrB90BfB3W6GjpqsYgK3Z3nzQrgBKrPsHEOHZ9EluDyt5t+KZM5tCZr0DL kM6i1cmubDV32hIfRdxa1CVatL/8DHtpr2GsdBm6c9btfnT5zjHuDFTmUBYZ/CeqmGELcEISxhMcq 9aS/EKwrtyWThJKr5+QQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gZGWI-0005Xr-IA; Tue, 18 Dec 2018 14:38:50 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gZGWF-0005X6-83 for linux-arm-kernel@lists.infradead.org; Tue, 18 Dec 2018 14:38:48 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E47EF80D; Tue, 18 Dec 2018 06:38:35 -0800 (PST) Received: from localhost (e113682-lin.copenhagen.arm.com [10.32.144.41]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7BA293F675; Tue, 18 Dec 2018 06:38:35 -0800 (PST) Date: Tue, 18 Dec 2018 15:38:33 +0100 From: Christoffer Dall To: Andrew Murray Subject: Re: [PATCH v8 4/5] arm64: arm_pmu: Add support for exclude_host/exclude_guest attributes Message-ID: <20181218143833.GD25383@e113682-lin.lund.arm.com> References: <1544610573-28446-1-git-send-email-andrew.murray@arm.com> <1544610573-28446-5-git-send-email-andrew.murray@arm.com> <20181218120226.GC25383@e113682-lin.lund.arm.com> <20181218132531.GF47018@e119886-lin.cambridge.arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181218132531.GF47018@e119886-lin.cambridge.arm.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181218_063847_296691_A0742E00 X-CRM114-Status: GOOD ( 41.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Julien Thierry , Marc Zyngier , Catalin Marinas , joro@8bytes.org, Suzuki K Poulose , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Dec 18, 2018 at 01:25:32PM +0000, Andrew Murray wrote: > On Tue, Dec 18, 2018 at 01:02:26PM +0100, Christoffer Dall wrote: > > On Wed, Dec 12, 2018 at 10:29:32AM +0000, Andrew Murray wrote: > > > Add support for the :G and :H attributes in perf by handling the > > > exclude_host/exclude_guest event attributes. > > > > > > We notify KVM of counters that we wish to be enabled or disabled on > > > guest entry/exit and thus defer from starting or stopping :G events > > > as per the events exclude_host attribute. > > > > > > With both VHE and non-VHE we switch the counters between host/guest > > > at EL2. We are able to eliminate counters counting host events on > > > the boundaries of guest entry/exit when using :G by filtering out > > > EL2 for exclude_host. However when using :H unless exclude_hv is set > > > on non-VHE then there is a small blackout window at the guest > > > entry/exit where host events are not captured. > > > > > > Signed-off-by: Andrew Murray > > > --- > > > arch/arm64/kernel/perf_event.c | 51 ++++++++++++++++++++++++++++++++++++------ > > > 1 file changed, 44 insertions(+), 7 deletions(-) > > > > > > diff --git a/arch/arm64/kernel/perf_event.c b/arch/arm64/kernel/perf_event.c > > > index de564ae..4a3c73d 100644 > > > --- a/arch/arm64/kernel/perf_event.c > > > +++ b/arch/arm64/kernel/perf_event.c > > > @@ -26,6 +26,7 @@ > > > > > > #include > > > #include > > > +#include > > > #include > > > #include > > > #include > > > @@ -647,11 +648,26 @@ static inline int armv8pmu_enable_counter(int idx) > > > > > > static inline void armv8pmu_enable_event_counter(struct perf_event *event) > > > { > > > + struct perf_event_attr *attr = &event->attr; > > > int idx = event->hw.idx; > > > + int flags = 0; > > > + u32 counter_bits = BIT(ARMV8_IDX_TO_COUNTER(idx)); > > > > > > - armv8pmu_enable_counter(idx); > > > if (armv8pmu_event_is_chained(event)) > > > - armv8pmu_enable_counter(idx - 1); > > > + counter_bits |= BIT(ARMV8_IDX_TO_COUNTER(idx - 1)); > > > + > > > + if (!attr->exclude_host) > > > + flags |= KVM_PMU_EVENTS_HOST; > > > + if (!attr->exclude_guest) > > > + flags |= KVM_PMU_EVENTS_GUEST; > > > + > > > + kvm_set_pmu_events(counter_bits, flags); > > > + > > > + if (!attr->exclude_host) { > > > + armv8pmu_enable_counter(idx); > > > + if (armv8pmu_event_is_chained(event)) > > > + armv8pmu_enable_counter(idx - 1); > > > + } > > > } > > > > > > static inline int armv8pmu_disable_counter(int idx) > > > @@ -664,11 +680,20 @@ static inline int armv8pmu_disable_counter(int idx) > > > static inline void armv8pmu_disable_event_counter(struct perf_event *event) > > > { > > > struct hw_perf_event *hwc = &event->hw; > > > + struct perf_event_attr *attr = &event->attr; > > > int idx = hwc->idx; > > > + u32 counter_bits = BIT(ARMV8_IDX_TO_COUNTER(idx)); > > > > > > if (armv8pmu_event_is_chained(event)) > > > - armv8pmu_disable_counter(idx - 1); > > > - armv8pmu_disable_counter(idx); > > > + counter_bits |= BIT(ARMV8_IDX_TO_COUNTER(idx - 1)); > > > + > > > + kvm_clr_pmu_events(counter_bits); > > > + > > > + if (!attr->exclude_host) { > > > + if (armv8pmu_event_is_chained(event)) > > > + armv8pmu_disable_counter(idx - 1); > > > + armv8pmu_disable_counter(idx); > > > + } > > > } > > > > > > static inline int armv8pmu_enable_intens(int idx) > > > @@ -943,16 +968,25 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event, > > > * Therefore we ignore exclude_hv in this configuration, since > > > * there's no hypervisor to sample anyway. This is consistent > > > * with other architectures (x86 and Power). > > > + * > > > + * To eliminate counting host events on the boundaries of > > > + * guest entry/exit we ensure EL2 is not included in hyp mode > > > + * with !exclude_host. > > > */ > > > if (is_kernel_in_hyp_mode()) { > > > - if (!attr->exclude_kernel) > > > + if (!attr->exclude_kernel && !attr->exclude_host) > > > config_base |= ARMV8_PMU_INCLUDE_EL2; > > > } else { > > > - if (attr->exclude_kernel) > > > - config_base |= ARMV8_PMU_EXCLUDE_EL1; > > > if (!attr->exclude_hv) > > > config_base |= ARMV8_PMU_INCLUDE_EL2; > > > > I'm not sure about the current use of exclude_hv here. The comment says > > it's consistent with other architectures, but I can't find an example to > > confirm this, and I don't think we have a comparable thing to the split > > of the hypervisor between EL1 and EL2 we have on non-VHE. > > > > Joerg told me the semantics were designed to be: > > > > exclude_hv: When running as a guest, stop counting events when > > the HV runs. > > Can the definition of "guest" here refer to both type 1 and type 2 > hypervisor guests? Or do we assume type 1 only? > A guest is a guest. Linux can run as a guest under a hypervisor with a type 1 or type 2 design, doesn't matter for this conversation. > > > > exclude_host: When Linux runs as a HV itself, only count events > > while a guest is running. > > > > exclude_guest: When Linux runs as a HV, only count events when > > running in host mode. > > > > (But tools/perf/design.txt does not really confirm this). > > > > On arm64 that would mean: > > > > exclude_hv: As a host, no effect. > > As a guest, set the counter to include EL2 for a > > hypervisor to emulate. [...] > > Though more correctly we should count EL2 *and EL1* events whilst pinned > to the KVM task and whilst running outside of the guest. This then > covers both !VHE and VHE and allows for fair comparasion between !VHE > and VHE systems. Yes, if the guest has cleared exclude_hv and if we can properly detect that from the hypervisor. > > This then gives us the unique benefit of the type 2 host being able to > examine the hypervisor overhead of its individual guests. Not sure I understand this part. > > The only issue here is that the type 2 host wouldn't be able to examine > the HV overhead of all its guests across the system as you wouldn't be > able to rely on the perf task pinning to distinguish between EL1 from > host and EL1 from guests in a !VHE system. I'm not sure the best way > to overcome this limitation. Why can't you disable EL1 counting whilst running in the host, and enable EL1 counting whilst running in the guest? > > > > exclude_host: As a guest, has no effect. > > Don't count EL1 host or EL2, but count EL1 guest > > by enabling EL1 counting at EL2 when entering a > > guest, and disabling EL1 counting when returning > > from a guest. > > > > exclude_guest: As a guest, has no effect. As a host, disable > > EL1 counting at EL2 when entering a guest. > > > > Not sure if we break anything by changing the behavior on arm64 now, but > > I really doubt that being able to exclude an arbitrary part (the one tha > > happens to run in EL2 on non-VHE systems) is meaningful, and the fact > > that behavior and semantics change depending on the version of the > > underlying CPU is not great, if what you care about is understanding the > > system's performance. > > This is a bit strange. It's arbitary as it only represents a bit of the > HV overhead - this is solved though by counting the whole overhead (EL1 > and EL2 instead (but only counting outside the guest and pinned to the > guest tasks). Not sure I understand your point here? > > > > > Thoughts? > > > > Though if I've understood you correctly, you're suggesting that the only > time we count EL2 is when exclude_hv is not set on the immediate guest > of a type 1 hypervisor? > No, I didn't say anything about a type 1 or type 2 hypervisor, and I think that distinction is completely irrelevant to the discussion at hand. I also don't know what an immediate guest is -- is there any other kind? I don't think exclude_hv, exclude_host, and exclude_guest are directly tied to a single CPU mode. The only 'modes' you need to consider for Linux are 'guest' and 'host' when Linux can run VMs and, 'self' and 'hypervisor' when Linux is a guest. When Linux can run VMs, you count EL2 events when exclude_host is not set. (When Linux is a guest, and you set/clear exclude_hv, for this to work, you need some way of informing your hypervisor that you want to know about events happening in the hypervisor. This could be a PV interface, or maybe this can work by the guest setting/clearing the NSH bit in its virtual PMU registers, which then amusingly can get translated into actually counting in EL1/EL2 (non-VHE) or EL2 (VHE) by KVM's PMU emulation code. The method used is specific to the hypervisor used, but not specific to whether the hypervisor is type-1 or type-2.) Thanks, Christoffer _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel