From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD1EBC43331 for ; Thu, 7 Nov 2019 14:52:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 894252187F for ; Thu, 7 Nov 2019 14:52:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1733250AbfKGOwS (ORCPT ); Thu, 7 Nov 2019 09:52:18 -0500 Received: from foss.arm.com ([217.140.110.172]:57500 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729450AbfKGOwS (ORCPT ); Thu, 7 Nov 2019 09:52:18 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7742731B; Thu, 7 Nov 2019 06:52:17 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3DD573F71A; Thu, 7 Nov 2019 06:52:16 -0800 (PST) Date: Thu, 7 Nov 2019 14:52:14 +0000 From: Mark Rutland To: Ganapatrao Kulkarni Cc: Ganapatrao Prabhakerrao Kulkarni , "linux-doc@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "peterz@infradead.org" , "mingo@redhat.com" , "will@kernel.org" , "corbet@lwn.net" Subject: Re: [PATCH 1/2] perf/core: Adding capability to disable PMUs event multiplexing Message-ID: <20191107145213.GB6888@lakrids.cambridge.arm.com> References: <1573002091-9744-1-git-send-email-gkulkarni@marvell.com> <1573002091-9744-2-git-send-email-gkulkarni@marvell.com> <20191106112810.GA50610@lakrids.cambridge.arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.1+11 (2f07cb52) (2018-12-01) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Nov 06, 2019 at 03:28:46PM -0800, Ganapatrao Kulkarni wrote: > Hi Peter, Mark, > > On Wed, Nov 6, 2019 at 3:28 AM Mark Rutland wrote: > > > > On Wed, Nov 06, 2019 at 01:01:40AM +0000, Ganapatrao Prabhakerrao Kulkarni wrote: > > > When PMUs are registered, perf core enables event multiplexing > > > support by default. There is no provision for PMUs to disable > > > event multiplexing, if PMUs want to disable due to unavoidable > > > circumstances like hardware errata etc. > > > > > > Adding PMU capability flag PERF_PMU_CAP_NO_MUX_EVENTS and support > > > to allow PMUs to explicitly disable event multiplexing. > > > > Even without multiplexing, this PMU activity can happen when switching > > tasks, or when creating/destroying events, so as-is I don't think this > > makes much sense. > > > > If there's an erratum whereby heavy access to the PMU can lockup the > > core, and it's possible to workaround that by minimzing accesses, that > > should be done in the back-end PMU driver. > > As said in errata, If there are heavy access to memory like stream > application running and along with that if PMU control registers are > also accessed frequently, then CPU lockup is seen. Ok. So the issue is the frequency of access to those registers. Which registers does that apply to? Is this the case for only reads, only writes, or both? Does the frequency of access actually matter, or is is just more likely that we see the issue with a greater number of accesses? i.e the increased frequency increases the probability of hitting the issue. I'd really like a better description of the HW issue here. > I ran perf stat with 4 events of thuderx2 PMU as well as with 6 events > for stream application. > For 4 events run, there is no event multiplexing, where as for 6 > events run the events are multiplexed. > > For 4 event run: > No of times pmu->add is called: 10 > No of times pmu->del is called: 10 > No of times pmu->read is called: 310 > > For 6 events run: > No of times pmu->add is called: 5216 > No of times pmu->del is called: 5216 > No of times pmu->read is called: 5216 > > Issue happens when the add and del are called too many times as seen > with 6 event case. Sure, but I can achieve similar by creating/destroying events in a loop. Multiplexing is _one_ way to cause this behaviour, but it's not the _only_ way. > The PMU hardware control registers are programmed when add and del > functions are called. > For pmu->read no issues since no h/w issue with the data path. As above, can you please describe the hardware conditions more thoroughly? > This is UNCORE driver, not sure context switch has any influence on this? I believe that today it's possible for this to happen for cgroup events, as non-sensical as it may be to have a cgroup-bound uncore PMU event. > Please suggest me, how can we fix this in back-end PMU driver without > any perf core help? In order to do so, I need a better explanation of the underlying hardware issue. Thanks, Mark. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78B04C5DF60 for ; Thu, 7 Nov 2019 14:52:25 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3E2D62178F for ; Thu, 7 Nov 2019 14:52:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="SjTSrZsA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E2D62178F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=zcx+n3WgX8QN+P3GOqK0PukxHoxGmCf1T2zzltPtw4s=; b=SjTSrZsA9HcQYi b59LDuXBMRJH2fcdawPBNHjIx4cMLnT2JTb01pbxGG/erK13Pc61kmxpA24Gr+eoYDWmNqwWjIVWB MbjBKgWGbQ7DDNNaTYJKfylH4Uk3psS75Q5C1LZZdqdK1p4eJffTFhziM826Sqesj2RphVHsm53XU 7vuXiVSPmxs646io1SBVjRmbwboeoYrBNYJtwVNZCZCnqj50ZT4Xy0fYTd1+iE74wi91/bN8a2TtF 78N6bc0e5CMc20a/7NGfs/dwyb4pbKsUHPT3smIdSYntg1Qbmq0qbl1H+p0Uu/pr3jZEI5rflJA7i 5+M4lNaesel5UwX7UY3A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iSj96-000141-LR; Thu, 07 Nov 2019 14:52:24 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iSj92-000131-58 for linux-arm-kernel@lists.infradead.org; Thu, 07 Nov 2019 14:52:21 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7742731B; Thu, 7 Nov 2019 06:52:17 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3DD573F71A; Thu, 7 Nov 2019 06:52:16 -0800 (PST) Date: Thu, 7 Nov 2019 14:52:14 +0000 From: Mark Rutland To: Ganapatrao Kulkarni Subject: Re: [PATCH 1/2] perf/core: Adding capability to disable PMUs event multiplexing Message-ID: <20191107145213.GB6888@lakrids.cambridge.arm.com> References: <1573002091-9744-1-git-send-email-gkulkarni@marvell.com> <1573002091-9744-2-git-send-email-gkulkarni@marvell.com> <20191106112810.GA50610@lakrids.cambridge.arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.1+11 (2f07cb52) (2018-12-01) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191107_065220_284210_0F804D4B X-CRM114-Status: GOOD ( 20.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ganapatrao Prabhakerrao Kulkarni , "linux-doc@vger.kernel.org" , "peterz@infradead.org" , "corbet@lwn.net" , "linux-kernel@vger.kernel.org" , "mingo@redhat.com" , "will@kernel.org" , "linux-arm-kernel@lists.infradead.org" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Nov 06, 2019 at 03:28:46PM -0800, Ganapatrao Kulkarni wrote: > Hi Peter, Mark, > > On Wed, Nov 6, 2019 at 3:28 AM Mark Rutland wrote: > > > > On Wed, Nov 06, 2019 at 01:01:40AM +0000, Ganapatrao Prabhakerrao Kulkarni wrote: > > > When PMUs are registered, perf core enables event multiplexing > > > support by default. There is no provision for PMUs to disable > > > event multiplexing, if PMUs want to disable due to unavoidable > > > circumstances like hardware errata etc. > > > > > > Adding PMU capability flag PERF_PMU_CAP_NO_MUX_EVENTS and support > > > to allow PMUs to explicitly disable event multiplexing. > > > > Even without multiplexing, this PMU activity can happen when switching > > tasks, or when creating/destroying events, so as-is I don't think this > > makes much sense. > > > > If there's an erratum whereby heavy access to the PMU can lockup the > > core, and it's possible to workaround that by minimzing accesses, that > > should be done in the back-end PMU driver. > > As said in errata, If there are heavy access to memory like stream > application running and along with that if PMU control registers are > also accessed frequently, then CPU lockup is seen. Ok. So the issue is the frequency of access to those registers. Which registers does that apply to? Is this the case for only reads, only writes, or both? Does the frequency of access actually matter, or is is just more likely that we see the issue with a greater number of accesses? i.e the increased frequency increases the probability of hitting the issue. I'd really like a better description of the HW issue here. > I ran perf stat with 4 events of thuderx2 PMU as well as with 6 events > for stream application. > For 4 events run, there is no event multiplexing, where as for 6 > events run the events are multiplexed. > > For 4 event run: > No of times pmu->add is called: 10 > No of times pmu->del is called: 10 > No of times pmu->read is called: 310 > > For 6 events run: > No of times pmu->add is called: 5216 > No of times pmu->del is called: 5216 > No of times pmu->read is called: 5216 > > Issue happens when the add and del are called too many times as seen > with 6 event case. Sure, but I can achieve similar by creating/destroying events in a loop. Multiplexing is _one_ way to cause this behaviour, but it's not the _only_ way. > The PMU hardware control registers are programmed when add and del > functions are called. > For pmu->read no issues since no h/w issue with the data path. As above, can you please describe the hardware conditions more thoroughly? > This is UNCORE driver, not sure context switch has any influence on this? I believe that today it's possible for this to happen for cgroup events, as non-sensical as it may be to have a cgroup-bound uncore PMU event. > Please suggest me, how can we fix this in back-end PMU driver without > any perf core help? In order to do so, I need a better explanation of the underlying hardware issue. Thanks, Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel