From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBAEAC433ED for ; Tue, 27 Apr 2021 15:50:01 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 21A22613C9 for ; Tue, 27 Apr 2021 15:50:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 21A22613C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=YffzCO9b/BVUNaIG2K0h2TJjg7yPX+nqFuHw1t2Br48=; b=q+uD8t32nJq9wpGb80uCz6GxT oLEsVdC3Zm03RwHvldjcchYQ2cvGGEKgVDkI7a4Mh3VKmHDI96zwAavGNCHep2lzMqnlyhSl6S38G gOcYqgjDltzCzadxbbR4AU5CPaXLS5qRrmzgTi38k445iN73qWlUKHpolrhi0SSG5FiIRhuVbjc9h Uc9zajiR1IvpYw02FUuFTjmrmi1j6Okas52LRQAGpNmS5K9orJ2iGqhqMzdb/DCvBRv/tARKF8w/0 AY+dTd8PvBD0sP33Fm7nJON/W7Q4JCtSJK6YzbrbLE3P1ofmomJcNKqpzocttRday61LU0/gbcVGr DO+fWNorw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lbPwV-001rO8-Bw; Tue, 27 Apr 2021 15:48:07 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lbPwR-001rO3-Q9 for linux-arm-kernel@desiato.infradead.org; Tue, 27 Apr 2021 15:48:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=In-Reply-To:Content-Type:MIME-Version :References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=PcuYm577EI6pyFM/O/WeAUmp1QeVPg6jL856dk+e7V8=; b=QkDW0mmWn+1SUtArR5tLFOu63J gfBbv59nCeJBl604bLIkWBH1XXJNGHjyyB0j2rNOGqFObfJ5yr0wFubxvNGYke840KCyoPa09+jKB Mi0t3sMrJJzN0M1Z1y2hoBzx9eXyq0Ng2j/vl8BIySXFe86y1xjaStX5MZvG4QWUOFS8rpswcnuQQ FpQCfSHHDelZ67iz5Qk0SgSdonTf3+s3Y3QtSy0UlGJvpecCeV+RkVHuVsUHCHKv5vTEL4XfC1YnB fLOpHYjgOOG24uCPQgDpyHuHF+jVASu4GQ8oZXEWkrsONrUh05mbS02w/2OLSUfJa8CLmEPKmAOIR 9Cqgn7cA==; Received: from mail-pg1-x530.google.com ([2607:f8b0:4864:20::530]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lbPwO-00Gpqy-0Y for linux-arm-kernel@lists.infradead.org; Tue, 27 Apr 2021 15:48:02 +0000 Received: by mail-pg1-x530.google.com with SMTP id m37so53297pgb.8 for ; Tue, 27 Apr 2021 08:47:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=PcuYm577EI6pyFM/O/WeAUmp1QeVPg6jL856dk+e7V8=; b=D7lRIfoAEyD5MrfPoUNiGgar6ELC3OogakZOVXo6IQeRidBFL6j0WsXPmup0hoQ7rh ejwZV6Xbxp+5njAcLc8Emw4ZW+QFv6UOqbRALavTcHi0h9tkVduwhPYgdXJk/VF2lLYv PQAntZfr9BSUYctnv8gIG1NSpkR3mOM4iS36GW8useVQfOdZK0LDmQdnUtXEo/NrsRhQ nhZMbcVh2MPpZ1m4ZImxs5BDCTrGAiUfhIeijbvbMlJAMl66YS9kV5AxU+34dXR2mSc4 uXrw/jlLBDLSNlOvwpMeWxZa9Y1I8o2jYEH1HEsOMxKczAksSuf0H3C25FsEma4nNH19 MtBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=PcuYm577EI6pyFM/O/WeAUmp1QeVPg6jL856dk+e7V8=; b=lL8V3LoAoZEKqHzZxakizJwrrhPGs1S+6jXq1qiJdrxUG8PAe2KYF66sJMHAKr4OHC mXMGHLQYPX34XsEtZlFjaCVb9AfzX+KEfXFHbqohl/6AD7AjtJWGYKPyawAK/Lw8aJfN ykzJtZCW/Av9W6JRW9miSTCPoo7Ja5XHPQR73ajwB44SdwKnFiLMHWtZ43FmTtHyVmXj z+FRRuuKbr77uepl82JY+YvNQ2bqwliP4ixVAm1nE9QduAlUamoAFYULX+GR+jovuBTS qAvLgVNKK1SOvjZJnS3i114bgsdNzNYqlyPhKzveT+oNLjjs0fu/eBlsCTPEUyf7GZLz SFwA== X-Gm-Message-State: AOAM531T81bCAqIuZc+4DXBlZyNCT0F6vwWtRFxKTcOYa/udK0dvvdbs /J2uN3AFRVnnLUpnHe/CgXAP1w== X-Google-Smtp-Source: ABdhPJyBLvunX4R5PV5zjH80BSxLTM6zIlP6q/CBRs6jcF9GwoE7QnmGPA/UBXCTzhfpxNJAJqZRzA== X-Received: by 2002:a65:6643:: with SMTP id z3mr21930391pgv.387.1619538475612; Tue, 27 Apr 2021 08:47:55 -0700 (PDT) Received: from xps15 (S0106889e681aac74.cg.shawcable.net. [68.147.0.187]) by smtp.gmail.com with ESMTPSA id gx21sm2836240pjb.29.2021.04.27.08.47.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Apr 2021 08:47:47 -0700 (PDT) Date: Tue, 27 Apr 2021 09:47:46 -0600 From: Mathieu Poirier To: Mike Leach Cc: "coresight@lists.linaro.org" , Al Grant , Daniel Kiss , "denik@google.com" , "linux-arm-kernel@lists.infradead.org" Subject: Re: [PATCH 0/4] coresight: Add ETR-PERF polling. Message-ID: <20210427154746.GA1422814@xps15> References: <20210421120413.3110775-1-daniel.kiss@arm.com> <20210426175425.GA1391779@xps15> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210427_084800_091369_F4AEDEE6 X-CRM114-Status: GOOD ( 68.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Good day Mike, On Tue, Apr 27, 2021 at 03:41:01PM +0100, Mike Leach wrote: > Hi Mathieu, > > I thought I'd add a little backgound to what has been said so far... > > On Tue, 27 Apr 2021 at 11:43, Al Grant wrote: > > > > > Hi Daniel, > > > > > > On Wed, Apr 21, 2021 at 02:04:09PM +0200, Daniel Kiss wrote: > > > > This series adds a feature to ETR-PERF that sync the ETR buffer to > > > > perf periodically. This is really handy when the system wide trace is > > > > used because in this case the perf won't sync during the trace. In a > > > > per-thread setup the traced program might not go to the kernel > > > > frequvently enought to collect trace. Polling helps in both usecases. Can be > > > used with strobing. > > > > Tuning polling period is challanging, I'm working on an additional > > > > patch that adds some metrics to help tune the polling period. > > > > > > > > > > Suzuki and Leo have already commented on a number of problems with this set > > > and as such I will concentrate on the general idea. > > > > > > Over the years we have thought long and hard about fixing the overflow issues > > > created by the lack of interrupt when a sink gets full, installing a timer to empty > > > the sink buffer at regular intervals is one of them. Ultimately we haven't moved > > > forward with the idea because it requires to stop the sink when an event is > > > active, something that introduces more trace data loss. > > > > > > To me this kind of interval snapshot should be achieved using Mike's new > > > strobing feature that came bundled with the complex configuration framework, > > > available on next-ETE-TRBE[1]. I will rebase that branch to 5.13-rc1 when it is > > > released in a couple of weeks from now. > > > > It's important to understand what strobing is. It acts internally to the ETM > > and switches the ETM on for a time and then off for a time. It is as the > > name suggests, like a stroboscope (or a lighthouse). > > > > There is no synchronization between the on-periods of different ETMs. > > When you have multiple ETMs funnelling into a common ETR, strobing > > does not guarantee you a window where you can safely harvest the buffer. > > It achieves a reduction in the overall bandwidth of trace being dumped > > into the buffer, and there may be times when no trace is being written > > at all because all the ETMs are in their off-period. > > > > At worst, it may create a false sense of security - tests that consistently > > fail without strobing, may pass often enough with strobing to create the > > impression that strobing has solved the problem. But these tests are also > > likely to fail eventually with strobing. To fix this problem without > > disabling either ETR or ETMs you would have to guarantee that you can > > harvest the ETR buffer in less time than it takes to fill it. That would need > > very careful quntitative arguments to be made about: > > > > - the rate of trace generation by each ETM (as modified by strobing) > > > > - the number of ETMs writing into the buffer > > > > - the time available to the kernel to harvest the buffer > > > > So if there are 10 ETMs generating trace at average 1Gb/s into a 1Mb > > buffer, the buffer will fill in 100us, and that gives the kernel 100us to > > harvest the buffer before its read pointer is caught up by the ETR's > > advancing write pointer. If strobing is used to reduce average ETM rate > > to 100Mb/s the kernel has 1ms to read the buffer, and so on. In short > > the kernel must *guarantee* a minimum readout rate equal to the > > maximum aggregate write rate of the ETMs. But can the kernel > > guarantee any minimum readout rate at all? > > > > The alternative would be double-buffering the ETR, which we've > > also discussed - so while the kernel is harvesting the contents of one > > buffer, the ETR is writing (and possibly wrapping) the other. > > Some trace will still be lost but it does mean the kernel will be > > harvesting monotonically increasing sequences of trace, and won't be > > seeing artefacts from its reads colliding with the ETR's writes. > > > > Al > > > > As Al mentions, ETR polling is designed to solve a different issue > than ETM strobing. These two techniques can be used together or > separately. > > It was noticed by users that the amount of trace captured during a > given trace run would vary greatly even when tracing the same > application for the same length of time. Indeed, that problem is well known. > This was also found to be sensitive to process scheduling - frequent > re-scheds did seem to result in more frequent ETR updates and more > trace data collected. If perf does not wake up during a trace run then > the ETR may wrap mulitple times and all the data will be a single > buffer biased towards the end of the trace session. > Right. > ETR polling is designed to ensure that more trace data is collected > consistently across the whole of the trace session. There are issues > of course, with stopping collection without stopping the sources. - > shared to some extent by the ETE / TRBE combination. > This can result in incomplete packets and other trace discontinuities. > For this reason it is necessary to ensure that the decoder is > restarted for each block of trace captured - which is where the patch > set from James that does this using AUX records in perf to correctly > split the AUXTRACE records into valid blocks is needed. I am still waiting for a new revision from James. > > In summary:- > 1) ETM strobing samples trace to allow greater coverage of the program > being traced for a given buffer. This is useful when building > statistical profiles such as for AutoFDO > 2) ETR polling ensures that more trace is collected across the entire > trace session - seeking to reduce inconsistent capture volumes. I am not convinced disabling a sink to collect traces while an event is active is the right way to go. To me it will add (more) complexity to the coresight subsystem for very little gains, if any. If I remember correctly Leo brought forward the exact same idea about a year ago and after discussion, we all agreed the benefit would not be important enough to offset the drawbacks. As usual I am open to discussion and my opinion is not set in stone. But as I mentioned I worry the feature will increase complexity in the driver and produce dubious results. And we also have to factor in usability which, as Al pointed, out will be a problem. > 3) Use AUX records to split the AUXTRACE buffer into valid capture > blocks and reset the decoder at the start of these blocks. This is > essential for ETE+TRBE, the ETR polling, and systems where we are > seeing hardware errata around the flush process causing similar > spurious packets. (an alternative for the ETR polling / flush errata > might be to insert barrier packets to force a decoder reset for every > ETR block copied to the perf buffer - but this does not work for > ETE/TRBE that uses no CoreSight formatted framing). > > Regards > > Mike > > > > > > > > > > Thanks, > > > Mathieu > > > > > > PS: Always run your work through checkpatch.pl before sending a patchset for > > > review. > > > > > > [1]. > > > https://git.kernel.org/pub/scm/linux/kernel/git/coresight/linux.git/log/?h=next- > > > ETE-TRBE > > > > > > > Daniel Kiss (4): > > > > coresight: tmc-etr: Advance buffer pointer in sync buffer. > > > > coresight: tmc-etr: Track perf handler. > > > > coresight: etm-perf: Export etm_event_cpu_path. > > > > coresight: Add ETR-PERF polling. > > > > > > > > .../testing/sysfs-bus-coresight-devices-tmc | 8 + > > > > drivers/hwtracing/coresight/Makefile | 2 +- > > > > .../hwtracing/coresight/coresight-etm-perf.c | 10 +- > > > > .../hwtracing/coresight/coresight-etm-perf.h | 1 + > > > > .../coresight/coresight-etr-perf-polling.c | 316 ++++++++++++++++++ > > > > .../coresight/coresight-etr-perf-polling.h | 42 +++ > > > > .../hwtracing/coresight/coresight-tmc-core.c | 2 + > > > > .../hwtracing/coresight/coresight-tmc-etr.c | 22 +- > > > > drivers/hwtracing/coresight/coresight-tmc.h | 2 + > > > > 9 files changed, 401 insertions(+), 4 deletions(-) create mode > > > > 100644 drivers/hwtracing/coresight/coresight-etr-perf-polling.c > > > > create mode 100644 > > > > drivers/hwtracing/coresight/coresight-etr-perf-polling.h > > > > > > > > -- > > > > 2.25.1 > > > > > > > _______________________________________________ > > > CoreSight mailing list > > > CoreSight@lists.linaro.org > > > https://lists.linaro.org/mailman/listinfo/coresight > > > > -- > Mike Leach > Principal Engineer, ARM Ltd. > Manchester Design Centre. UK _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel