From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44F7FC433F5 for ; Wed, 23 Mar 2022 10:41:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243586AbiCWKmo (ORCPT ); Wed, 23 Mar 2022 06:42:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234917AbiCWKmk (ORCPT ); Wed, 23 Mar 2022 06:42:40 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 1F4336C95D; Wed, 23 Mar 2022 03:41:10 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D4A86D6E; Wed, 23 Mar 2022 03:41:09 -0700 (PDT) Received: from [10.1.197.1] (ewhatever.cambridge.arm.com [10.1.197.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DA52B3F73B; Wed, 23 Mar 2022 03:41:07 -0700 (PDT) Message-ID: Date: Wed, 23 Mar 2022 10:41:06 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: [PATCH 00/10] coresight: Add new API to allocate trace source ID values Content-Language: en-US To: Mike Leach Cc: coresight@lists.linaro.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, mathieu.poirier@linaro.org, peterz@infradead.org, mingo@redhat.com, acme@kernel.org, linux-perf-users@vger.kernel.org, leo.yan@linaro.org, James Clark References: <20220308205000.27646-1-mike.leach@linaro.org> <639c902b-535e-8627-de04-9bcbbdc98ed5@arm.com> <17215ebc-dbf2-81ca-7ef4-b4ebc3cb9d28@arm.com> From: Suzuki Kuruppassery Poulose In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Mike On 23/03/2022 10:07, Mike Leach wrote: > Hi Suzuki, > > On Tue, 22 Mar 2022 at 18:52, Suzuki K Poulose wrote: >> >> Hi Mike >> >> On 22/03/2022 14:27, Mike Leach wrote: >>> Hi Suzuki >>> >>> On Tue, 22 Mar 2022 at 12:35, Suzuki Kuruppassery Poulose >>> wrote: >>>> >>>> On 22/03/2022 11:38, Mike Leach wrote: >>>>> HI Suzuki, >>>>> >>>>> On Tue, 22 Mar 2022 at 10:43, Suzuki Kuruppassery Poulose >>>>> wrote: >>>>>> >>>>>> + Cc: James Clark >>>>>> >>>>>> Hi Mike, >>>>>> >>>>>> On 08/03/2022 20:49, Mike Leach wrote: >>>>>>> The current method for allocating trace source ID values to sources is >>>>>>> to use a fixed algorithm for CPU based sources of (cpu_num * 2 + 0x10). >>>>>>> The STM is allocated ID 0x1. >>>>>>> >>>>>>> This fixed algorithm is used in both the CoreSight driver code, and by >>>>>>> perf when writing the trace metadata in the AUXTRACE_INFO record. >>>>>>> >>>>>>> The method needs replacing as currently:- >>>>>>> 1. It is inefficient in using available IDs. >>>>>>> 2. Does not scale to larger systems with many cores and the algorithm >>>>>>> has no limits so will generate invalid trace IDs for cpu number > 44. >>>>>> >>>>>> Thanks for addressing this issue. >>>>>> >>>>>>> >>>>>>> Additionally requirements to allocate additional system IDs on some >>>>>>> systems have been seen. >>>>>>> >>>>>>> This patch set introduces an API that allows the allocation of trace IDs >>>>>>> in a dynamic manner. >>>>>>> >>>>>>> Architecturally reserved IDs are never allocated, and the system is >>>>>>> limited to allocating only valid IDs. >>>>>>> >>>>>>> Each of the current trace sources ETM3.x, ETM4.x and STM is updated to use >>>>>>> the new API. >>>>>>> >>>>>>> perf handling is changed so that the ID associated with the CPU is read >>>>>>> from sysfs. The ID allocator is notified when perf events start and stop >>>>>>> so CPU based IDs are kept constant throughout any perf session. >>>>>>> >>>>>>> For the ETMx.x devices IDs are allocated on certain events >>>>>>> a) When using sysfs, an ID will be allocated on hardware enable, and freed >>>>>>> when the sysfs reset is written. >>>>>>> b) When using perf, ID is allocated on hardware enable, and freed on >>>>>>> hardware disable. >>>>>>> >>>>>>> For both cases the ID is allocated when sysfs is read to get the current >>>>>>> trace ID. This ensures that consistent decode metadata can be extracted >>>>>>> from the system where this read occurs before device enable. >>>>>> >>>>>> >>>>>>> >>>>>>> Note: This patchset breaks backward compatibility for perf record. >>>>>>> Because the method for generating the AUXTRACE_INFO meta data has >>>>>>> changed, using an older perf record will result in metadata that >>>>>>> does not match the trace IDs used in the recorded trace data. >>>>>>> This mismatch will cause subsequent decode to fail. Older versions of >>>>>>> perf will still be able to decode data generated by the updated system. >>>>>> >>>>>> I have some concerns over this and the future plans for the dynamic >>>>>> allocation per sink. i.e., we are breaking/modifying the perf now to >>>>>> accommodate the dynamic nature of the trace id of a given CPU/ETM. >>>>> >>>>> I don't beleive we have a choice for this - we cannot retain what is >>>>> an essentially broken allocation mechanism. >>>>> >>>> >>>> I completely agree and I am happy with the current step by step approach >>>> of moving to a dynamic allocation scheme. Apologies, this wasn't >>>> conveyed appropriately. >>>> >>>>>> The proposed approach of exposing this via sysfs may (am not sure if >>>>>> this would be the case) break for the trace-id per sink change, as a >>>>>> sink could assign different trace-id for a CPU depending. >>>>>> >>>>> >>>>> If a path exists between a CPU and a sink - the current framework as >>>>> far as I can tell would not allow for a new path to be set up between >>>>> the cpu and another sink. >>>> >>>> e.g, if we have concurrent perf sessions, in the future with sink based >>>> allocation : >>>> >>>> perf record -e cs_etm/@sink1/... payload1 >>>> perf record -e cs_etm/@sink2/... payload2 >>>> perf record -e cs_etm// ... payload3 >>>> >>>> The trace id allocated for first session for CPU0 *could* be different >>>> from that of the second or the third. >>> >>> If these sessions run concurrently then the same Trace ID will be used >>> for CPU0 for all the sessions. >>> We ensure this by notifications that a cs_etm session is starting and >>> stopping - and keep a refcount. >> >> The scheme is fine now, with a global trace-id map. But with per-sink >> allocation, this could cause problems. >> >> e.g., there could be a situation where: >> >> trace_id[CPU0][sink0] == trace_id[CPU1][sink1] >> >> So if we have a session where both CPU0 and CPU1 trace to a common sink, >> we get the trace mixed with no way of splitting them. As the perf will >> read the trace-id for CPU0 from that of sink0 and CPU1 from sink1. > > I think we need to consider the CoreSight hardware topology here. > > Any CPUx that can trace to a sink reachable by another CPUy must > always get trace IDs from the same pool as CPUy. > > Consider the options for multi sink topologies:- > > CPU0->funnel0->ETF->ETR > CPU1--+^ > > Clearly - in-line sinks can never have simultaneous independent > sessions - the session into ETR traces through ETF > > Now we could have replicators / programmable replicators - > > ATB->Prog replicator->ETR0 > +->ETR1 > > however programmable replicators use trace ID for filtering - this is > effectively a single sink on the input, so once again the Trace IDs > must come from the same pool. > > Now, we could have independent per cluster / socket topology > Cluster 0 > CPU0->funnel0->ETF0->ETR0 > CPU1--+^ > > Cluster 1 > CPU2->funnel1->ETF1->ETR1 > CPU3--+^ > What if the ETR was a common one ? i.e. Cluster0 CPU0 -> ETF0 ..... \ Cluster1 -- ETR0 CPU1 -> ETF1 ..... / And lets there are 3 perf sessions in parallel, started in the order below perf record -e cs_etm/@etf0/ app1 # CPU0 gets a trace-id[etf0] -> 0x50 perf record -e cs_etm/@etf1/ app2 # CPU1 gets a trace-id[etf1] -> 0x50 perf record -e cs_etm/@etr/ app3 # CPU0 and CPU1 both use the existing trace ids from the allocations. So, when app3 threads are scheduled on CPU0 & CPU1, we get the trace in ETR with the same trace-id of 0x50. Suzuki From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 660FFC433F5 for ; Wed, 23 Mar 2022 10:42:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=spOxhOPTC/q8nzc2HDYgFMWiHzuXWkTqEll5j91mBAg=; b=L/EI31dS/N07uz Awt4dCM8wI/KruNIYUdWeXus68hS7w790e0++2Prq0IBiaPSYOw7OsgPuMN3Z5IYhP9RcVQ51HNQ0 mL6yjhl5aN5/IuqbLBMJiTo1FI2oh2F9UGjxchdWiTy1N0nkFpbsesSa4r9ypzg1f9W2QywUqE33i Mj62HPsSO9Dojxx2XdCADeVGYuDowNrL3HokumlTVGgHT7zG1UlIoSGLPjRF2er3LY65dzqBMPgyZ wd6XtzGEJJOHExy+0Lj236EZU6pfMjFQKEsJ0Idb95W3P9oPcgAGLebhxTqUsa28bX8tCzmz5Iv0j 2RslHd3JyJxKtl5BHUSg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nWyQU-00DLzk-Md; Wed, 23 Mar 2022 10:41:14 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nWyQQ-00DLyY-TF for linux-arm-kernel@lists.infradead.org; Wed, 23 Mar 2022 10:41:12 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D4A86D6E; Wed, 23 Mar 2022 03:41:09 -0700 (PDT) Received: from [10.1.197.1] (ewhatever.cambridge.arm.com [10.1.197.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DA52B3F73B; Wed, 23 Mar 2022 03:41:07 -0700 (PDT) Message-ID: Date: Wed, 23 Mar 2022 10:41:06 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: [PATCH 00/10] coresight: Add new API to allocate trace source ID values Content-Language: en-US To: Mike Leach Cc: coresight@lists.linaro.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, mathieu.poirier@linaro.org, peterz@infradead.org, mingo@redhat.com, acme@kernel.org, linux-perf-users@vger.kernel.org, leo.yan@linaro.org, James Clark References: <20220308205000.27646-1-mike.leach@linaro.org> <639c902b-535e-8627-de04-9bcbbdc98ed5@arm.com> <17215ebc-dbf2-81ca-7ef4-b4ebc3cb9d28@arm.com> From: Suzuki Kuruppassery Poulose In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220323_034111_081871_C8E814AD X-CRM114-Status: GOOD ( 32.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Mike On 23/03/2022 10:07, Mike Leach wrote: > Hi Suzuki, > > On Tue, 22 Mar 2022 at 18:52, Suzuki K Poulose wrote: >> >> Hi Mike >> >> On 22/03/2022 14:27, Mike Leach wrote: >>> Hi Suzuki >>> >>> On Tue, 22 Mar 2022 at 12:35, Suzuki Kuruppassery Poulose >>> wrote: >>>> >>>> On 22/03/2022 11:38, Mike Leach wrote: >>>>> HI Suzuki, >>>>> >>>>> On Tue, 22 Mar 2022 at 10:43, Suzuki Kuruppassery Poulose >>>>> wrote: >>>>>> >>>>>> + Cc: James Clark >>>>>> >>>>>> Hi Mike, >>>>>> >>>>>> On 08/03/2022 20:49, Mike Leach wrote: >>>>>>> The current method for allocating trace source ID values to sources is >>>>>>> to use a fixed algorithm for CPU based sources of (cpu_num * 2 + 0x10). >>>>>>> The STM is allocated ID 0x1. >>>>>>> >>>>>>> This fixed algorithm is used in both the CoreSight driver code, and by >>>>>>> perf when writing the trace metadata in the AUXTRACE_INFO record. >>>>>>> >>>>>>> The method needs replacing as currently:- >>>>>>> 1. It is inefficient in using available IDs. >>>>>>> 2. Does not scale to larger systems with many cores and the algorithm >>>>>>> has no limits so will generate invalid trace IDs for cpu number > 44. >>>>>> >>>>>> Thanks for addressing this issue. >>>>>> >>>>>>> >>>>>>> Additionally requirements to allocate additional system IDs on some >>>>>>> systems have been seen. >>>>>>> >>>>>>> This patch set introduces an API that allows the allocation of trace IDs >>>>>>> in a dynamic manner. >>>>>>> >>>>>>> Architecturally reserved IDs are never allocated, and the system is >>>>>>> limited to allocating only valid IDs. >>>>>>> >>>>>>> Each of the current trace sources ETM3.x, ETM4.x and STM is updated to use >>>>>>> the new API. >>>>>>> >>>>>>> perf handling is changed so that the ID associated with the CPU is read >>>>>>> from sysfs. The ID allocator is notified when perf events start and stop >>>>>>> so CPU based IDs are kept constant throughout any perf session. >>>>>>> >>>>>>> For the ETMx.x devices IDs are allocated on certain events >>>>>>> a) When using sysfs, an ID will be allocated on hardware enable, and freed >>>>>>> when the sysfs reset is written. >>>>>>> b) When using perf, ID is allocated on hardware enable, and freed on >>>>>>> hardware disable. >>>>>>> >>>>>>> For both cases the ID is allocated when sysfs is read to get the current >>>>>>> trace ID. This ensures that consistent decode metadata can be extracted >>>>>>> from the system where this read occurs before device enable. >>>>>> >>>>>> >>>>>>> >>>>>>> Note: This patchset breaks backward compatibility for perf record. >>>>>>> Because the method for generating the AUXTRACE_INFO meta data has >>>>>>> changed, using an older perf record will result in metadata that >>>>>>> does not match the trace IDs used in the recorded trace data. >>>>>>> This mismatch will cause subsequent decode to fail. Older versions of >>>>>>> perf will still be able to decode data generated by the updated system. >>>>>> >>>>>> I have some concerns over this and the future plans for the dynamic >>>>>> allocation per sink. i.e., we are breaking/modifying the perf now to >>>>>> accommodate the dynamic nature of the trace id of a given CPU/ETM. >>>>> >>>>> I don't beleive we have a choice for this - we cannot retain what is >>>>> an essentially broken allocation mechanism. >>>>> >>>> >>>> I completely agree and I am happy with the current step by step approach >>>> of moving to a dynamic allocation scheme. Apologies, this wasn't >>>> conveyed appropriately. >>>> >>>>>> The proposed approach of exposing this via sysfs may (am not sure if >>>>>> this would be the case) break for the trace-id per sink change, as a >>>>>> sink could assign different trace-id for a CPU depending. >>>>>> >>>>> >>>>> If a path exists between a CPU and a sink - the current framework as >>>>> far as I can tell would not allow for a new path to be set up between >>>>> the cpu and another sink. >>>> >>>> e.g, if we have concurrent perf sessions, in the future with sink based >>>> allocation : >>>> >>>> perf record -e cs_etm/@sink1/... payload1 >>>> perf record -e cs_etm/@sink2/... payload2 >>>> perf record -e cs_etm// ... payload3 >>>> >>>> The trace id allocated for first session for CPU0 *could* be different >>>> from that of the second or the third. >>> >>> If these sessions run concurrently then the same Trace ID will be used >>> for CPU0 for all the sessions. >>> We ensure this by notifications that a cs_etm session is starting and >>> stopping - and keep a refcount. >> >> The scheme is fine now, with a global trace-id map. But with per-sink >> allocation, this could cause problems. >> >> e.g., there could be a situation where: >> >> trace_id[CPU0][sink0] == trace_id[CPU1][sink1] >> >> So if we have a session where both CPU0 and CPU1 trace to a common sink, >> we get the trace mixed with no way of splitting them. As the perf will >> read the trace-id for CPU0 from that of sink0 and CPU1 from sink1. > > I think we need to consider the CoreSight hardware topology here. > > Any CPUx that can trace to a sink reachable by another CPUy must > always get trace IDs from the same pool as CPUy. > > Consider the options for multi sink topologies:- > > CPU0->funnel0->ETF->ETR > CPU1--+^ > > Clearly - in-line sinks can never have simultaneous independent > sessions - the session into ETR traces through ETF > > Now we could have replicators / programmable replicators - > > ATB->Prog replicator->ETR0 > +->ETR1 > > however programmable replicators use trace ID for filtering - this is > effectively a single sink on the input, so once again the Trace IDs > must come from the same pool. > > Now, we could have independent per cluster / socket topology > Cluster 0 > CPU0->funnel0->ETF0->ETR0 > CPU1--+^ > > Cluster 1 > CPU2->funnel1->ETF1->ETR1 > CPU3--+^ > What if the ETR was a common one ? i.e. Cluster0 CPU0 -> ETF0 ..... \ Cluster1 -- ETR0 CPU1 -> ETF1 ..... / And lets there are 3 perf sessions in parallel, started in the order below perf record -e cs_etm/@etf0/ app1 # CPU0 gets a trace-id[etf0] -> 0x50 perf record -e cs_etm/@etf1/ app2 # CPU1 gets a trace-id[etf1] -> 0x50 perf record -e cs_etm/@etr/ app3 # CPU0 and CPU1 both use the existing trace ids from the allocations. So, when app3 threads are scheduled on CPU0 & CPU1, we get the trace in ETR with the same trace-id of 0x50. Suzuki _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel