From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AE956C433F5 for ; Tue, 11 Oct 2022 10:23:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hsZ1Z5UZF7AhTCs8o8JP+CwhgM6sjhDuV0/E9nB1XBI=; b=p82ynK10vIJJmX fIP72BMKrtUX9b0RE3UfSdKYEidty2EAog4nu/1Dkbarq1nXm1fx4TUuu+AuysJcnNk/2YHvSTp99 o7RjvVY3TmhpvqHO5Fya2kS4yCLbEb9i7ZKZnCPRn683FV2lpu2AB17AF2QSi5fVZiOT997eV15/+ F3HaB8BRV8BzoQpwFnXyRu6btmEF5Bcf/+slul2Xz81HzqCkmQ3LLw8jn6y45pe/bZxQg00v39Xph DJd+PoDy8Q/XnzoOu6sids7A97AH4GjoEfJZtOk2vl/E+91lJ2scbFuD2GywqsEjmtqjE83eXJgiW BkTaZWuy+g2m4axGBnGA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oiCP1-003zva-Uf; Tue, 11 Oct 2022 10:22:24 +0000 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oiCOy-003zur-A5 for linux-arm-kernel@lists.infradead.org; Tue, 11 Oct 2022 10:22:22 +0000 Received: by mail-pl1-x635.google.com with SMTP id n7so12818368plp.1 for ; Tue, 11 Oct 2022 03:22:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=QUMoVWsAtMoav1G38Z0NBtICcSa4s5t0l3sQsGKy1zo=; b=sm2o1yqlseQc0qK6Rd57YP82oQAya806xt8R2Jvi60nVxCPCEuDviHLCgMf4UatxJy NLlzmwrqHL2OHxnMULqpPiAQ1GKdW/zZ8sMbONpvmYe/ExYAFL6NRaLHucUGNZSpaH0W mOSvOPEfSCzt+ZEiiFrN5GWajkxgTwZhAK0ffEYm6ps/UseqzWbW0miFJ2ql7PBhTrya hZP8tSXKD8MIjzMu+m+BoUD9BhTfevASxR13+jci+6B8h0DMLN7jt8XExJeDTb1ZKTwv G3IhvgpIDIgD+LrXlfbKU44X06huWQJNo/JuR5ex4pg+0kh5WfK1s8UDN5jWDLkJ5/oH kwNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QUMoVWsAtMoav1G38Z0NBtICcSa4s5t0l3sQsGKy1zo=; b=W2T7YxsDP2iyz9MxwOZ7X8JRZeiaWRZgOs8TljNzu1kZSK+VOFJvaB0IohZ5hSOjnl KUyIfAIHlYgSC+/NLrCEAbU+dVNuV1dPU4hq9Bw1TNmtKtd+RbkWXoCSmUKvTSrn4ii+ bZ1pwBl5vwE4Ew4GlOYGYS3FhNFuZW2LKZbk0YoLwPEZt8ze89Ky3/crSU+FRG+To0i/ ADYCszRg6fWek74b0DwuubWBViYzwDJgcAklAFxOIXvEOgaGq168OINMpH9x7PAofUDK jk8hULNtOxEk/z46jVvr9Wk80nwF7jF6ij5fYSWQgz9RcdEXlwe7LHNcRYiujfR0E9Ca aNKA== X-Gm-Message-State: ACrzQf3Vlpi9qDJzpyiWszMYAYO4xmkAZc16bE99j6I4hKGhoaZ0RMXL pxNmJYm73c30Tj9ae5Vvsozf6tEuXisRFy9QprX9YQ== X-Google-Smtp-Source: AMsMyM5SBiGHU9OPu+NDA4WXMhFRBdjOl6fZwXyoghXFl5obIgwyAXj1D4WQyAQdldcd4eksh93sDVG7IQLO4U2AF4A= X-Received: by 2002:a17:90b:4c52:b0:20d:7917:4cb3 with SMTP id np18-20020a17090b4c5200b0020d79174cb3mr448313pjb.6.1665483736491; Tue, 11 Oct 2022 03:22:16 -0700 (PDT) MIME-Version: 1.0 References: <20220809223401.24599-1-mike.leach@linaro.org> <20220809223401.24599-2-mike.leach@linaro.org> <14411a02-5058-1c03-b98c-9a17975110cd@arm.com> In-Reply-To: <14411a02-5058-1c03-b98c-9a17975110cd@arm.com> From: Mike Leach Date: Tue, 11 Oct 2022 11:22:05 +0100 Message-ID: Subject: Re: [PATCH v3 01/13] coresight: trace-id: Add API to dynamically assign Trace ID values To: Suzuki K Poulose Cc: coresight@lists.linaro.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, mathieu.poirier@linaro.org, peterz@infradead.org, mingo@redhat.com, acme@kernel.org, linux-perf-users@vger.kernel.org, leo.yan@linaro.org, quic_jinlmao@quicinc.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221011_032220_451304_97885A97 X-CRM114-Status: GOOD ( 53.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Suzuki On Mon, 3 Oct 2022 at 09:55, Suzuki K Poulose wrote: > > On 09/08/2022 23:33, Mike Leach wrote: > > The existing mechanism to assign Trace ID values to sources is limited > > and does not scale for larger multicore / multi trace source systems. > > > > The API introduces functions that reserve IDs based on availabilty > > represented by a coresight_trace_id_map structure. This records the > > used and free IDs in a bitmap. > > > > CPU bound sources such as ETMs use the coresight_trace_id_get_cpu_id / > > coresight_trace_id_put_cpu_id pair of functions. The API will record > > the ID associated with the CPU. This ensures that the same ID will be > > re-used while perf events are active on the CPU. The put_cpu_id function > > will pend release of the ID until all perf cs_etm sessions are complete. > > > > Non-cpu sources, such as the STM can use coresight_trace_id_get_system_id / > > coresight_trace_id_put_system_id. > > > > Signed-off-by: Mike Leach > > --- > > drivers/hwtracing/coresight/Makefile | 2 +- > > drivers/hwtracing/coresight/coresight-core.c | 4 + > > .../hwtracing/coresight/coresight-trace-id.c | 230 ++++++++++++++++++ > > .../hwtracing/coresight/coresight-trace-id.h | 78 ++++++ > > include/linux/coresight-pmu.h | 23 +- > > 5 files changed, 324 insertions(+), 13 deletions(-) > > create mode 100644 drivers/hwtracing/coresight/coresight-trace-id.c > > create mode 100644 drivers/hwtracing/coresight/coresight-trace-id.h > > > > diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile > > index b6c4a48140ec..329a0c704b87 100644 > > --- a/drivers/hwtracing/coresight/Makefile > > +++ b/drivers/hwtracing/coresight/Makefile > > @@ -6,7 +6,7 @@ obj-$(CONFIG_CORESIGHT) += coresight.o > > coresight-y := coresight-core.o coresight-etm-perf.o coresight-platform.o \ > > coresight-sysfs.o coresight-syscfg.o coresight-config.o \ > > coresight-cfg-preload.o coresight-cfg-afdo.o \ > > - coresight-syscfg-configfs.o > > + coresight-syscfg-configfs.o coresight-trace-id.o > > obj-$(CONFIG_CORESIGHT_LINK_AND_SINK_TMC) += coresight-tmc.o > > coresight-tmc-y := coresight-tmc-core.o coresight-tmc-etf.o \ > > coresight-tmc-etr.o > > diff --git a/drivers/hwtracing/coresight/coresight-core.c b/drivers/hwtracing/coresight/coresight-core.c > > index 1edfec1e9d18..c7b7c518a0a3 100644 > > --- a/drivers/hwtracing/coresight/coresight-core.c > > +++ b/drivers/hwtracing/coresight/coresight-core.c > > @@ -22,6 +22,7 @@ > > #include "coresight-etm-perf.h" > > #include "coresight-priv.h" > > #include "coresight-syscfg.h" > > +#include "coresight-trace-id.h" > > > > static DEFINE_MUTEX(coresight_mutex); > > static DEFINE_PER_CPU(struct coresight_device *, csdev_sink); > > @@ -1775,6 +1776,9 @@ static int __init coresight_init(void) > > if (ret) > > goto exit_bus_unregister; > > > > + /* initialise the trace ID allocator */ > > + coresight_trace_id_init(); > > + > > /* initialise the coresight syscfg API */ > > ret = cscfg_init(); > > if (!ret) > > diff --git a/drivers/hwtracing/coresight/coresight-trace-id.c b/drivers/hwtracing/coresight/coresight-trace-id.c > > new file mode 100644 > > index 000000000000..ac9092896dec > > --- /dev/null > > +++ b/drivers/hwtracing/coresight/coresight-trace-id.c > > @@ -0,0 +1,230 @@ > > +// SPDX-License-Identifier: GPL-2.0 > > +/* > > + * Copyright (c) 2022, Linaro Limited, All rights reserved. > > + * Author: Mike Leach > > + */ > > +#include > > +#include > > +#include > > +#include > > + > > +#include "coresight-trace-id.h" > > + > > +/* need to keep data on ids & association with cpus. */ > > +struct cpu_id_info { > > + atomic_t id; > > + bool pend_rel; > > +}; > > + > > +/* default trace ID map. Used for systems that do not require per sink mappings */ > > +static struct coresight_trace_id_map id_map_default; > > + > > +/* maintain a record of the current mapping of cpu IDs */ > > +static DEFINE_PER_CPU(struct cpu_id_info, cpu_ids); > > + > > +/* perf session active counter */ > > +static atomic_t perf_cs_etm_session_active = ATOMIC_INIT(0); > > + > > +/* lock to protect id_map and cpu data */ > > +static DEFINE_SPINLOCK(id_map_lock); > > + > > +/* > > + * allocate new ID and set in use > > + * if @preferred_id is a valid id then try to use that value if available. > > + */ > > +static int coresight_trace_id_alloc_new_id(struct coresight_trace_id_map *id_map, > > + int preferred_id) > > +{ > > + int id; > > + > > + /* for backwards compatibility reasons, cpu Ids may have a preferred value */ > > + if (IS_VALID_ID(preferred_id) && !test_bit(preferred_id, id_map->used_ids)) > > + id = preferred_id; > > + else { > > + /* skip reserved bit 0, look from bit 1 to CORESIGHT_TRACE_ID_RES_TOP */ > > + id = find_next_zero_bit(id_map->used_ids, 1, CORESIGHT_TRACE_ID_RES_TOP); > > + if (id >= CORESIGHT_TRACE_ID_RES_TOP) > > + return -EINVAL; > > + } > > + > > + /* mark as used */ > > + set_bit(id, id_map->used_ids); > > + return id; > > +} > > + > > +static void coresight_trace_id_free(int id, struct coresight_trace_id_map *id_map) > > +{ > > + if (WARN(!IS_VALID_ID(id), "%s: Invalid Trace ID %d\n", __func__, id)) > > + return; > > + if (WARN(!test_bit(id, id_map->used_ids), > > + "%s: Freeing unused ID %d\n", __func__, id)) > > + return; > > + clear_bit(id, id_map->used_ids); > > +} > > + > > +static void coresight_trace_id_set_pend_rel(int id, struct coresight_trace_id_map *id_map) > > +{ > > + if (WARN(!IS_VALID_ID(id), "%s: Invalid Trace ID %d\n", __func__, id)) > > + return; > > + set_bit(id, id_map->pend_rel_ids); > > +} > > + > > +/* release all pending IDs for all current maps & clear CPU associations */ > > +static void coresight_trace_id_release_all_pending(void) > > +{ > > + struct coresight_trace_id_map *id_map = &id_map_default; > > + unsigned long flags; > > + int cpu, bit; > > + > > + spin_lock_irqsave(&id_map_lock, flags); > > + for_each_set_bit(bit, id_map->pend_rel_ids, CORESIGHT_TRACE_ID_RES_TOP) { > > + clear_bit(bit, id_map->used_ids); > > + clear_bit(bit, id_map->pend_rel_ids); > > + } > > > > + for_each_possible_cpu(cpu) { > > + if (per_cpu(cpu_ids, cpu).pend_rel) { > > + per_cpu(cpu_ids, cpu).pend_rel = false; > > + atomic_set(&per_cpu(cpu_ids, cpu).id, 0); > > + } > > + } > > Could we use a bitmask for the pending CPUs and use that to track > the pending CPUs ? Looping over the "possible" CPUs could be quite > time consuming for large system with spin lock held. > > DECLARE_BITMAP(cpuid_release_pending, NR_CPUS); > We could do that yes -probably more efficient to scan a bitfield if sparse traced cpus in play. > > > + spin_unlock_irqrestore(&id_map_lock, flags); > > +} > > + > > +static int coresight_trace_id_map_get_cpu_id(int cpu, struct coresight_trace_id_map *id_map) > > +{ > > + unsigned long flags; > > + int id; > > + > > + spin_lock_irqsave(&id_map_lock, flags); > > + > > + /* check for existing allocation for this CPU */ > > + id = atomic_read(&per_cpu(cpu_ids, cpu).id); > > + if (id) > > + goto get_cpu_id_out; > > + > > + /* > > + * Find a new ID. > > + * > > + * Use legacy values where possible in the dynamic trace ID allocator to > > + * allow tools like Android simpleperf to continue working if they are not > > I would rather not mention tools name in here. Could we say : > > * keep as much backward compatibility as possible with the > * older userspace tools, by requesting the "legacy" traceid > * if available. > sure > > + * upgraded at the same time as the kernel drivers. > > + * > > + * If the generated legacy ID is invalid, or not available then the next > > + * available dynamic ID will be used. > > + */ > > + id = coresight_trace_id_alloc_new_id(id_map, CORESIGHT_LEGACY_CPU_TRACE_ID(cpu)); > > + if (IS_VALID_ID(id)) { > > + /* got a valid new ID - save details */ > > + atomic_set(&per_cpu(cpu_ids, cpu).id, id); > > + per_cpu(cpu_ids, cpu).pend_rel = false; > > > > + clear_bit(id, id_map->pend_rel_ids); > > Couldn't this be moved to coresight_trace_id_alloc_new_id() ? For system > pool this is ignored anyways and doesn't matter. > Actually this should always be cleared as we need to ensure an ID we re-use is not also marked as pending. New version will reflect this. > > + } > > + > > +get_cpu_id_out: > > + spin_unlock_irqrestore(&id_map_lock, flags); > > + > > + return id; > > +} > > + > > +static void coresight_trace_id_map_put_cpu_id(int cpu, struct coresight_trace_id_map *id_map) > > +{ > > + unsigned long flags; > > + int id; > > + > > + /* check for existing allocation for this CPU */ > > + id = atomic_read(&per_cpu(cpu_ids, cpu).id); > > + if (!id) > > + goto put_cpu_id_out; > > nit: return; ? > > > + > > + spin_lock_irqsave(&id_map_lock, flags); > > + > > + if (atomic_read(&perf_cs_etm_session_active)) { > > > > + /* set release at pending if perf still active */ > > + coresight_trace_id_set_pend_rel(id, id_map); > > + per_cpu(cpu_ids, cpu).pend_rel = true; > > + } else { > > What prevents another refcount on perf_cs_etm_session_active, after > the above read and before we clear this out ? As far as I can see > the spinlock doesn't prevent this situation. > It does not, and it does not matter. The spinlock prevents IDs being simultaneously allocated and freed. If a new perf session increments perf_cs_etm_session_active from 0->1 within this spinlock, then that perf session cannot start allocating ids for CPUs in the session till the spinlock is released. If a current perf session decrements perf_cs_etm_session_active from 1->0, within this spinlock, then the action in coresight_trace_id_perf_stop() to call coresight_trace_id_release_all_pending() cannot complete until the spinlock is released. Thanks and regards Mike > > + /* otherwise clear id */ > > + coresight_trace_id_free(id, id_map); > > + atomic_set(&per_cpu(cpu_ids, cpu).id, 0); > > + } > > + > > + spin_unlock_irqrestore(&id_map_lock, flags); > > > +put_cpu_id_out: > > This can be removed ? > > > +} > > + > > Suzuki -- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel