From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9137C433EF for ; Tue, 28 Jun 2022 15:41:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345823AbiF1Pln (ORCPT ); Tue, 28 Jun 2022 11:41:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346928AbiF1Plm (ORCPT ); Tue, 28 Jun 2022 11:41:42 -0400 Received: from mail-lj1-x22f.google.com (mail-lj1-x22f.google.com [IPv6:2a00:1450:4864:20::22f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 061BB62C6 for ; Tue, 28 Jun 2022 08:41:41 -0700 (PDT) Received: by mail-lj1-x22f.google.com with SMTP id s10so15375326ljh.12 for ; Tue, 28 Jun 2022 08:41:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=45OuMvop0l12Xy16UbND2bLY50+0LD4yzVw7v19VYg8=; b=tixw4tZb7j9f6SMyAeckCFJC+hYJd8xNh7Rx6rUbM/t0yc2+e7419dt2uO+oMHq+zO 40DQhCgjRzBC8jIJwc5wNZx9n+3dWDxXOv6J405pFrtubNVZkW8D05c4a+3QB2iU1cKU PaVPtDne0KNJrhAdHohNCk8fWX5rX9alPL+j7ZBAp1p52k5MI//Ox0NugOMeBvPEqgcJ HLIqAy91EnzZgcBxfKGKD9e47TCc4mOEgCkI2sb4rxFOermX2xYbCZXKe9g+ZsvmQVFX 00l/SHC+TEHX9UHONQpmON3QI6MWzbaUxPMzJN/IMUE/oQY9kXWVIvrpK2EK+4V7H6H1 xpTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=45OuMvop0l12Xy16UbND2bLY50+0LD4yzVw7v19VYg8=; b=DuthuDh7zTy9VgHZl9qwxXwBc2n3/QFQZrclJmRZFxeFir/IryxStGcFe3wUhlVxvX Zw+zMp9QPEfJEMtmXS6WeNurwOBdGgo36FCDkptr+o5mCtx+CzUOjHy5eA8xhg2TT/m3 OKM1sTzXS4oN6G4e0xVAKCYUoh6KFC7q83imV7kCu1huKx28HpXeSoZcJx7K6e7d9myH 7xGIQYuYnS5pUJlRoTGz8BN4CHyGJD3Gk5B542NNh0yGlqof245DhkbsqiJzDIOAtGGI qV+WFzqW6PRBs4ugZpp50Lx9Gqp2GVCDNzhkbukF06v1JjTDnZ479a+WLZd8MvWqbMpl v2Sg== X-Gm-Message-State: AJIora9duFkMQ1XE0Y3wnhJyXfUYT2poRCupaBG08ZuDHs8RsYQBl0Mv 0mWFW/PXkcRtrrNvF+3hWKmBJDCYBtdnNBHYgB3fjg== X-Google-Smtp-Source: AGRyM1t8CoDJ/rKg0rWWuralRxMR0DkCBugAQ7kbwNId8MGTrF8HS9iDshsoR/7Luv6iaWw7wW9bZcNDAn9f5DsvrJE= X-Received: by 2002:a2e:998c:0:b0:25b:b2ae:b2 with SMTP id w12-20020a2e998c000000b0025bb2ae00b2mr8470900lji.92.1656430899134; Tue, 28 Jun 2022 08:41:39 -0700 (PDT) MIME-Version: 1.0 References: <20220628095833.2579903-1-elver@google.com> <20220628095833.2579903-13-elver@google.com> In-Reply-To: <20220628095833.2579903-13-elver@google.com> From: Dmitry Vyukov Date: Tue, 28 Jun 2022 17:41:27 +0200 Message-ID: Subject: Re: [PATCH v2 12/13] perf/hw_breakpoint: Optimize max_bp_pinned_slots() for CPU-independent task targets To: Marco Elver Cc: Peter Zijlstra , Frederic Weisbecker , Ingo Molnar , Thomas Gleixner , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, linux-perf-users@vger.kernel.org, x86@kernel.org, linux-sh@vger.kernel.org, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-sh@vger.kernel.org On Tue, 28 Jun 2022 at 11:59, Marco Elver wrote: > > Running the perf benchmark with (note: more aggressive parameters vs. > preceding changes, but same 256 CPUs host): > > | $> perf bench -r 100 breakpoint thread -b 4 -p 128 -t 512 > | # Running 'breakpoint/thread' benchmark: > | # Created/joined 100 threads with 4 breakpoints and 128 parallelism > | Total time: 1.989 [sec] > | > | 38.854160 usecs/op > | 4973.332500 usecs/op/cpu > > 20.43% [kernel] [k] queued_spin_lock_slowpath > 18.75% [kernel] [k] osq_lock > 16.98% [kernel] [k] rhashtable_jhash2 > 8.34% [kernel] [k] task_bp_pinned > 4.23% [kernel] [k] smp_cfm_core_cond > 3.65% [kernel] [k] bcmp > 2.83% [kernel] [k] toggle_bp_slot > 1.87% [kernel] [k] find_next_bit > 1.49% [kernel] [k] __reserve_bp_slot > > We can see that a majority of the time is now spent hashing task > pointers to index into task_bps_ht in task_bp_pinned(). > > Obtaining the max_bp_pinned_slots() for CPU-independent task targets > currently is O(#cpus), and calls task_bp_pinned() for each CPU, even if > the result of task_bp_pinned() is CPU-independent. > > The loop in max_bp_pinned_slots() wants to compute the maximum slots > across all CPUs. If task_bp_pinned() is CPU-independent, we can do so by > obtaining the max slots across all CPUs and adding task_bp_pinned(). > > To do so in O(1), use a bp_slots_histogram for CPU-pinned slots. > > After this optimization: > > | $> perf bench -r 100 breakpoint thread -b 4 -p 128 -t 512 > | # Running 'breakpoint/thread' benchmark: > | # Created/joined 100 threads with 4 breakpoints and 128 parallelism > | Total time: 1.930 [sec] > | > | 37.697832 usecs/op > | 4825.322500 usecs/op/cpu > > 19.13% [kernel] [k] queued_spin_lock_slowpath > 18.21% [kernel] [k] rhashtable_jhash2 > 15.46% [kernel] [k] osq_lock > 6.27% [kernel] [k] toggle_bp_slot > 5.91% [kernel] [k] task_bp_pinned > 5.05% [kernel] [k] smp_cfm_core_cond > 1.78% [kernel] [k] update_sg_lb_stats > 1.36% [kernel] [k] llist_reverse_order > 1.34% [kernel] [k] find_next_bit > 1.19% [kernel] [k] bcmp > > Suggesting that time spent in task_bp_pinned() has been reduced. > However, we're still hashing too much, which will be addressed in the > subsequent change. > > Signed-off-by: Marco Elver Reviewed-by: Dmitry Vyukov > --- > v2: > * New patch. > --- > kernel/events/hw_breakpoint.c | 45 +++++++++++++++++++++++++++++++---- > 1 file changed, 41 insertions(+), 4 deletions(-) > > diff --git a/kernel/events/hw_breakpoint.c b/kernel/events/hw_breakpoint.c > index 18886f115abc..b5180a2ccfbf 100644 > --- a/kernel/events/hw_breakpoint.c > +++ b/kernel/events/hw_breakpoint.c > @@ -64,6 +64,9 @@ static struct bp_cpuinfo *get_bp_info(int cpu, enum bp_type_idx type) > return per_cpu_ptr(bp_cpuinfo + type, cpu); > } > > +/* Number of pinned CPU breakpoints globally. */ > +static struct bp_slots_histogram cpu_pinned[TYPE_MAX]; > + > /* Keep track of the breakpoints attached to tasks */ > static struct rhltable task_bps_ht; > static const struct rhashtable_params task_bps_ht_params = { > @@ -194,6 +197,10 @@ static __init int init_breakpoint_slots(void) > goto err; > } > } > + for (i = 0; i < TYPE_MAX; i++) { > + if (!bp_slots_histogram_alloc(&cpu_pinned[i], i)) > + goto err; > + } > > return 0; > err: > @@ -203,6 +210,8 @@ static __init int init_breakpoint_slots(void) > if (err_cpu == cpu) > break; > } > + for (i = 0; i < TYPE_MAX; i++) > + bp_slots_histogram_free(&cpu_pinned[i]); > > return -ENOMEM; > } > @@ -270,6 +279,9 @@ static unsigned int max_task_bp_pinned(int cpu, enum bp_type_idx type) > /* > * Count the number of breakpoints of the same type and same task. > * The given event must be not on the list. > + * > + * If @cpu is -1, but the result of task_bp_pinned() is not CPU-independent, > + * returns a negative value. > */ > static int task_bp_pinned(int cpu, struct perf_event *bp, enum bp_type_idx type) > { > @@ -288,9 +300,18 @@ static int task_bp_pinned(int cpu, struct perf_event *bp, enum bp_type_idx type) > goto out; > > rhl_for_each_entry_rcu(iter, pos, head, hw.bp_list) { > - if (find_slot_idx(iter->attr.bp_type) == type && > - (iter->cpu < 0 || cpu == iter->cpu)) > - count += hw_breakpoint_weight(iter); > + if (find_slot_idx(iter->attr.bp_type) != type) > + continue; > + > + if (iter->cpu >= 0) { > + if (cpu == -1) { > + count = -1; > + goto out; > + } else if (cpu != iter->cpu) > + continue; > + } > + > + count += hw_breakpoint_weight(iter); > } > > out: > @@ -316,6 +337,19 @@ max_bp_pinned_slots(struct perf_event *bp, enum bp_type_idx type) > int pinned_slots = 0; > int cpu; > > + if (bp->hw.target && bp->cpu < 0) { > + int max_pinned = task_bp_pinned(-1, bp, type); > + > + if (max_pinned >= 0) { > + /* > + * Fast path: task_bp_pinned() is CPU-independent and > + * returns the same value for any CPU. > + */ > + max_pinned += bp_slots_histogram_max(&cpu_pinned[type], type); > + return max_pinned; > + } > + } > + > for_each_cpu(cpu, cpumask) { > struct bp_cpuinfo *info = get_bp_info(cpu, type); > int nr; > @@ -366,8 +400,11 @@ toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, > > /* Pinned counter cpu profiling */ > if (!bp->hw.target) { > + struct bp_cpuinfo *info = get_bp_info(bp->cpu, type); > + > lockdep_assert_held_write(&bp_cpuinfo_sem); > - get_bp_info(bp->cpu, type)->cpu_pinned += weight; > + bp_slots_histogram_add(&cpu_pinned[type], info->cpu_pinned, weight); > + info->cpu_pinned += weight; > return 0; > } > > -- > 2.37.0.rc0.161.g10f37bed90-goog > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2B692C433EF for ; Tue, 28 Jun 2022 15:42:19 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4LXTP556sjz3cht for ; Wed, 29 Jun 2022 01:42:17 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=tixw4tZb; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=google.com (client-ip=2a00:1450:4864:20::22f; helo=mail-lj1-x22f.google.com; envelope-from=dvyukov@google.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=tixw4tZb; dkim-atps=neutral Received: from mail-lj1-x22f.google.com (mail-lj1-x22f.google.com [IPv6:2a00:1450:4864:20::22f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4LXTNS624bz3btG for ; Wed, 29 Jun 2022 01:41:43 +1000 (AEST) Received: by mail-lj1-x22f.google.com with SMTP id a11so15398172ljb.5 for ; Tue, 28 Jun 2022 08:41:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=45OuMvop0l12Xy16UbND2bLY50+0LD4yzVw7v19VYg8=; b=tixw4tZb7j9f6SMyAeckCFJC+hYJd8xNh7Rx6rUbM/t0yc2+e7419dt2uO+oMHq+zO 40DQhCgjRzBC8jIJwc5wNZx9n+3dWDxXOv6J405pFrtubNVZkW8D05c4a+3QB2iU1cKU PaVPtDne0KNJrhAdHohNCk8fWX5rX9alPL+j7ZBAp1p52k5MI//Ox0NugOMeBvPEqgcJ HLIqAy91EnzZgcBxfKGKD9e47TCc4mOEgCkI2sb4rxFOermX2xYbCZXKe9g+ZsvmQVFX 00l/SHC+TEHX9UHONQpmON3QI6MWzbaUxPMzJN/IMUE/oQY9kXWVIvrpK2EK+4V7H6H1 xpTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=45OuMvop0l12Xy16UbND2bLY50+0LD4yzVw7v19VYg8=; b=qk0hPR9Hc4UanqhWp6nvSwqiPr86V3iv8mmeP9kYXUyXIm/gd4VOlMJX8rsufFTOBa yCNIsFBm28YDDQ1dsfS3/aZwtsNNZBx2Yycpbml3Hcg28iG+B7k9PFyUHMP768UzxbXh CjSG1c8NwegkDYu3v5xjJmnR+dFzl0RjHkqJaurfILc0S3HzLQ/xjs+t295f2Z1aUk7W +rkH6KSZgI5LfdJC6BjxVhJnqvcNICiOmvFgxGlGz8apFBeOqItxSYeDnuk75q5jJs9V aWerhuJQhvZZjKh0iXBYAO8vnHcQnyWqGj9dAKDTRGaW9x4S/zw/YBJvqHVCdY543xBU d7QA== X-Gm-Message-State: AJIora/y7VOjhKF1NwAOOYcIfhePD3k+gwcf1xwB258HryguzYzTGCPd cMUNB4B/HUICfhjA1kWREZCs9Uk/WjX+SikFYXYGdA== X-Google-Smtp-Source: AGRyM1t8CoDJ/rKg0rWWuralRxMR0DkCBugAQ7kbwNId8MGTrF8HS9iDshsoR/7Luv6iaWw7wW9bZcNDAn9f5DsvrJE= X-Received: by 2002:a2e:998c:0:b0:25b:b2ae:b2 with SMTP id w12-20020a2e998c000000b0025bb2ae00b2mr8470900lji.92.1656430899134; Tue, 28 Jun 2022 08:41:39 -0700 (PDT) MIME-Version: 1.0 References: <20220628095833.2579903-1-elver@google.com> <20220628095833.2579903-13-elver@google.com> In-Reply-To: <20220628095833.2579903-13-elver@google.com> From: Dmitry Vyukov Date: Tue, 28 Jun 2022 17:41:27 +0200 Message-ID: Subject: Re: [PATCH v2 12/13] perf/hw_breakpoint: Optimize max_bp_pinned_slots() for CPU-independent task targets To: Marco Elver Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , linux-sh@vger.kernel.org, Peter Zijlstra , Frederic Weisbecker , x86@kernel.org, linuxppc-dev@lists.ozlabs.org, Arnaldo Carvalho de Melo , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Alexander Shishkin , kasan-dev@googlegroups.com, Namhyung Kim , Thomas Gleixner , Jiri Olsa , Ingo Molnar Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Tue, 28 Jun 2022 at 11:59, Marco Elver wrote: > > Running the perf benchmark with (note: more aggressive parameters vs. > preceding changes, but same 256 CPUs host): > > | $> perf bench -r 100 breakpoint thread -b 4 -p 128 -t 512 > | # Running 'breakpoint/thread' benchmark: > | # Created/joined 100 threads with 4 breakpoints and 128 parallelism > | Total time: 1.989 [sec] > | > | 38.854160 usecs/op > | 4973.332500 usecs/op/cpu > > 20.43% [kernel] [k] queued_spin_lock_slowpath > 18.75% [kernel] [k] osq_lock > 16.98% [kernel] [k] rhashtable_jhash2 > 8.34% [kernel] [k] task_bp_pinned > 4.23% [kernel] [k] smp_cfm_core_cond > 3.65% [kernel] [k] bcmp > 2.83% [kernel] [k] toggle_bp_slot > 1.87% [kernel] [k] find_next_bit > 1.49% [kernel] [k] __reserve_bp_slot > > We can see that a majority of the time is now spent hashing task > pointers to index into task_bps_ht in task_bp_pinned(). > > Obtaining the max_bp_pinned_slots() for CPU-independent task targets > currently is O(#cpus), and calls task_bp_pinned() for each CPU, even if > the result of task_bp_pinned() is CPU-independent. > > The loop in max_bp_pinned_slots() wants to compute the maximum slots > across all CPUs. If task_bp_pinned() is CPU-independent, we can do so by > obtaining the max slots across all CPUs and adding task_bp_pinned(). > > To do so in O(1), use a bp_slots_histogram for CPU-pinned slots. > > After this optimization: > > | $> perf bench -r 100 breakpoint thread -b 4 -p 128 -t 512 > | # Running 'breakpoint/thread' benchmark: > | # Created/joined 100 threads with 4 breakpoints and 128 parallelism > | Total time: 1.930 [sec] > | > | 37.697832 usecs/op > | 4825.322500 usecs/op/cpu > > 19.13% [kernel] [k] queued_spin_lock_slowpath > 18.21% [kernel] [k] rhashtable_jhash2 > 15.46% [kernel] [k] osq_lock > 6.27% [kernel] [k] toggle_bp_slot > 5.91% [kernel] [k] task_bp_pinned > 5.05% [kernel] [k] smp_cfm_core_cond > 1.78% [kernel] [k] update_sg_lb_stats > 1.36% [kernel] [k] llist_reverse_order > 1.34% [kernel] [k] find_next_bit > 1.19% [kernel] [k] bcmp > > Suggesting that time spent in task_bp_pinned() has been reduced. > However, we're still hashing too much, which will be addressed in the > subsequent change. > > Signed-off-by: Marco Elver Reviewed-by: Dmitry Vyukov > --- > v2: > * New patch. > --- > kernel/events/hw_breakpoint.c | 45 +++++++++++++++++++++++++++++++---- > 1 file changed, 41 insertions(+), 4 deletions(-) > > diff --git a/kernel/events/hw_breakpoint.c b/kernel/events/hw_breakpoint.c > index 18886f115abc..b5180a2ccfbf 100644 > --- a/kernel/events/hw_breakpoint.c > +++ b/kernel/events/hw_breakpoint.c > @@ -64,6 +64,9 @@ static struct bp_cpuinfo *get_bp_info(int cpu, enum bp_type_idx type) > return per_cpu_ptr(bp_cpuinfo + type, cpu); > } > > +/* Number of pinned CPU breakpoints globally. */ > +static struct bp_slots_histogram cpu_pinned[TYPE_MAX]; > + > /* Keep track of the breakpoints attached to tasks */ > static struct rhltable task_bps_ht; > static const struct rhashtable_params task_bps_ht_params = { > @@ -194,6 +197,10 @@ static __init int init_breakpoint_slots(void) > goto err; > } > } > + for (i = 0; i < TYPE_MAX; i++) { > + if (!bp_slots_histogram_alloc(&cpu_pinned[i], i)) > + goto err; > + } > > return 0; > err: > @@ -203,6 +210,8 @@ static __init int init_breakpoint_slots(void) > if (err_cpu == cpu) > break; > } > + for (i = 0; i < TYPE_MAX; i++) > + bp_slots_histogram_free(&cpu_pinned[i]); > > return -ENOMEM; > } > @@ -270,6 +279,9 @@ static unsigned int max_task_bp_pinned(int cpu, enum bp_type_idx type) > /* > * Count the number of breakpoints of the same type and same task. > * The given event must be not on the list. > + * > + * If @cpu is -1, but the result of task_bp_pinned() is not CPU-independent, > + * returns a negative value. > */ > static int task_bp_pinned(int cpu, struct perf_event *bp, enum bp_type_idx type) > { > @@ -288,9 +300,18 @@ static int task_bp_pinned(int cpu, struct perf_event *bp, enum bp_type_idx type) > goto out; > > rhl_for_each_entry_rcu(iter, pos, head, hw.bp_list) { > - if (find_slot_idx(iter->attr.bp_type) == type && > - (iter->cpu < 0 || cpu == iter->cpu)) > - count += hw_breakpoint_weight(iter); > + if (find_slot_idx(iter->attr.bp_type) != type) > + continue; > + > + if (iter->cpu >= 0) { > + if (cpu == -1) { > + count = -1; > + goto out; > + } else if (cpu != iter->cpu) > + continue; > + } > + > + count += hw_breakpoint_weight(iter); > } > > out: > @@ -316,6 +337,19 @@ max_bp_pinned_slots(struct perf_event *bp, enum bp_type_idx type) > int pinned_slots = 0; > int cpu; > > + if (bp->hw.target && bp->cpu < 0) { > + int max_pinned = task_bp_pinned(-1, bp, type); > + > + if (max_pinned >= 0) { > + /* > + * Fast path: task_bp_pinned() is CPU-independent and > + * returns the same value for any CPU. > + */ > + max_pinned += bp_slots_histogram_max(&cpu_pinned[type], type); > + return max_pinned; > + } > + } > + > for_each_cpu(cpu, cpumask) { > struct bp_cpuinfo *info = get_bp_info(cpu, type); > int nr; > @@ -366,8 +400,11 @@ toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type, > > /* Pinned counter cpu profiling */ > if (!bp->hw.target) { > + struct bp_cpuinfo *info = get_bp_info(bp->cpu, type); > + > lockdep_assert_held_write(&bp_cpuinfo_sem); > - get_bp_info(bp->cpu, type)->cpu_pinned += weight; > + bp_slots_histogram_add(&cpu_pinned[type], info->cpu_pinned, weight); > + info->cpu_pinned += weight; > return 0; > } > > -- > 2.37.0.rc0.161.g10f37bed90-goog >