From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C209C433E6 for ; Wed, 15 Jul 2020 02:05:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3127020663 for ; Wed, 15 Jul 2020 02:05:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="vtbH+rvI" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728223AbgGOCFn (ORCPT ); Tue, 14 Jul 2020 22:05:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726356AbgGOCFn (ORCPT ); Tue, 14 Jul 2020 22:05:43 -0400 Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15716C061755 for ; Tue, 14 Jul 2020 19:05:42 -0700 (PDT) Received: by mail-pj1-x1042.google.com with SMTP id ls15so1302519pjb.1 for ; Tue, 14 Jul 2020 19:05:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=i8X88B9XAc//7jPUSbPAl58G8UKTwAt4ZD/ImrmTyEo=; b=vtbH+rvISfZr3PPUvE3WJyrY4/qP4B3Fob4wEq2EtThH5+99scLGykM5e49ArE3EDg TgkZiFzxOCewMSAsHX4Fl6dFG5cN88ATrvgsO8z+CDP8NFbm6F7H3JY73TD8TaeG+IxO /ZI6SAnebOojnCePbOew3LkEO8p0aQqorF8RPCtjkRwnxLtDdHN6PfkOeHNPRdZ8hjW9 NeuwnDum9pacKFdyd8fwTei9SRJ6nIX1UuolQc/eY5c3+FUtg9C3hxL7ms/TuvW0Tx9C qNaipQEj9/i5qd8Sdx6/6kg9lymoTISvMelwcA5NDSBKSDfj5wXUrxroJhI0no2hwNbw Afhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=i8X88B9XAc//7jPUSbPAl58G8UKTwAt4ZD/ImrmTyEo=; b=MvVy69452OH4XISSH5OXB6w/A2iMrArl080KLjwkKlDVgVJXYuCVp+ALhXbAbgwhPZ ++FvjHAKpMb1Ox8WXMScKj1n7QoB3Rt7rpF4M+xB4s0+j9HgAyGYe2eqUjuLsYLy4d75 R6aRQKWddO0NSowX4ZXD+xNQxSg52WKkpSm1XnnK0zNE5NusgrEtYjX1HvXTbbUv9QCU 11b+PbZhEDlIfsnh4jYuEW1lIecAYzP12evsA+WK9NMvwSBBmpejlROsb7kL0KrLBAwx a6iqZfNNx6Fc7tLiQBxlZhjNNVDO68p0aSdDqOEhQ/G+qJko2dnTNyeu6eXv9YT6MYhU LXtQ== X-Gm-Message-State: AOAM5310DCEsdV3QFTLHpQU8otoCLReibQeL/1wLsTZKAT1sjDEvsRUt /iRSVsRKKuXYdTnjaGSK5cztGg== X-Google-Smtp-Source: ABdhPJwj/KisWOWdgz943aOT+B94CHGzkWgMyhvzA6RiDEU6FfzxEsHjV5np2ZCI6pLzOMWql82Nag== X-Received: by 2002:a17:90a:3684:: with SMTP id t4mr7394442pjb.91.1594778741505; Tue, 14 Jul 2020 19:05:41 -0700 (PDT) Received: from localhost ([2400:8904::f03c:91ff:fe8a:bbe4]) by smtp.gmail.com with ESMTPSA id q11sm297218pjj.17.2020.07.14.19.05.40 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 14 Jul 2020 19:05:41 -0700 (PDT) From: Leo Yan To: Will Deacon , Mark Rutland , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Alexander Shishkin , Jiri Olsa , Namhyung Kim , Catalin Marinas , Thomas Gleixner , Paul Cercueil , "Ben Dooks (Codethink)" , "Ahmed S. Darwish" , Adrian Hunter , Kan Liang , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/6] arm64: perf: Implement correct cap_user_time Date: Wed, 15 Jul 2020 10:05:08 +0800 Message-Id: <20200715020512.20991-3-leo.yan@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200715020512.20991-1-leo.yan@linaro.org> References: <20200715020512.20991-1-leo.yan@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Peter Zijlstra As reported by Leo; the existing implementation is broken when the clock and counter don't intersect at 0. Use the sched_clock's struct clock_read_data information to correctly implement cap_user_time and cap_user_time_zero. Note that the ARM64 counter is architecturally only guaranteed to be 56bit wide (implementations are allowed to be wider) and the existing perf ABI cannot deal with wrap-around. This implementation should also be faster than the old; seeing how we don't need to recompute mult and shift all the time. [leoyan: Use quot/rem to convert cyc to ns to avoid overflow] Reported-by: Leo Yan Signed-off-by: Peter Zijlstra (Intel) --- arch/arm64/kernel/perf_event.c | 40 ++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kernel/perf_event.c b/arch/arm64/kernel/perf_event.c index 4d7879484cec..35c2c737d4af 100644 --- a/arch/arm64/kernel/perf_event.c +++ b/arch/arm64/kernel/perf_event.c @@ -19,6 +19,7 @@ #include #include #include +#include #include /* ARMv8 Cortex-A53 specific event types. */ @@ -1165,28 +1166,49 @@ device_initcall(armv8_pmu_driver_init) void arch_perf_update_userpage(struct perf_event *event, struct perf_event_mmap_page *userpg, u64 now) { - u32 freq; - u32 shift; + struct clock_read_data *rd; + unsigned int seq; + u64 quot, rem, ns; /* * Internal timekeeping for enabled/running/stopped times * is always computed with the sched_clock. */ - freq = arch_timer_get_rate(); userpg->cap_user_time = 1; + userpg->cap_user_time_zero = 1; + + do { + rd = sched_clock_read_begin(&seq); + + userpg->time_mult = rd->mult; + userpg->time_shift = rd->shift; + userpg->time_zero = rd->epoch_ns; + + /* + * This isn't strictly correct, the ARM64 counter can be + * 'short' and then we get funnies when it wraps. The correct + * thing would be to extend the perf ABI with a cycle and mask + * value, but because wrapping on ARM64 is very rare in + * practise this 'works'. + */ + quot = rd->epoch_cyc >> rd->shift; + rem = rd->epoch_cyc & (((u64)1 << rd->shift) - 1); + ns = quot * rd->mult + ((rem * rd->mult) >> rd->shift); + userpg->time_zero -= ns; + + } while (sched_clock_read_retry(seq)); + + userpg->time_offset = userpg->time_zero - now; - clocks_calc_mult_shift(&userpg->time_mult, &shift, freq, - NSEC_PER_SEC, 0); /* * time_shift is not expected to be greater than 31 due to * the original published conversion algorithm shifting a * 32-bit value (now specifies a 64-bit value) - refer * perf_event_mmap_page documentation in perf_event.h. */ - if (shift == 32) { - shift = 31; + if (userpg->time_shift == 32) { + userpg->time_shift = 31; userpg->time_mult >>= 1; } - userpg->time_shift = (u16)shift; - userpg->time_offset = -now; + } -- 2.17.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1CDDC433E0 for ; Wed, 15 Jul 2020 02:07:14 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 869AA20729 for ; Wed, 15 Jul 2020 02:07:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Kdxl+rar"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="vtbH+rvI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 869AA20729 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:To: From:Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=fLmmDLlI+qFgxiuJqH90hgSUPXSkzilZwKYSK4FSEI8=; b=Kdxl+rarCD9ekLgFkHiFdtrGnl +a98lJ4od6hLKu3FvofGYES4B/fuj47ojW6zPkMn+U1ETlXEVt49SWO9OiG/Xo38Ef/oJlVzrVsTe BWqMIWdLPp3djdOh29MSgEa/B/S4JpjaYBUd2X44wbdgcLDgrZwxMY+MsNbWORQH6taV3joW8N96i AIl2ik51SIjoosktn86oq9yIpjuUznJ/QpDArq8BXPyRtUnkwrP5+4LLRbMCe1OwtSmH1KT/VSS6B FA61eKCyLup1bGe1vXntxuZM6EVIQgumsctfUuHqTuzvm3NBZ4eF8+za1uqm+HplQbo6NoBRZp/DT BBdmNLcA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jvWnq-0007Ok-TZ; Wed, 15 Jul 2020 02:05:46 +0000 Received: from mail-pj1-x1042.google.com ([2607:f8b0:4864:20::1042]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jvWnn-0007NX-FM for linux-arm-kernel@lists.infradead.org; Wed, 15 Jul 2020 02:05:44 +0000 Received: by mail-pj1-x1042.google.com with SMTP id ch3so1296749pjb.5 for ; Tue, 14 Jul 2020 19:05:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=i8X88B9XAc//7jPUSbPAl58G8UKTwAt4ZD/ImrmTyEo=; b=vtbH+rvISfZr3PPUvE3WJyrY4/qP4B3Fob4wEq2EtThH5+99scLGykM5e49ArE3EDg TgkZiFzxOCewMSAsHX4Fl6dFG5cN88ATrvgsO8z+CDP8NFbm6F7H3JY73TD8TaeG+IxO /ZI6SAnebOojnCePbOew3LkEO8p0aQqorF8RPCtjkRwnxLtDdHN6PfkOeHNPRdZ8hjW9 NeuwnDum9pacKFdyd8fwTei9SRJ6nIX1UuolQc/eY5c3+FUtg9C3hxL7ms/TuvW0Tx9C qNaipQEj9/i5qd8Sdx6/6kg9lymoTISvMelwcA5NDSBKSDfj5wXUrxroJhI0no2hwNbw Afhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=i8X88B9XAc//7jPUSbPAl58G8UKTwAt4ZD/ImrmTyEo=; b=ovMIHK0PeWb9h+819go1qrC5SVMYknEiuULL32Q8le0EJps1vT3uJvI77JrSfkrEp+ 3AcR0t621SHaPoZpCq3BYbMpWoFyI3yZYdCDYBFQN1X6KBCQEmjAIwmDfwUSKCeBENfk dyTIxFJm3PR27/j+jg4+jBlM3r4qtuT4HyoMiGILRoqjPnco0ue5mYgXpsZ08EmYVSfU 6jbudSY4oMleVVWqj7WQIH+JBxjxlPl+ibFfsa9jCrG2vy9OFebDiuM6P5Zlxlr1gflO FSy7IlTii5QpdjMvIW9+wZKnJyEv9TGMiFKl5O/kI8c79mFgXzFr26mkAjZobTEa/HQ0 Wx7A== X-Gm-Message-State: AOAM532TB45o7o1UAKcz6AvG3gKiA1QD/2SaMYs2ZG/hRPehDXL9MP/s Mc0gTrmB6XUdOvqgqev1V1Z4IQ== X-Google-Smtp-Source: ABdhPJwj/KisWOWdgz943aOT+B94CHGzkWgMyhvzA6RiDEU6FfzxEsHjV5np2ZCI6pLzOMWql82Nag== X-Received: by 2002:a17:90a:3684:: with SMTP id t4mr7394442pjb.91.1594778741505; Tue, 14 Jul 2020 19:05:41 -0700 (PDT) Received: from localhost ([2400:8904::f03c:91ff:fe8a:bbe4]) by smtp.gmail.com with ESMTPSA id q11sm297218pjj.17.2020.07.14.19.05.40 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 14 Jul 2020 19:05:41 -0700 (PDT) From: Leo Yan To: Will Deacon , Mark Rutland , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Alexander Shishkin , Jiri Olsa , Namhyung Kim , Catalin Marinas , Thomas Gleixner , Paul Cercueil , "Ben Dooks (Codethink)" , "Ahmed S. Darwish" , Adrian Hunter , Kan Liang , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/6] arm64: perf: Implement correct cap_user_time Date: Wed, 15 Jul 2020 10:05:08 +0800 Message-Id: <20200715020512.20991-3-leo.yan@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200715020512.20991-1-leo.yan@linaro.org> References: <20200715020512.20991-1-leo.yan@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200714_220543_534757_02F15F75 X-CRM114-Status: GOOD ( 17.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Peter Zijlstra As reported by Leo; the existing implementation is broken when the clock and counter don't intersect at 0. Use the sched_clock's struct clock_read_data information to correctly implement cap_user_time and cap_user_time_zero. Note that the ARM64 counter is architecturally only guaranteed to be 56bit wide (implementations are allowed to be wider) and the existing perf ABI cannot deal with wrap-around. This implementation should also be faster than the old; seeing how we don't need to recompute mult and shift all the time. [leoyan: Use quot/rem to convert cyc to ns to avoid overflow] Reported-by: Leo Yan Signed-off-by: Peter Zijlstra (Intel) --- arch/arm64/kernel/perf_event.c | 40 ++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kernel/perf_event.c b/arch/arm64/kernel/perf_event.c index 4d7879484cec..35c2c737d4af 100644 --- a/arch/arm64/kernel/perf_event.c +++ b/arch/arm64/kernel/perf_event.c @@ -19,6 +19,7 @@ #include #include #include +#include #include /* ARMv8 Cortex-A53 specific event types. */ @@ -1165,28 +1166,49 @@ device_initcall(armv8_pmu_driver_init) void arch_perf_update_userpage(struct perf_event *event, struct perf_event_mmap_page *userpg, u64 now) { - u32 freq; - u32 shift; + struct clock_read_data *rd; + unsigned int seq; + u64 quot, rem, ns; /* * Internal timekeeping for enabled/running/stopped times * is always computed with the sched_clock. */ - freq = arch_timer_get_rate(); userpg->cap_user_time = 1; + userpg->cap_user_time_zero = 1; + + do { + rd = sched_clock_read_begin(&seq); + + userpg->time_mult = rd->mult; + userpg->time_shift = rd->shift; + userpg->time_zero = rd->epoch_ns; + + /* + * This isn't strictly correct, the ARM64 counter can be + * 'short' and then we get funnies when it wraps. The correct + * thing would be to extend the perf ABI with a cycle and mask + * value, but because wrapping on ARM64 is very rare in + * practise this 'works'. + */ + quot = rd->epoch_cyc >> rd->shift; + rem = rd->epoch_cyc & (((u64)1 << rd->shift) - 1); + ns = quot * rd->mult + ((rem * rd->mult) >> rd->shift); + userpg->time_zero -= ns; + + } while (sched_clock_read_retry(seq)); + + userpg->time_offset = userpg->time_zero - now; - clocks_calc_mult_shift(&userpg->time_mult, &shift, freq, - NSEC_PER_SEC, 0); /* * time_shift is not expected to be greater than 31 due to * the original published conversion algorithm shifting a * 32-bit value (now specifies a 64-bit value) - refer * perf_event_mmap_page documentation in perf_event.h. */ - if (shift == 32) { - shift = 31; + if (userpg->time_shift == 32) { + userpg->time_shift = 31; userpg->time_mult >>= 1; } - userpg->time_shift = (u16)shift; - userpg->time_offset = -now; + } -- 2.17.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel