From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB21EC10F00 for ; Fri, 6 Mar 2020 13:33:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 967FF206D5 for ; Fri, 6 Mar 2020 13:33:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727539AbgCFNdk (ORCPT ); Fri, 6 Mar 2020 08:33:40 -0500 Received: from foss.arm.com ([217.140.110.172]:33760 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726947AbgCFNdj (ORCPT ); Fri, 6 Mar 2020 08:33:39 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 48DA231B; Fri, 6 Mar 2020 05:33:39 -0800 (PST) Received: from e119884-lin.cambridge.arm.com (e119884-lin.cambridge.arm.com [10.1.196.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6D45D3F6CF; Fri, 6 Mar 2020 05:33:36 -0800 (PST) From: Vincenzo Frascino To: linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, clang-built-linux@googlegroups.com, x86@kernel.org Cc: Vincenzo Frascino , Catalin Marinas , Will Deacon , Arnd Bergmann , Russell King , Paul Burton , Thomas Gleixner , Andy Lutomirski , Ingo Molnar , Borislav Petkov , Stephen Boyd , Mark Salyzyn , Kees Cook , Peter Collingbourne , Dmitry Safonov <0x7f454c46@gmail.com>, Andrei Vagin , Nick Desaulniers , Will Deacon Subject: [PATCH v2 15/20] arm64: Introduce asm/common/arch_timer.h Date: Fri, 6 Mar 2020 13:32:37 +0000 Message-Id: <20200306133242.26279-16-vincenzo.frascino@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200306133242.26279-1-vincenzo.frascino@arm.com> References: <20200306133242.26279-1-vincenzo.frascino@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The vDSO library should only include the necessary headers required for a userspace library (UAPI and a minimal set of kernel headers). To make this possible it is necessary to isolate from the kernel headers the common parts that are strictly necessary to build the library. Introduce asm/common/arch_timer.h to contain all the arm64 specific code. This allows to replace the second isb() in __arch_get_hw_counter() with a fake dependent stack read of the counter which improves the vdso library peformances of ~4.5%. Below the results of vdsotest [1] ran for 100 iterations. Before the patch: ================= clock-gettime-monotonic: syscall: 771 nsec/call clock-gettime-monotonic: libc: 130 nsec/call clock-gettime-monotonic: vdso: 111 nsec/call ... clock-gettime-realtime: syscall: 762 nsec/call clock-gettime-realtime: libc: 130 nsec/call clock-gettime-realtime: vdso: 111 nsec/call After the patch: ================ clock-gettime-monotonic: syscall: 792 nsec/call clock-gettime-monotonic: libc: 124 nsec/call clock-gettime-monotonic: vdso: 106 nsec/call ... clock-gettime-realtime: syscall: 776 nsec/call clock-gettime-realtime: libc: 124 nsec/call clock-gettime-realtime: vdso: 106 nsec/call [1] https://github.com/nathanlynch/vdsotest Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Vincenzo Frascino --- arch/arm64/include/asm/arch_timer.h | 29 ++++--------------- arch/arm64/include/asm/common/arch_timer.h | 33 ++++++++++++++++++++++ arch/arm64/include/asm/vdso/gettimeofday.h | 7 +++-- 3 files changed, 42 insertions(+), 27 deletions(-) create mode 100644 arch/arm64/include/asm/common/arch_timer.h diff --git a/arch/arm64/include/asm/arch_timer.h b/arch/arm64/include/asm/arch_timer.h index 7ae54d7d333a..6567e20a76b2 100644 --- a/arch/arm64/include/asm/arch_timer.h +++ b/arch/arm64/include/asm/arch_timer.h @@ -164,24 +164,7 @@ static inline void arch_timer_set_cntkctl(u32 cntkctl) isb(); } -/* - * Ensure that reads of the counter are treated the same as memory reads - * for the purposes of ordering by subsequent memory barriers. - * - * This insanity brought to you by speculative system register reads, - * out-of-order memory accesses, sequence locks and Thomas Gleixner. - * - * http://lists.infradead.org/pipermail/linux-arm-kernel/2019-February/631195.html - */ -#define arch_counter_enforce_ordering(val) do { \ - u64 tmp, _val = (val); \ - \ - asm volatile( \ - " eor %0, %1, %1\n" \ - " add %0, sp, %0\n" \ - " ldr xzr, [%0]" \ - : "=r" (tmp) : "r" (_val)); \ -} while (0) +#include static __always_inline u64 __arch_counter_get_cntpct_stable(void) { @@ -189,7 +172,7 @@ static __always_inline u64 __arch_counter_get_cntpct_stable(void) isb(); cnt = arch_timer_reg_read_stable(cntpct_el0); - arch_counter_enforce_ordering(cnt); + cnt = arch_counter_enforce_ordering(cnt); return cnt; } @@ -199,7 +182,7 @@ static __always_inline u64 __arch_counter_get_cntpct(void) isb(); cnt = read_sysreg(cntpct_el0); - arch_counter_enforce_ordering(cnt); + cnt = arch_counter_enforce_ordering(cnt); return cnt; } @@ -209,7 +192,7 @@ static __always_inline u64 __arch_counter_get_cntvct_stable(void) isb(); cnt = arch_timer_reg_read_stable(cntvct_el0); - arch_counter_enforce_ordering(cnt); + cnt = arch_counter_enforce_ordering(cnt); return cnt; } @@ -219,12 +202,10 @@ static __always_inline u64 __arch_counter_get_cntvct(void) isb(); cnt = read_sysreg(cntvct_el0); - arch_counter_enforce_ordering(cnt); + cnt = arch_counter_enforce_ordering(cnt); return cnt; } -#undef arch_counter_enforce_ordering - static inline int arch_timer_arch_init(void) { return 0; diff --git a/arch/arm64/include/asm/common/arch_timer.h b/arch/arm64/include/asm/common/arch_timer.h new file mode 100644 index 000000000000..b11bd67a366e --- /dev/null +++ b/arch/arm64/include/asm/common/arch_timer.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 ARM Ltd. + */ +#ifndef __ASM_COMMON_ARCH_TIMER_H +#define __ASM_COMMON_ARCH_TIMER_H + +#include + +/* + * Ensure that reads of the counter are treated the same as memory reads + * for the purposes of ordering by subsequent memory barriers. + * + * This insanity brought to you by speculative system register reads, + * out-of-order memory accesses, sequence locks and Thomas Gleixner. + * + * http://lists.infradead.org/pipermail/linux-arm-kernel/2019-February/631195.html + * + */ +static u64 arch_counter_enforce_ordering(u64 val) +{ + u64 tmp, _val = (val); + + asm volatile( + " eor %0, %1, %1\n" + " add %0, sp, %0\n" + " ldr xzr, [%0]" + : "=r" (tmp) : "r" (_val)); + + return _val; +} + +#endif /* __ASM_COMMON_ARCH_TIMER_H */ diff --git a/arch/arm64/include/asm/vdso/gettimeofday.h b/arch/arm64/include/asm/vdso/gettimeofday.h index cc3456416096..628bb3b187c4 100644 --- a/arch/arm64/include/asm/vdso/gettimeofday.h +++ b/arch/arm64/include/asm/vdso/gettimeofday.h @@ -8,6 +8,7 @@ #ifndef __ASSEMBLY__ #include +#include #define __VDSO_USE_SYSCALL ULLONG_MAX @@ -83,10 +84,10 @@ static __always_inline u64 __arch_get_hw_counter(s32 clock_mode) isb(); asm volatile("mrs %0, cntvct_el0" : "=r" (res) :: "memory"); /* - * This isb() is required to prevent that the seq lock is - * speculated.# + * arch_counter_enforce_ordering() is required to prevent that + * the seq lock is speculated. */ - isb(); + res = arch_counter_enforce_ordering(res); return res; } -- 2.25.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF450C10DCE for ; Fri, 6 Mar 2020 13:37:18 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A198B2072D for ; Fri, 6 Mar 2020 13:37:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="eONz7FUs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A198B2072D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9ctPx58G0TCP6bVsYuRPPb1S5ohE2EpNBRY3l8IKgpQ=; b=eONz7FUsq2/cHp BdBRmJIIG1TLBmg6AYvQprWLDdrGBj0n3OPRpr3gaARojB0TH68GjJZRIMUj1ocbPMZZbBBMiaZ3r yErOyDPhJ1EP/4K7A9kM+ZewgByPJzRFKnVKUpMfi1gMJGnnVEPHgfVaJF9PXZJVwQpD5aUwz+1oS sq/Utk3Laljq9JbvexSRRhRg1luJZt6kMdHBibTfTB11beL+VUsE1kcCZQBLJel4h70Mb2Bpc4M34 ugp/GHbRsTFXpV26YwyP329Uh1KPsFJZRufJo/Uk1MO5GyGUn4cuOpD8/dE+djb+Jjm7WzdMHF39M 0U/jqwc2ha6qBwlW0whA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jADAE-0001Lf-0r; Fri, 06 Mar 2020 13:37:18 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jAD6i-0004wl-5t for linux-arm-kernel@lists.infradead.org; Fri, 06 Mar 2020 13:33:42 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 48DA231B; Fri, 6 Mar 2020 05:33:39 -0800 (PST) Received: from e119884-lin.cambridge.arm.com (e119884-lin.cambridge.arm.com [10.1.196.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6D45D3F6CF; Fri, 6 Mar 2020 05:33:36 -0800 (PST) From: Vincenzo Frascino To: linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, clang-built-linux@googlegroups.com, x86@kernel.org Subject: [PATCH v2 15/20] arm64: Introduce asm/common/arch_timer.h Date: Fri, 6 Mar 2020 13:32:37 +0000 Message-Id: <20200306133242.26279-16-vincenzo.frascino@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200306133242.26279-1-vincenzo.frascino@arm.com> References: <20200306133242.26279-1-vincenzo.frascino@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200306_053340_357635_E1D89250 X-CRM114-Status: GOOD ( 17.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Will Deacon , Nick Desaulniers , Dmitry Safonov <0x7f454c46@gmail.com>, Kees Cook , Arnd Bergmann , Stephen Boyd , Catalin Marinas , Will Deacon , Russell King , Mark Salyzyn , Paul Burton , Ingo Molnar , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , Vincenzo Frascino , Peter Collingbourne , Andrei Vagin Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org The vDSO library should only include the necessary headers required for a userspace library (UAPI and a minimal set of kernel headers). To make this possible it is necessary to isolate from the kernel headers the common parts that are strictly necessary to build the library. Introduce asm/common/arch_timer.h to contain all the arm64 specific code. This allows to replace the second isb() in __arch_get_hw_counter() with a fake dependent stack read of the counter which improves the vdso library peformances of ~4.5%. Below the results of vdsotest [1] ran for 100 iterations. Before the patch: ================= clock-gettime-monotonic: syscall: 771 nsec/call clock-gettime-monotonic: libc: 130 nsec/call clock-gettime-monotonic: vdso: 111 nsec/call ... clock-gettime-realtime: syscall: 762 nsec/call clock-gettime-realtime: libc: 130 nsec/call clock-gettime-realtime: vdso: 111 nsec/call After the patch: ================ clock-gettime-monotonic: syscall: 792 nsec/call clock-gettime-monotonic: libc: 124 nsec/call clock-gettime-monotonic: vdso: 106 nsec/call ... clock-gettime-realtime: syscall: 776 nsec/call clock-gettime-realtime: libc: 124 nsec/call clock-gettime-realtime: vdso: 106 nsec/call [1] https://github.com/nathanlynch/vdsotest Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Vincenzo Frascino --- arch/arm64/include/asm/arch_timer.h | 29 ++++--------------- arch/arm64/include/asm/common/arch_timer.h | 33 ++++++++++++++++++++++ arch/arm64/include/asm/vdso/gettimeofday.h | 7 +++-- 3 files changed, 42 insertions(+), 27 deletions(-) create mode 100644 arch/arm64/include/asm/common/arch_timer.h diff --git a/arch/arm64/include/asm/arch_timer.h b/arch/arm64/include/asm/arch_timer.h index 7ae54d7d333a..6567e20a76b2 100644 --- a/arch/arm64/include/asm/arch_timer.h +++ b/arch/arm64/include/asm/arch_timer.h @@ -164,24 +164,7 @@ static inline void arch_timer_set_cntkctl(u32 cntkctl) isb(); } -/* - * Ensure that reads of the counter are treated the same as memory reads - * for the purposes of ordering by subsequent memory barriers. - * - * This insanity brought to you by speculative system register reads, - * out-of-order memory accesses, sequence locks and Thomas Gleixner. - * - * http://lists.infradead.org/pipermail/linux-arm-kernel/2019-February/631195.html - */ -#define arch_counter_enforce_ordering(val) do { \ - u64 tmp, _val = (val); \ - \ - asm volatile( \ - " eor %0, %1, %1\n" \ - " add %0, sp, %0\n" \ - " ldr xzr, [%0]" \ - : "=r" (tmp) : "r" (_val)); \ -} while (0) +#include static __always_inline u64 __arch_counter_get_cntpct_stable(void) { @@ -189,7 +172,7 @@ static __always_inline u64 __arch_counter_get_cntpct_stable(void) isb(); cnt = arch_timer_reg_read_stable(cntpct_el0); - arch_counter_enforce_ordering(cnt); + cnt = arch_counter_enforce_ordering(cnt); return cnt; } @@ -199,7 +182,7 @@ static __always_inline u64 __arch_counter_get_cntpct(void) isb(); cnt = read_sysreg(cntpct_el0); - arch_counter_enforce_ordering(cnt); + cnt = arch_counter_enforce_ordering(cnt); return cnt; } @@ -209,7 +192,7 @@ static __always_inline u64 __arch_counter_get_cntvct_stable(void) isb(); cnt = arch_timer_reg_read_stable(cntvct_el0); - arch_counter_enforce_ordering(cnt); + cnt = arch_counter_enforce_ordering(cnt); return cnt; } @@ -219,12 +202,10 @@ static __always_inline u64 __arch_counter_get_cntvct(void) isb(); cnt = read_sysreg(cntvct_el0); - arch_counter_enforce_ordering(cnt); + cnt = arch_counter_enforce_ordering(cnt); return cnt; } -#undef arch_counter_enforce_ordering - static inline int arch_timer_arch_init(void) { return 0; diff --git a/arch/arm64/include/asm/common/arch_timer.h b/arch/arm64/include/asm/common/arch_timer.h new file mode 100644 index 000000000000..b11bd67a366e --- /dev/null +++ b/arch/arm64/include/asm/common/arch_timer.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 ARM Ltd. + */ +#ifndef __ASM_COMMON_ARCH_TIMER_H +#define __ASM_COMMON_ARCH_TIMER_H + +#include + +/* + * Ensure that reads of the counter are treated the same as memory reads + * for the purposes of ordering by subsequent memory barriers. + * + * This insanity brought to you by speculative system register reads, + * out-of-order memory accesses, sequence locks and Thomas Gleixner. + * + * http://lists.infradead.org/pipermail/linux-arm-kernel/2019-February/631195.html + * + */ +static u64 arch_counter_enforce_ordering(u64 val) +{ + u64 tmp, _val = (val); + + asm volatile( + " eor %0, %1, %1\n" + " add %0, sp, %0\n" + " ldr xzr, [%0]" + : "=r" (tmp) : "r" (_val)); + + return _val; +} + +#endif /* __ASM_COMMON_ARCH_TIMER_H */ diff --git a/arch/arm64/include/asm/vdso/gettimeofday.h b/arch/arm64/include/asm/vdso/gettimeofday.h index cc3456416096..628bb3b187c4 100644 --- a/arch/arm64/include/asm/vdso/gettimeofday.h +++ b/arch/arm64/include/asm/vdso/gettimeofday.h @@ -8,6 +8,7 @@ #ifndef __ASSEMBLY__ #include +#include #define __VDSO_USE_SYSCALL ULLONG_MAX @@ -83,10 +84,10 @@ static __always_inline u64 __arch_get_hw_counter(s32 clock_mode) isb(); asm volatile("mrs %0, cntvct_el0" : "=r" (res) :: "memory"); /* - * This isb() is required to prevent that the seq lock is - * speculated.# + * arch_counter_enforce_ordering() is required to prevent that + * the seq lock is speculated. */ - isb(); + res = arch_counter_enforce_ordering(res); return res; } -- 2.25.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel