From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B92F0C33CA9 for ; Mon, 13 Jan 2020 17:09:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8FBC72187F for ; Mon, 13 Jan 2020 17:09:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="HkoD9yf+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728975AbgAMRJX (ORCPT ); Mon, 13 Jan 2020 12:09:23 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:19156 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728795AbgAMRIr (ORCPT ); Mon, 13 Jan 2020 12:08:47 -0500 Received: from localhost (mailhub1-int [192.168.12.234]) by localhost (Postfix) with ESMTP id 47xKmm0TxYz9txyy; Mon, 13 Jan 2020 18:08:40 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=HkoD9yf+; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id AhN4_motg1-N; Mon, 13 Jan 2020 18:08:40 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47xKml6VQMz9txyq; Mon, 13 Jan 2020 18:08:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1578935319; bh=cvbU2rxV7d1q9wYdCmwBCFoJiVS7xULkB1KeM2H3ETE=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=HkoD9yf+S0JdjhQBX9GLB+qrGSymRCPhEkMAxEXjLEOdNAvKhhDlUPDxpHh2HNFdW CzpaCaMCDNxStXTyqbU29ulrMrxuqmxqG9L9O5BEemjhOeEc4b7csNcBH3AfmVw6h4 iLOxo05eaysCpZu+S4hyZQS04XqsjNjJCjw0g14w= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 353978B7C9; Mon, 13 Jan 2020 18:08:45 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id MStqpQe-bhfD; Mon, 13 Jan 2020 18:08:45 +0100 (CET) Received: from po14934vm.idsi0.si.c-s.fr (unknown [192.168.4.90]) by messagerie.si.c-s.fr (Postfix) with ESMTP id E7BFB8B7BE; Mon, 13 Jan 2020 18:08:44 +0100 (CET) Received: by localhost.localdomain (Postfix, from userid 0) id BDD3E64A1D; Mon, 13 Jan 2020 17:08:44 +0000 (UTC) Message-Id: <5b38617a2ca4f719760aafbdb6115eaad28c0640.1578934751.git.christophe.leroy@c-s.fr> In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v3 06/12] lib: vdso: __iter_div_u64_rem() is suboptimal for 32 bit time To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 13 Jan 2020 17:08:44 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Using __iter_div_ulong_rem() is suboptimal on 32 bits. Nanoseconds are only 32 bits, and VDSO data is updated every 10ms so nsec will never overflow 32 bits. Add an equivalent of __iter_div_u64_rem() but based on unsigned long to better fit with 32 bits arches. Before: gettimeofday: vdso: 1078 nsec/call clock-gettime-monotonic-raw: vdso: 1317 nsec/call clock-gettime-monotonic: vdso: 1255 nsec/call After: gettimeofday: vdso: 1032 nsec/call clock-gettime-monotonic-raw: vdso: 1312 nsec/call clock-gettime-monotonic: vdso: 1243 nsec/call Signed-off-by: Christophe Leroy --- lib/vdso/gettimeofday.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index decd3f2b37af..da15a8842825 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -38,12 +38,32 @@ u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) } #endif +static __always_inline u32 +__iter_div_ulong_rem(unsigned long dividend, u32 divisor, unsigned long *remainder) +{ + u32 ret = 0; + + while (dividend >= divisor) { + /* The following asm() prevents the compiler from + optimising this loop into a modulo operation. */ + asm("" : "+rm"(dividend)); + + dividend -= divisor; + ret++; + } + + *remainder = dividend; + + return ret; +} + static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, struct __kernel_timespec *ts) { const struct vdso_timestamp *vdso_ts = &vd->basetime[clk]; u64 cycles, last, sec, ns; u32 seq; + unsigned long nsec; do { seq = vdso_read_begin(vd); @@ -54,7 +74,7 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, return -1; ns += vdso_calc_delta(cycles, last, vd->mask, vd->mult); - ns >>= vd->shift; + nsec = ns >> vd->shift; sec = vdso_ts->sec; } while (unlikely(vdso_read_retry(vd, seq))); @@ -62,8 +82,8 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, * Do this outside the loop: a race inside the loop could result * in __iter_div_u64_rem() being extremely slow. */ - ts->tv_sec = sec + __iter_div_u64_rem(ns, NSEC_PER_SEC, &ns); - ts->tv_nsec = ns; + ts->tv_sec = sec + __iter_div_ulong_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; } -- 2.13.3 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4174DC33CA9 for ; Mon, 13 Jan 2020 17:25:15 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E4626206DA for ; Mon, 13 Jan 2020 17:25:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="HkoD9yf+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E4626206DA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=c-s.fr Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 47xL7q0wkBzDqNB for ; Tue, 14 Jan 2020 04:25:11 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=c-s.fr (client-ip=93.17.236.30; helo=pegase1.c-s.fr; envelope-from=christophe.leroy@c-s.fr; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=c-s.fr Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=c-s.fr header.i=@c-s.fr header.a=rsa-sha256 header.s=mail header.b=HkoD9yf+; dkim-atps=neutral Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 47xKmx4FGyzDq9V for ; Tue, 14 Jan 2020 04:08:49 +1100 (AEDT) Received: from localhost (mailhub1-int [192.168.12.234]) by localhost (Postfix) with ESMTP id 47xKmm0TxYz9txyy; Mon, 13 Jan 2020 18:08:40 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=HkoD9yf+; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id AhN4_motg1-N; Mon, 13 Jan 2020 18:08:40 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47xKml6VQMz9txyq; Mon, 13 Jan 2020 18:08:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1578935319; bh=cvbU2rxV7d1q9wYdCmwBCFoJiVS7xULkB1KeM2H3ETE=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=HkoD9yf+S0JdjhQBX9GLB+qrGSymRCPhEkMAxEXjLEOdNAvKhhDlUPDxpHh2HNFdW CzpaCaMCDNxStXTyqbU29ulrMrxuqmxqG9L9O5BEemjhOeEc4b7csNcBH3AfmVw6h4 iLOxo05eaysCpZu+S4hyZQS04XqsjNjJCjw0g14w= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 353978B7C9; Mon, 13 Jan 2020 18:08:45 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id MStqpQe-bhfD; Mon, 13 Jan 2020 18:08:45 +0100 (CET) Received: from po14934vm.idsi0.si.c-s.fr (unknown [192.168.4.90]) by messagerie.si.c-s.fr (Postfix) with ESMTP id E7BFB8B7BE; Mon, 13 Jan 2020 18:08:44 +0100 (CET) Received: by localhost.localdomain (Postfix, from userid 0) id BDD3E64A1D; Mon, 13 Jan 2020 17:08:44 +0000 (UTC) Message-Id: <5b38617a2ca4f719760aafbdb6115eaad28c0640.1578934751.git.christophe.leroy@c-s.fr> In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v3 06/12] lib: vdso: __iter_div_u64_rem() is suboptimal for 32 bit time To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Date: Mon, 13 Jan 2020 17:08:44 +0000 (UTC) X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" Using __iter_div_ulong_rem() is suboptimal on 32 bits. Nanoseconds are only 32 bits, and VDSO data is updated every 10ms so nsec will never overflow 32 bits. Add an equivalent of __iter_div_u64_rem() but based on unsigned long to better fit with 32 bits arches. Before: gettimeofday: vdso: 1078 nsec/call clock-gettime-monotonic-raw: vdso: 1317 nsec/call clock-gettime-monotonic: vdso: 1255 nsec/call After: gettimeofday: vdso: 1032 nsec/call clock-gettime-monotonic-raw: vdso: 1312 nsec/call clock-gettime-monotonic: vdso: 1243 nsec/call Signed-off-by: Christophe Leroy --- lib/vdso/gettimeofday.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index decd3f2b37af..da15a8842825 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -38,12 +38,32 @@ u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) } #endif +static __always_inline u32 +__iter_div_ulong_rem(unsigned long dividend, u32 divisor, unsigned long *remainder) +{ + u32 ret = 0; + + while (dividend >= divisor) { + /* The following asm() prevents the compiler from + optimising this loop into a modulo operation. */ + asm("" : "+rm"(dividend)); + + dividend -= divisor; + ret++; + } + + *remainder = dividend; + + return ret; +} + static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, struct __kernel_timespec *ts) { const struct vdso_timestamp *vdso_ts = &vd->basetime[clk]; u64 cycles, last, sec, ns; u32 seq; + unsigned long nsec; do { seq = vdso_read_begin(vd); @@ -54,7 +74,7 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, return -1; ns += vdso_calc_delta(cycles, last, vd->mask, vd->mult); - ns >>= vd->shift; + nsec = ns >> vd->shift; sec = vdso_ts->sec; } while (unlikely(vdso_read_retry(vd, seq))); @@ -62,8 +82,8 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, * Do this outside the loop: a race inside the loop could result * in __iter_div_u64_rem() being extremely slow. */ - ts->tv_sec = sec + __iter_div_u64_rem(ns, NSEC_PER_SEC, &ns); - ts->tv_nsec = ns; + ts->tv_sec = sec + __iter_div_ulong_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; } -- 2.13.3 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 382D2C33CA9 for ; Mon, 13 Jan 2020 17:10:53 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E584C2075B for ; Mon, 13 Jan 2020 17:10:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="LZwRvD9k"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="HkoD9yf+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E584C2075B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=c-s.fr Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Date:To:Subject: From:References:In-Reply-To:Message-Id:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZqfRYXcakM7mVQjrtVIr21saDycfw74eavxseF0HEO0=; b=LZwRvD9k5jIaNXgBaTo6tcmibM e7JjgBXsYqRt6GWJ1wOJdYw6w30WamUGvh/NaE1v5HqM4vHOp2aeXxhm2sWwED82tep/Ez7BmwauU PEJfYGrzhi4YhqCbjDWLm2dm78YVvIQsDqDLjFptRm63p8alSFGEC/aFD00Xnx6iOa2Knkk8EMr8t DQ/RYZU5ePc2FLp5CP9JX+43/ZhKGQiUWBaOuEoPczb+HehruJJMiXC9FuRkgmMGFaakHPbtVc77i fiTgBpzaCqqBXFGyL70X0IY0NepcYEcYC9OCzl3JCUODMyOw41lZ2qdJaixNO1+C7E8XEi0hnmaLS m4LqMSJA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ir3Ep-0005No-PG; Mon, 13 Jan 2020 17:10:51 +0000 Received: from pegase1.c-s.fr ([93.17.236.30]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1ir3Ct-0002OF-08 for linux-arm-kernel@lists.infradead.org; Mon, 13 Jan 2020 17:08:53 +0000 Received: from localhost (mailhub1-int [192.168.12.234]) by localhost (Postfix) with ESMTP id 47xKmm0TxYz9txyy; Mon, 13 Jan 2020 18:08:40 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=HkoD9yf+; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id AhN4_motg1-N; Mon, 13 Jan 2020 18:08:40 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47xKml6VQMz9txyq; Mon, 13 Jan 2020 18:08:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1578935319; bh=cvbU2rxV7d1q9wYdCmwBCFoJiVS7xULkB1KeM2H3ETE=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=HkoD9yf+S0JdjhQBX9GLB+qrGSymRCPhEkMAxEXjLEOdNAvKhhDlUPDxpHh2HNFdW CzpaCaMCDNxStXTyqbU29ulrMrxuqmxqG9L9O5BEemjhOeEc4b7csNcBH3AfmVw6h4 iLOxo05eaysCpZu+S4hyZQS04XqsjNjJCjw0g14w= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 353978B7C9; Mon, 13 Jan 2020 18:08:45 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id MStqpQe-bhfD; Mon, 13 Jan 2020 18:08:45 +0100 (CET) Received: from po14934vm.idsi0.si.c-s.fr (unknown [192.168.4.90]) by messagerie.si.c-s.fr (Postfix) with ESMTP id E7BFB8B7BE; Mon, 13 Jan 2020 18:08:44 +0100 (CET) Received: by localhost.localdomain (Postfix, from userid 0) id BDD3E64A1D; Mon, 13 Jan 2020 17:08:44 +0000 (UTC) Message-Id: <5b38617a2ca4f719760aafbdb6115eaad28c0640.1578934751.git.christophe.leroy@c-s.fr> In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v3 06/12] lib: vdso: __iter_div_u64_rem() is suboptimal for 32 bit time To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Date: Mon, 13 Jan 2020 17:08:44 +0000 (UTC) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200113_090851_369571_424D330E X-CRM114-Status: GOOD ( 11.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Using __iter_div_ulong_rem() is suboptimal on 32 bits. Nanoseconds are only 32 bits, and VDSO data is updated every 10ms so nsec will never overflow 32 bits. Add an equivalent of __iter_div_u64_rem() but based on unsigned long to better fit with 32 bits arches. Before: gettimeofday: vdso: 1078 nsec/call clock-gettime-monotonic-raw: vdso: 1317 nsec/call clock-gettime-monotonic: vdso: 1255 nsec/call After: gettimeofday: vdso: 1032 nsec/call clock-gettime-monotonic-raw: vdso: 1312 nsec/call clock-gettime-monotonic: vdso: 1243 nsec/call Signed-off-by: Christophe Leroy --- lib/vdso/gettimeofday.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index decd3f2b37af..da15a8842825 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -38,12 +38,32 @@ u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) } #endif +static __always_inline u32 +__iter_div_ulong_rem(unsigned long dividend, u32 divisor, unsigned long *remainder) +{ + u32 ret = 0; + + while (dividend >= divisor) { + /* The following asm() prevents the compiler from + optimising this loop into a modulo operation. */ + asm("" : "+rm"(dividend)); + + dividend -= divisor; + ret++; + } + + *remainder = dividend; + + return ret; +} + static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, struct __kernel_timespec *ts) { const struct vdso_timestamp *vdso_ts = &vd->basetime[clk]; u64 cycles, last, sec, ns; u32 seq; + unsigned long nsec; do { seq = vdso_read_begin(vd); @@ -54,7 +74,7 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, return -1; ns += vdso_calc_delta(cycles, last, vd->mask, vd->mult); - ns >>= vd->shift; + nsec = ns >> vd->shift; sec = vdso_ts->sec; } while (unlikely(vdso_read_retry(vd, seq))); @@ -62,8 +82,8 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, * Do this outside the loop: a race inside the loop could result * in __iter_div_u64_rem() being extremely slow. */ - ts->tv_sec = sec + __iter_div_u64_rem(ns, NSEC_PER_SEC, &ns); - ts->tv_nsec = ns; + ts->tv_sec = sec + __iter_div_ulong_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; } -- 2.13.3 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel