From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, USER_AGENT_MUTT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57236C43219 for ; Wed, 1 May 2019 16:01:58 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 222DF2085A for ; Wed, 1 May 2019 16:01:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="aHwBzg0+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 222DF2085A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=w5y5MNn23SiVP2pTbaSVF6L4KPoVEMtpFhcPkNoQWAI=; b=aHwBzg0+BlbRV+ mzQ4ec+7TVtQqPYLIRwhU56Gw/biG85vFtER5Pmw1toQfYHV3XltFVPziOSybYO1RFsb7dA0yclKp J4fgTJf23q4GXOH03Q9Wo6mfv3ojJ7STd0nzmGJrTIJ/Lgnlzyk6R9SSMEaFgFOESNXtgWWiwWL1V buIHFj5fYNNzhJtIXcBhP2py8/OzP2nk6jHleexUjWlGdV/pAIy4BGCbUy6k1ak+H/CsEW1LcxTqz hxOWuvEyABqJW6EDSWMNiY2nEwljB2qbFp6xpq1/ytBR5qnUoD/EZEBIVeKLzEvD4ERyiCFAYP6dP nA+KHBzvfGSi+6y8RhEw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hLrg5-0006Jg-KR; Wed, 01 May 2019 16:01:49 +0000 Received: from foss.arm.com ([217.140.101.70]) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hLrg2-0006JE-6M for linux-arm-kernel@lists.infradead.org; Wed, 01 May 2019 16:01:47 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D73FBA78; Wed, 1 May 2019 09:01:44 -0700 (PDT) Received: from fuggles.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 719143F719; Wed, 1 May 2019 09:01:43 -0700 (PDT) Date: Wed, 1 May 2019 17:01:40 +0100 From: Will Deacon To: Jan Glauber Subject: Re: [RFC] Disable lockref on arm64 Message-ID: <20190501160140.GC28109@fuggles.cambridge.arm.com> References: <20190429145159.GA29076@hc> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190429145159.GA29076@hc> User-Agent: Mutt/1.11.1+86 (6f28e57d73f2) () X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190501_090146_273444_58D8E725 X-CRM114-Status: GOOD ( 15.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peterz@infradead.org, "catalin.marinas@arm.com" , "linux-kernel@vger.kernel.org" , Jayachandran Chandrasekharan Nair , torvalds@linux-foundation.org, "linux-arm-kernel@lists.infradead.org" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Jan, [+Peter and Linus, since they enjoy this stuff] On Mon, Apr 29, 2019 at 02:52:11PM +0000, Jan Glauber wrote: > I've been looking into performance issues that were reported for several > test-cases, for instance an nginx benchmark. Could you share enough specifics here so that we can reproduce the issue locally, please? That would help us in our attempts to develop a fix without simply disabling the option for everybody else. > It turned out the issue we have on ThunderX2 is the file open-close sequence > with small read sizes. If the used files are opened read-only the > lockref code (enabled by ARCH_USE_CMPXCHG_LOCKREF) is used. > > The lockref CMPXCHG_LOOP uses an unbound (as long as the associated > spinlock isn't taken) while loop to change the lock count. This behaves > badly under heavy contention (~25x retries for one cmpxchg to succeed > with 28 threads operating on the same file). In case of a NUMA system > it also behaves badly as the access from the other socket is much slower. It's surprising that this hasn't been reported on x86. I suspect their implementation of cmpxchg is a little more forgiving under contention. > The fact that on ThunderX2 cpu_relax() turns only into one NOP > instruction doesn't help either. On Intel pause seems to block the thread > much longer, avoiding the heavy contention thereby. NOPing out the yield instruction seems like a poor choice for an SMT CPU such as TX2. That said, the yield was originally added to cpu_relax() as a scheduling hint for QEMU. > With the queued spinlocks implementation I can see a major improvement > when I disable lockref. A trivial open-close test-case improves by > factor 2 while system time is decreasing also 2x. Looking at kernel compile > and dbench numbers didn't show any regression with lockref disabled. > > Can we simply disable lockref? Is anyone else seeing this issue? Is there > an arm64 platform that actually implements yield? There are two issues with disabling lockref like this: 1. It's a compile-time thing, so systems that would benefit from the code are unfairly penalised. 2. You're optimising for the contended case at the cost of the uncontended case, which should actually be the common case as well. Now, nobody expects contended CAS to scale well, so the middle ground probably involves backing off to the lock under contention, a bit like an optimistic trylock(). Unfortunately, that will need some tuning, hence my initial request for a reproducer. Cheers, Will _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel