From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S945738AbdDTSXf (ORCPT ); Thu, 20 Apr 2017 14:23:35 -0400 Received: from mail-by2nam01on0083.outbound.protection.outlook.com ([104.47.34.83]:13568 "EHLO NAM01-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S943693AbdDTSXc (ORCPT ); Thu, 20 Apr 2017 14:23:32 -0400 Authentication-Results: infradead.org; dkim=none (message not signed) header.d=none;infradead.org; dmarc=none action=none header.from=caviumnetworks.com; Date: Thu, 20 Apr 2017 21:23:18 +0300 From: Yury Norov To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Ingo Molnar , Arnd Bergmann , Catalin Marinas , Will Deacon , Jan Glauber Subject: Re: [PATCH 3/3] arm64/locking: qspinlocks and qrwlocks support Message-ID: <20170420182318.4ddtfiobxz6hgbo4@yury-N73SV> References: <1491860104-4103-1-git-send-email-ynorov@caviumnetworks.com> <1491860104-4103-4-git-send-email-ynorov@caviumnetworks.com> <20170413181212.y3ezah76qoztxhnn@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170413181212.y3ezah76qoztxhnn@hirez.programming.kicks-ass.net> User-Agent: NeoMutt/20170113 (1.7.2) X-Originating-IP: [176.59.44.254] X-ClientProxiedBy: AM5PR0101CA0006.eurprd01.prod.exchangelabs.com (10.169.240.16) To DM2PR0701MB1277.namprd07.prod.outlook.com (10.161.225.15) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3eecb7a7-f0c2-4af1-8d9c-08d4881a58a1 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:DM2PR0701MB1277; X-Microsoft-Exchange-Diagnostics: 1;DM2PR0701MB1277;3:nhsx6MW92+pRw3V/5kFAXtoV3tsMY5Qot8w9SPWFvIHjTkOj2hpiRb4RQONjspYsRVfDTrwS9tqfwXHVUR0YlMV/pcj5j99JA0eFgZNr4w5MD8xCxAU++0aBJ5ynBPy4c7koq1N1UDiWv8WZ2UvRo17aPqX0VScVKT+nSntwr9q6qygERfs4IsUshHAhkbrfZBS2usrEbwl4vJo3qRyEwZuFPdH64E1NGfF8yq+EMUH4OMBeuUnrXbuJ1sxal5Xrd2CX3+H4QdSfxy/TWx0Qi/G7v77BNM9dEa4C6fc+MZ9UjoFKH0cgCy9At5I4oUkI6wkIFgve0+5AboHPgmoSeQ==;25:AbPoU+5Bo8a3AbfzyoLryXJcxDN8X8c/91h7ypVS6KHm3MlpIzrkkIV2bfvbf0OnLhYikEkFbimyE9GcHIdCUfKw03i9+CTpr8fc7fRlGj2+XL1TTBBqoX3/rXXv4ap6T9LACE7alV0Ld5cdDMmeM/JX54oXnSVYXOHoXfXz5FNb14Gojmdg8k7V33WpOX2EDWKqGaEw5KfSwnyCIjQjCFGGXMdWvwaR4q+r2ohjIfnqQCgPeVZc8Z0ATexSanvoeXa3mopSTf/i4YaGPrqoIk6hg27JbCpMwmHXKj5rCT/02Bno4xJ4Z+2Y5QZnJvFBxfLTilzEM6KEW3+vTjoYjdel0kfxv5n4nBtLSom84Ey5luYAem7/6/SO7ELOZXI7L0bgR1ZNnOhNGgH2ygsgnlhTqUrHdDSPutcWkVu1SBGkXJy1VM+HHZuxi7RpKY24Q4hltl/4LxYS1AAFPIOJqQ== X-Microsoft-Exchange-Diagnostics: 1;DM2PR0701MB1277;31:Kbk0y/5aluJQVmTWw1ozk9MSOHPOfvxhz8WG7MflMDm8FhRowrVFb3pHs9AyAAw9IqCK/bGttNkQBbzdlVu8DjvfwJlq5JeL1smYeJFiijSEMiezJTNqKoQ/1dIkXmkSdgPJv97rnvGqgpI2zoHXJq8z/eYCcrtKb29vIwsBMKRP6+6zfXj3P2a1DnZd1l5+pqtXh1uBo+w5vesL5e+x+Sdmgl/5YvNmDdWIHuR7VWq2eQ8ETtdN19x6mLERol54;20:2WuomjcG+he6yh0IisOL43QYMkuJp/pGB8/k/4PeWFOqDUaYuB9YmELrW2YQQa9K2rZD4/RMj5SOj8Nq2DiMNe42nAsdj5K3CodzF34DEfsqVR5oncPO/OdgSAfAsGXgJOdHFwP5ZhQ6cC9I4jfXEf15dCtPhAadEhy38rkr806fmrGJFs+y9F61G7al888R5JVz/2IO6AQpiqzTuqoeWIP4CwWouxHuASkNpmSqA6qFw3L1N9XVgyLcIzUo3l+S0OTKWxB8qJwC1Z/BONZeS/XGhk3RfGVX6w6+qf9P5Usjr6dR42aAfypfxouUcNQvcwJo7ZJGdmUDOggbRG34bl67Ae2UgGSRQpIg/PgUPH5QOlDW4xfy6UCK9TKuvYW428y2yqZh0Q5FTUDZ+j7vWnGGOqITcAd/xAjiQNV4hIAABGxSSvY12ghmewk+RQznR/eYHJlH2tIIgXJZeiz27zzlGpr6gTuLVoTO7vhZ5DkyT6GOvzKRWLt82MXsYg/IKlwdt1TEe1sg9z5pjmCcb5dGth81t+qRFCBihZa2Qzs+lhIX1POzHj95LWcYJlfmybOI53wFE/vRqepf0Lk9HUWvnCahCtPQDX1Nse2jKSk= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3002001)(6041248)(20161123564025)(20161123560025)(20161123555025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(6072148);SRVR:DM2PR0701MB1277;BCL:0;PCL:0;RULEID:;SRVR:DM2PR0701MB1277; X-Microsoft-Exchange-Diagnostics: 1;DM2PR0701MB1277;4:wapiOzWJRK4eD5zfk05zNyWMnLMJtaG+2XPdB9tB2oGOXW5Ah+W+NHD4EkWgPi0sB7sJeHzbL5w6CLlnyt37gc23joF/KR2zNbFH8qyB0H8ao6v+p0kW9rfAsvh47LN8GLVaxzpwSns1lJG7F+0QtFcZPliNVbH6MGUT8TfqGUKMvgLwkNqmnsrF1ZxhZdZwGzd3qr4PFnqk8CDhz7mDgCEM2IJuvzAtRf+KBwjXk2X3eJ05ZVidwnWPqTXbM/rUOWnXZAa+Fscj8UXR1XrRpeKRs2xkD0EFu3WffsDnSXuGpr5yxUyxyZ6jiMLykpIAdx7vgxZpxk3XzJ+ePHw52iTiXn9ljcu/ym59Gi8QbE97bofp4NFtY3ntHjXXB4SpxfW7pnWZQFoAa2yaNw3Jl/z1EBNrrGs2sbflLp+J+isF0fwpCzaUuAnC31VywxUNL7fYgfmommZep2ZqpSx8MI20WUvFv3Z2x8A57RmlvihXRu9HU8v1SlEvhg3jmi0lkFQa7/+QTmeSjgjxWrRV+P3IhvhHGrGXQnCuqhe2QCRRFzbGfV8Q1jaMWsLNCsn6Ke7Jq2QzV2r8i+cOvA1zVy4PSm0LK0b8OPbb7M0EUkSK2CR0hdcrPUJkqeeTUW00FU1+TOLSE+1T/yZ47AfXvYl+OgmLG1Y4erWDDSGlUXX+EHlpO6F/4SyfOQSLeERRZiytipGCOKZ9XBVMzJOgAQ== X-Forefront-PRVS: 02830F0362 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6069001)(6009001)(39410400002)(39400400002)(39850400002)(39450400003)(39840400002)(52314003)(24454002)(33716001)(305945005)(66066001)(25786009)(6666003)(42186005)(189998001)(47776003)(76506005)(107886003)(50466002)(4326008)(5660300001)(110136004)(38730400002)(7736002)(6916009)(6486002)(81166006)(6246003)(6496005)(33646002)(53936002)(9686003)(23726003)(1076002)(54356999)(50986999)(229853002)(3846002)(54906002)(76176999)(2950100002)(2906002)(8676002)(6116002);DIR:OUT;SFP:1101;SCL:1;SRVR:DM2PR0701MB1277;H:localhost;FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;DM2PR0701MB1277;23:en4JUz0b4vq6zGcKqrdrxoujIQfZE6GGAiL+Ulh?= =?us-ascii?Q?bMn3+IshPcLQc5E2uLUTN5K8LpL+FW8SEiCKQ0/gWyX+lGKi9vzq8egmg8Zq?= =?us-ascii?Q?0KAoijYMsUvI4QnhodmWsGGRtlZAzt6XAnyt+FPrv4tUFs37h5e9q0BCCinH?= =?us-ascii?Q?zz8JCTxAeKyvW3wFBae/yNKhlYDhSm62d9S+HC8GgURg7cvIYW3QqRQFBuRR?= =?us-ascii?Q?YQhrwLce0cAnwNmcGziMlXJVNIZm5pXk7X918yuZBQGPz5fiF2Yn0DG7Wzou?= =?us-ascii?Q?J1HarIwT8GJvT82962GzBmFzB1IrSFDCEwDOIP9PMB9mS4S6CgG4bNTmcdFq?= =?us-ascii?Q?/0uiEeX3BRC2EDYDsKpQvaDANkhviw5PDhUOd5Uwc6Dx1nKgzjPtfuEpmajq?= =?us-ascii?Q?C+9ML4jPz0knLVy+CclpnDUjqxxsHp73He4VB7g4fo5FuZIVI0qmhr2mf9+w?= =?us-ascii?Q?iCyozxVfy2eSbMODvYneseZmmyHkEyo3GG5QN2H1MDS+djBFazApAzcx78xv?= =?us-ascii?Q?P3YolQafc6Y+QP3F44xIjx0U+I7kc1BPwreIpdLYStFB97galX6WpPIBMiMe?= =?us-ascii?Q?gnoR9bxeCbXtGFL00zfBvhzEKP4xaaChLfXS3+pBLkriZ510ipiHRIoG9DC2?= =?us-ascii?Q?Cr036NO9UTsjhySFN429vLQbGH2SNgoY0zaUe0twCIY8hufdGPToI9U+Ruda?= =?us-ascii?Q?MjSqUWS+hL/oVoaeaVz1o1Z/3J/6PF5a3l26SST4bWisLcGuV03NEd/jKKSc?= =?us-ascii?Q?danqSXBVlK25IBZAkHqG4RDHT5zl6UBerMc+p1kZEIrIOJuxQg88oIL+snUY?= =?us-ascii?Q?so2Z8TVbq4QtP71c2CtOi1G7sBrc2MeOlWa178BI9Kl26Cy8KLAI2WjV7Ayd?= =?us-ascii?Q?97icRe0WF/Wd4TxKYdSnlflsSGRWMNgzP7B3c11UNX96mNYRyQCim7vBFYOF?= =?us-ascii?Q?1L3VpSndHmIkLZmz08UOr0znk+hAOfdNv40H01RKNifK2kyH/BlRSw3q+l8H?= =?us-ascii?Q?ogMaxQYexcoNdiLR8YJfMH6uWRWoKwkbhITFFgg3Y0sccDq0iDIeshuouB5R?= =?us-ascii?Q?BcQ2NoQTG8F0EZgXTvd8aHEbTxY4nQYHXA5Zg0YiK4GVMQuzo58hhz33Lfg0?= =?us-ascii?Q?fUX2f2rF+iWdC0TKnGfn5dZj3TbzA6i9a?= X-Microsoft-Exchange-Diagnostics: 1;DM2PR0701MB1277;6:OU+XKkU5ZE071V3utJ7eawE5G+6JMS0MXeg0pNNhQ/7nOQbHWsrsrwepN3geN2XRoxSgNUVeL5DgUdbjcdLuiXkGY2KAR/t/jynx4V01Y5W4kMUNunM3ncswbPoiJk8bQ+i/fMP1hX8JJdVifk+2QqHPlyx9oce0dUlNqp2Y7nIaQaGTpbiIj3lgxvgHblCDROA2XaojiqzxvGjTHe8sztOOHaJOIpViFAiF/BpoQsj4LPiJ9JJg4ZM82tHVYlKCbG+p1Vv9irH9CT2Gd7ozTONVso7RhiwuzrGLZiFjfsA/SqmCFhQdgzFaSrKF0Yl+AFpbMZPiDw0GZYJ0hDYxTXRIZw566GxC6iu7BAXPpdorCSTSquUUNFRcivtL5+R/4KOyClU5fcJhgVBuI8hwlrwv1Y2DkW6w9z3DzGynqUTJGvwiKtOu1bQ1XsymtXFsyCXzTp3uDBRonauxBhccAjNphoc/tWsh6FSTkk7uXCZu9izjoeRmGZidYrgBdcqA0uf/resxJftNmjwEcl/Riw==;5:IYA13VzpwwW/SLD/9ggl198bWVk8heNThDaKQ7yOr68pPjEK2mQimcG3EcBALOj+ODrWRcxQYCAakjr9eFowDAuQWnkIDjpDmAhEdSj6cB0xPF7o2tKO/MdZ56nBuMP0QwOeL6PEGJq2J4yP+GIC8w==;24:Iy5Vf43Xzjo9BprO9JKGT9uzrpsXuEHXTZAhj2bqjJxiwYqNv+PEV6SqHGLDHnYClf+oyCVEhJPw+baxSfCCFi1PbSwFoUdT0qAK0c0N0D8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM2PR0701MB1277;7:yWy8TIi47LWnF0/UtXR0c5pr8vIhCspPpW+iT8NWgtcNhmeB/knHMZnMDBmoXwRoiQ1Vt5JxoKcTmN6KEkpRFYLQiotoH1I91XxPG6jIpSIdtJ2UJZD+6HM9o4gJz4lFw3rs1KSjOvecrUPrI8uzoaOEsehDea4/6LIEDuu0yj+av8abkJio3lO3BoJLhBeSMFne49n6ZXEkesMGoCEIgPj2c5xvgxcYwQbH84hpPwnUSvPNhrPhJH3lOX6EvTF/hdxreF85Vfagm1WlYJpPI7hGCxC1wgUHFoJesgYSs1tI+NZjYXpR5RZtFMuLttUF4B5aGyVhsXUs0AIceW9ZZA== X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Apr 2017 18:23:29.8558 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR0701MB1277 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Apr 13, 2017 at 08:12:12PM +0200, Peter Zijlstra wrote: > On Tue, Apr 11, 2017 at 01:35:04AM +0400, Yury Norov wrote: > > > +++ b/arch/arm64/include/asm/qspinlock.h > > @@ -0,0 +1,20 @@ > > +#ifndef _ASM_ARM64_QSPINLOCK_H > > +#define _ASM_ARM64_QSPINLOCK_H > > + > > +#include > > + > > +#define queued_spin_unlock queued_spin_unlock > > +/** > > + * queued_spin_unlock - release a queued spinlock > > + * @lock : Pointer to queued spinlock structure > > + * > > + * A smp_store_release() on the least-significant byte. > > + */ > > +static inline void queued_spin_unlock(struct qspinlock *lock) > > +{ > > + smp_store_release((u8 *)lock, 0); > > +} > > I'm afraid this isn't enough for arm64. I suspect you want your own > variant of queued_spin_unlock_wait() and queued_spin_is_locked() as > well. > > Much memory ordering fun to be had there. Hi Peter, Is there some test to reproduce the locking failure for the case. I ask because I run loctorture for many hours on my qemu (emulating cortex-a57), and I see no failures in the test reports. And Jan did it on ThunderX, and Adam on QDF2400 without any problems. So even if I rework those functions, how could I check them for correctness? Anyway, regarding the queued_spin_unlock_wait(), is my understanding correct that you assume adding smp_mb() before entering the for(;;) cycle, and using ldaxr/strxr instead of atomic_read()? Yury From mboxrd@z Thu Jan 1 00:00:00 1970 From: ynorov@caviumnetworks.com (Yury Norov) Date: Thu, 20 Apr 2017 21:23:18 +0300 Subject: [PATCH 3/3] arm64/locking: qspinlocks and qrwlocks support In-Reply-To: <20170413181212.y3ezah76qoztxhnn@hirez.programming.kicks-ass.net> References: <1491860104-4103-1-git-send-email-ynorov@caviumnetworks.com> <1491860104-4103-4-git-send-email-ynorov@caviumnetworks.com> <20170413181212.y3ezah76qoztxhnn@hirez.programming.kicks-ass.net> Message-ID: <20170420182318.4ddtfiobxz6hgbo4@yury-N73SV> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On Thu, Apr 13, 2017 at 08:12:12PM +0200, Peter Zijlstra wrote: > On Tue, Apr 11, 2017 at 01:35:04AM +0400, Yury Norov wrote: > > > +++ b/arch/arm64/include/asm/qspinlock.h > > @@ -0,0 +1,20 @@ > > +#ifndef _ASM_ARM64_QSPINLOCK_H > > +#define _ASM_ARM64_QSPINLOCK_H > > + > > +#include > > + > > +#define queued_spin_unlock queued_spin_unlock > > +/** > > + * queued_spin_unlock - release a queued spinlock > > + * @lock : Pointer to queued spinlock structure > > + * > > + * A smp_store_release() on the least-significant byte. > > + */ > > +static inline void queued_spin_unlock(struct qspinlock *lock) > > +{ > > + smp_store_release((u8 *)lock, 0); > > +} > > I'm afraid this isn't enough for arm64. I suspect you want your own > variant of queued_spin_unlock_wait() and queued_spin_is_locked() as > well. > > Much memory ordering fun to be had there. Hi Peter, Is there some test to reproduce the locking failure for the case. I ask because I run loctorture for many hours on my qemu (emulating cortex-a57), and I see no failures in the test reports. And Jan did it on ThunderX, and Adam on QDF2400 without any problems. So even if I rework those functions, how could I check them for correctness? Anyway, regarding the queued_spin_unlock_wait(), is my understanding correct that you assume adding smp_mb() before entering the for(;;) cycle, and using ldaxr/strxr instead of atomic_read()? Yury