From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52434C433E0 for ; Thu, 28 Jan 2021 07:37:51 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E43996146D for ; Thu, 28 Jan 2021 07:37:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E43996146D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nokia.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:Date:Message-ID:From: References:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7rbiTW6xcMhy7ysCdCVIWBerMY65CowIQVbtPOwbj1k=; b=PDmwmxiFyfCrnaoYkWjTWbq/x 5odINgL2yzMuaka6qulvzp0iNi4wOMqCBkpDS6Ks1aDl+X1grvt5vehjdp5/X6xZxMTBACLMvcQBd aScijcDHe7ByxH8IK1G6cU1hWTbkwlcV+dq8LHMT/GUGbkPgkQjIAhiQgjkFopAaeXlwzY72Kj5MO Jh6r2wwJ3c2Vgm5m+SYEdgBYLK9iBKBx9wKdji7vg2MdssEYvmhIBvL+QwF1tk6D2N3rWIUlF7yXj /oOSaEBU5oQTJfsMymeFAwmwLO3qepkao2ihIDDutnxfayBTEkSGTC7oYU+yXrmtVmquneV6MLwQH 8tzDBgEZA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l51qz-0001iS-JX; Thu, 28 Jan 2021 07:36:33 +0000 Received: from mail-vi1eur05on2123.outbound.protection.outlook.com ([40.107.21.123] helo=EUR05-VI1-obe.outbound.protection.outlook.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l51qv-0001hP-Hd for linux-arm-kernel@lists.infradead.org; Thu, 28 Jan 2021 07:36:31 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Z3/X2hBQm+4bcjQikWuvR1vgaS7sj60fH9AIr6ApYSF91FV3Vaihl2D6/lJn+0ymwVnAYAd5YAai9vKqIokcT0G/vTH6ybrMwPSXvFyB7dxLyP8yUNrEc/qgWofKJizz7GnVGVUgCwvjAYBxw4FTNUU7vvrXTXl1BOTg8+GKMyv/tk2WQgzuC1nOInGKXJDXI/6Q8FQuxMNtV48ykR2w8+YsjbQZgTEvaWwi4SSdgjLyBNI5LaA1wdH6Gso4C7SmU0veAwGhYETW4f6yU52ejRzukEBDq436Q6EKrGVVFkuhA+vF/Mx23Mc0X9vwKZmvDL8yyKWm88gPxIZn3w/e7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iL13ODPs4W+ymXpahJMDU2VwtYyuHAJswRQqzrpX/KI=; b=QOyLuzlUMMIYkwh/QuPheu2MEvU9d5R84cym+RIvCevhnPQqQYsn6jWh2mgtzMEgp9eslYwh9PVb7uOjpyd/PvSw+AAWpfEEdnjZJJdU2Qyhz58e7D2zmjs/0wT73JJk1FEkhGS/tXPGZnSkbRk9MSmYGO9j8OgTdDW5NA3uUHR53AoTveYAL51QaFmqFrAQM0Gx+ltrUCImA4ZFmnCqOxaEZFoO7GY8H7DFDe4NOoUs8hfmjR1FfWCXAQuyroytT6bPCVbpmgoDrnL0jlqmOao9RM4Kb64OmK0FnndCOpFf1VLZXac//9ljkjniS+xcvmWCC7mzb+BbsOCjbKRrSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nokia.com; dmarc=pass action=none header.from=nokia.com; dkim=pass header.d=nokia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nokia.onmicrosoft.com; s=selector1-nokia-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iL13ODPs4W+ymXpahJMDU2VwtYyuHAJswRQqzrpX/KI=; b=v5oVmlAgipzGCMhupYVQxZ1SKeJXWscmRMHD/q11DkfUdut2wS1p6reeKYGLiPfbw4qK8EAVZbghMupfhDEsy801KeKPhyR8rcB4o6SQTuQYpE/Roqx3r4FiaBBflYw5cKOiAXVQT6P7y5ulp8X1+7coegwU6gsRT5YBEO7KvpQ= Authentication-Results: lists.infradead.org; dkim=none (message not signed) header.d=none;lists.infradead.org; dmarc=none action=none header.from=nokia.com; Received: from (2603:10a6:208:6e::15) by AM8PR07MB7393.eurprd07.prod.outlook.com (2603:10a6:20b:246::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.7; Thu, 28 Jan 2021 07:36:27 +0000 Received: from AM0PR07MB4531.eurprd07.prod.outlook.com ([fe80::e965:2884:260b:b29a]) by AM0PR07MB4531.eurprd07.prod.outlook.com ([fe80::e965:2884:260b:b29a%3]) with mapi id 15.20.3825.008; Thu, 28 Jan 2021 07:36:27 +0000 Subject: Re: [PATCH 1/2] qspinlock: Ensure writes are pushed out of core write buffer To: Will Deacon References: <20210127200109.16412-1-alexander.sverdlin@nokia.com> <20210127222158.GB848@willie-the-truck> From: Alexander Sverdlin Message-ID: Date: Thu, 28 Jan 2021 08:36:24 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 In-Reply-To: <20210127222158.GB848@willie-the-truck> Content-Language: en-US X-Originating-IP: [131.228.32.168] X-ClientProxiedBy: AM0PR04CA0055.eurprd04.prod.outlook.com (2603:10a6:208:1::32) To AM0PR07MB4531.eurprd07.prod.outlook.com (2603:10a6:208:6e::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from ulegcpsvhp1.emea.nsn-net.net (131.228.32.168) by AM0PR04CA0055.eurprd04.prod.outlook.com (2603:10a6:208:1::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.17 via Frontend Transport; Thu, 28 Jan 2021 07:36:26 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 5e9e0bea-5f49-49ae-226d-08d8c35f6bab X-MS-TrafficTypeDiagnostic: AM8PR07MB7393: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zjAKS4htau/AFMtwOYOFjS2Qpoxb4Yh2xMDTJqQBVg0JtbwcRIx7YVMn3Kb27dR8RmBD0Ev0ExhoQJWBW7Wfv1WtlW+lqhV7lRqEcNj+ZNkw41O0N/ZHNLYz/iSotEmsfEnUstzTuJo5Vk/obPV3KCqxq/wje7BBG7xKhmajytDGNifdxIAB2N06B8vYsmGFkTx00fgeL4AjxunppK4YmHFPHIDYSKR8VqgNgGpFh0Fw/bVh+O0+mHhch2oEU5FkfEGcwDsAvfk6NJRXQ8pF21GwnPxv8J2j09qqiHCJzWqz1rte+kEHr/fvzDTOyOD9+l58ThtSG85gLXpAPda/KxGVUdbwRDrlkHgxyJ55FtvRFzgyOEbMqjXWsTL1s0NtH7ORirzov7vyE2dhf3jVVzvU1uN3ixeDr+c3NGOXGTKpba5bqaQc4PK4CmwuEfmgJEHjTYlZEPq3NrGy59EVTyeFjQj7qRm0YIG9JqebzbKgQsqXJDBNnmsfOPsevxpGQ3eWLX4zVr3PO0YfbWD1ZbRjm3E1mb5Ra9tRbHtDkkcDAsGq9PHGvBtsZPp/GfuSaXB6TkR/zRncaSQNYrfg5zm7EC3qt+YmamWalJryva7dFHNd1/eSy2X5yf8SzF5zrySjsukKDODK579QAp4S7RM2RH7KiacxQnTh1C/fbBKYgAz2GLMT93kX4FH3+lhJ X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR07MB4531.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(366004)(376002)(136003)(346002)(39860400002)(8676002)(186003)(956004)(16526019)(2616005)(36756003)(26005)(86362001)(478600001)(966005)(316002)(31696002)(2906002)(6512007)(53546011)(6506007)(6486002)(5660300002)(4326008)(31686004)(66946007)(8936002)(66556008)(66476007)(83380400001)(52116002)(54906003)(44832011)(6916009)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?RlBMclFEVjlQdGxQMHRwbFdDdU8rOFp0VkNUckVXUG5CN0VaL0RkOXdoQjc4?= =?utf-8?B?YzE1SzhXeGMrSUlrTm4vdk9FemVqZkVZZWxtalY0Q3ZWeVRyK2QwckVsd0Jp?= =?utf-8?B?cUpkZXBVT20xU1NnTU1DUVkyTEFSQnJRblR3NzdBM0hoZWYzbWxXRUhrZ2Ns?= =?utf-8?B?UmIzdzJQcE04RmZNcmN3RHRndU9HM3JYdXppRDdMY050bW5YVDM0anhoaHMy?= =?utf-8?B?ME9sNGwwSXR1dTYzeDlOc3U4SWNVMEhmdXVvcW5aaEpIWE52RklldGNXMmw0?= =?utf-8?B?dmUvM0ZPTituWHJ0Q0RaWkdMdVVFa0ovSDdWc1VITXZaNFFIdWNSR0Zhd0xk?= =?utf-8?B?SmRYTXovZWhyZ1ZDZ3NaU1JqSWxsQjBtUk52TEFZNDR6ZmQrYXRrNGVNanlH?= =?utf-8?B?KzMrNHF6TWtZZ21XRmdwekZ1L1NmQmpSeW5nM3NyMmlZbUlzckhkdnMvUmxv?= =?utf-8?B?QjB2bGwxUlhjeVJoVGpXcWJ0SHhMclNjVWt2Sm1ibzdIN0QvYnZnQ3R4S09s?= =?utf-8?B?SjlxckVDWFhnU1hKRWJaeVNFVFBydmVrNE4vRHplU1FVUEhnUmxIWHpqUStw?= =?utf-8?B?VXh3RlY0bDBtK2dMUVo1SzFaUWFqVUlRRmdpbXpCM2xXQmlvVXJrNFluM3Rn?= =?utf-8?B?N1FwcFZWOEJxcXJ1eGVBVGpDS211Z2tSNWx4cndQVzk3c0dpczBYQ250SHh0?= =?utf-8?B?ZndpSm82ck5URUlhdW92dUVRNUJOYjZaa3VJUm1LRzJhalZpL0RTNlRYTEpx?= =?utf-8?B?UDR6TEtOeVRqVkpteFBHL1dOeWZzd2VNUDFONWFGT281aUdlalhTMlE0cUVm?= =?utf-8?B?TmNHTEYrckFnejJSUVpNWHVyejFDVkZDcW5oQlBsQ1liR3VUVDNZYm9RZVZU?= =?utf-8?B?ekllQjg1N3dUNG5KZjEvS0tQendXT0pkNU1hK0lNVkw4VkV2VlNGTTdQbjI5?= =?utf-8?B?bjU4b0pLVkNTdHM2NVNDN0duck01WlVYTkljSXg4blkySVZsMGp2MEVYMkZX?= =?utf-8?B?UDZZTlBsUzBBZTMrM0FIcnBiaElyMks5SUxkcWZ6SCthWVE3TU5OalJSV3Jy?= =?utf-8?B?ZjlXS3NpNTJoK2xSTm5OMzgzcHdycmhNMW1vK1Q1OWd1T204cjhIakplL0Zq?= =?utf-8?B?aTFiTHZuUml3T1BvS1VlUVI5ZHVFVkYwSGl6RHdhN3NJdXFtYmZIRE1zQnc1?= =?utf-8?B?WkZKR1hHS1h1VHVOemZXWlNxRHhJMnRSeWxsYWdtbmlIUjhWNFl3U3FHRWVQ?= =?utf-8?B?ZWFPNElJM3dqMlZNRWNxR2dGRWtsUWxITHY3dU81czNLTVM0VTVNT2VFUlhQ?= =?utf-8?B?S0o5a09pSW45bTl5blNQMDY2TTdoR1QwRGJzQ3AvWVNveWNXU080ejZxZDBX?= =?utf-8?B?US91Q0xrNG5jVEU2VmlvTXlBdjB6cmk0bkthY05vb2l0cjI2bGYrNVpkaUt5?= =?utf-8?Q?590zhqdh?= X-OriginatorOrg: nokia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5e9e0bea-5f49-49ae-226d-08d8c35f6bab X-MS-Exchange-CrossTenant-AuthSource: AM0PR07MB4531.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jan 2021 07:36:27.0101 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d471751-9675-428d-917b-70f44f9630b0 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: e+pf8FIP7xQYOacD4V9Ah1U2Psr0uwf4Qxw2tOkvzi4SAYsTG+jRxWaoFZvcQ/+UuBjz8PGG4r1b/2NuUfEEupo/VviN2QNMACnb5HdWNek= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR07MB7393 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210128_023630_198776_A9CBFC43 X-CRM114-Status: GOOD ( 21.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Zijlstra , Ingo Molnar , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Russell King Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi! On 27/01/2021 23:21, Will Deacon wrote: > On Wed, Jan 27, 2021 at 09:01:08PM +0100, Alexander A Sverdlin wrote: >> From: Alexander Sverdlin >> >> Ensure writes are pushed out of core write buffer to prevent waiting code >> on another cores from spinning longer than necessary. >> >> 6 threads running tight spinlock loop competing for the same lock >> on 6 cores on MIPS/Octeon do 1000000 iterations... >> >> before the patch in: 4.3 sec >> after the patch in: 1.2 sec > If you only have 6 cores, I'm not sure qspinlock makes any sense... That's my impression as well and I even proposed to revert qspinlocks for MIPS: https://lore.kernel.org/linux-mips/20170610002644.8434-1-paul.burton@imgtec.com/T/#ma1506c80472129b2ac41554cc2d863c9a24518c0 >> Same 6-core Octeon machine: >> sysbench --test=mutex --num-threads=64 --memory-scope=local run >> >> w/o patch: 1.53s >> with patch: 1.28s >> >> This will also allow to remove the smp_wmb() in >> arch/arm/include/asm/mcs_spinlock.h (was it actually addressing the same >> issue?). >> >> Finally our internal quite diverse test suite of different IPC/network >> aspects didn't detect any regressions on ARM/ARM64/x86_64. >> >> Signed-off-by: Alexander Sverdlin >> --- >> kernel/locking/mcs_spinlock.h | 5 +++++ >> kernel/locking/qspinlock.c | 6 ++++++ >> 2 files changed, 11 insertions(+) >> >> diff --git a/kernel/locking/mcs_spinlock.h b/kernel/locking/mcs_spinlock.h >> index 5e10153..10e497a 100644 >> --- a/kernel/locking/mcs_spinlock.h >> +++ b/kernel/locking/mcs_spinlock.h >> @@ -89,6 +89,11 @@ void mcs_spin_lock(struct mcs_spinlock **lock, struct mcs_spinlock *node) >> return; >> } >> WRITE_ONCE(prev->next, node); >> + /* >> + * This is necessary to make sure that the corresponding "while" in the >> + * mcs_spin_unlock() doesn't loop forever >> + */ >> + smp_wmb(); > If it loops forever, that's broken hardware design; store buffers need to > drain. I don't think we should add unconditional barriers to bodge this. The comment is a bit exaggerating the situation, but it's undeterministic and you see the measurements above. Something is wrong in the qspinlocks code, please consider this patch "RFC", but something has to be done here. -- Best regards, Alexander Sverdlin. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel