From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E15A7C433DB for ; Mon, 28 Dec 2020 23:09:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 92F832222A for ; Mon, 28 Dec 2020 23:09:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730960AbgL1Wz6 (ORCPT ); Mon, 28 Dec 2020 17:55:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729415AbgL1UZl (ORCPT ); Mon, 28 Dec 2020 15:25:41 -0500 Received: from pandora.armlinux.org.uk (pandora.armlinux.org.uk [IPv6:2001:4d48:ad52:32c8:5054:ff:fe00:142]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B6AFC061796; Mon, 28 Dec 2020 12:25:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Sender:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=aXiztOLVDI1xQ0cgx87QbDrwMbVe0SKDRuOye74fEvg=; b=qu34zBWTNFCGapiNvzkB0lQRr BKwxHW3PEiplLNcr8ZBFrF1PnKkCoS2SrTM7Mas5assUmmgvlu0GDn7DXicumZ8o8HlCBeyew5nPZ cIiXyk9NaJUBGnGpWwZpnurevAG9CwY0q0D77xGnfpsIoLRMVTyEoXY3FZjMQgsrKEzKNzPeqigVE VJxcYb21gQTcj4pZeA3SHCfe0hGy2G/U79AbGUEAjDTY2jn5C8HxuHvp0cV3GGnKbDv3gUK8BCdIj 2mfWeEfNyLbgrvPcoPBmc/0WTtYMuScVaC0I7QllSrp6Za0U4vQfJOG+0DH5PNV+TduGJ0OdLuBwg nn00bsjHA==; Received: from shell.armlinux.org.uk ([fd8f:7570:feb6:1:5054:ff:fe00:4ec]:44614) by pandora.armlinux.org.uk with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ktz4Y-0004g3-6K; Mon, 28 Dec 2020 20:24:54 +0000 Received: from linux by shell.armlinux.org.uk with local (Exim 4.92) (envelope-from ) id 1ktz4V-0000Yw-DR; Mon, 28 Dec 2020 20:24:51 +0000 Date: Mon, 28 Dec 2020 20:24:51 +0000 From: Russell King - ARM Linux admin To: Andy Lutomirski Cc: Jann Horn , Will Deacon , Mathieu Desnoyers , x86 , linux-kernel , Nicholas Piggin , Arnd Bergmann , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , linuxppc-dev , Catalin Marinas , linux-arm-kernel , stable Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() Message-ID: <20201228202451.GJ1551@shell.armlinux.org.uk> References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> <20201228102537.GG1551@shell.armlinux.org.uk> <20201228190852.GI1551@shell.armlinux.org.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: Russell King - ARM Linux admin Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Dec 28, 2020 at 11:44:33AM -0800, Andy Lutomirski wrote: > On Mon, Dec 28, 2020 at 11:09 AM Russell King - ARM Linux admin > wrote: > > > > On Mon, Dec 28, 2020 at 07:29:34PM +0100, Jann Horn wrote: > > > After chatting with rmk about this (but without claiming that any of > > > this is his opinion), based on the manpage, I think membarrier() > > > currently doesn't really claim to be synchronizing caches? It just > > > serializes cores. So arguably if userspace wants to use membarrier() > > > to synchronize code changes, userspace should first do the code > > > change, then flush icache as appropriate for the architecture, and > > > then do the membarrier() to ensure that the old code is unused? > > > > > > For 32-bit arm, rmk pointed out that that would be the cacheflush() > > > syscall. That might cause you to end up with two IPIs instead of one > > > in total, but we probably don't care _that_ much about extra IPIs on > > > 32-bit arm? > > > > > > For arm64, I believe userspace can flush icache across the entire > > > system with some instructions from userspace - "DC CVAU" followed by > > > "DSB ISH", or something like that, I think? (See e.g. > > > compat_arm_syscall(), the arm64 compat code that implements the 32-bit > > > arm cacheflush() syscall.) > > > > Note that the ARM cacheflush syscall calls flush_icache_user_range() > > over the range of addresses that userspace has passed - it's intention > > since day one is to support cases where userspace wants to change > > executable code. > > > > It will issue the appropriate write-backs to the data cache (DCCMVAU), > > the invalidates to the instruction cache (ICIMVAU), invalidate the > > branch target buffer (BPIALLIS or BPIALL as appropriate), and issue > > the appropriate barriers (DSB ISHST, ISB). > > > > Note that neither flush_icache_user_range() nor flush_icache_range() > > result in IPIs; cache operations are broadcast across all CPUs (which > > is one of the minimums we require for SMP systems.) > > > > Now, that all said, I think the question that has to be asked is... > > > > What is the basic purpose of membarrier? > > > > Is the purpose of it to provide memory barriers, or is it to provide > > memory coherence? > > > > If it's the former and not the latter, then cache flushes are out of > > scope, and expecting memory written to be visible to the instruction > > stream is totally out of scope of the membarrier interface, whether > > or not the writes happen on the same or a different CPU to the one > > executing the rewritten code. > > > > The documentation in the kernel does not seem to describe what it's > > supposed to be doing - the only thing I could find is this: > > Documentation/features/sched/membarrier-sync-core/arch-support.txt > > which describes it as "arch supports core serializing membarrier" > > whatever that means. > > > > Seems to be the standard and usual case of utterly poor to non-existent > > documentation within the kernel tree, or even a pointer to where any > > useful documentation can be found. > > > > Reading the membarrier(2) man page, I find nothing in there that talks > > about any kind of cache coherency for self-modifying code - it only > > seems to be about _barriers_ and nothing more, and barriers alone do > > precisely nothing to save you from non-coherent Harvard caches. > > > > So, either Andy has a misunderstanding, or the man page is wrong, or > > my rudimentary understanding of what membarrier is supposed to be > > doing is wrong... > > Look at the latest man page: > > https://man7.org/linux/man-pages/man2/membarrier.2.html > > for MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE. The result may not be > all that enlightening. MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE (since Linux 4.16) In addition to providing the memory ordering guarantees de■ scribed in MEMBARRIER_CMD_PRIVATE_EXPEDITED, upon return from system call the calling thread has a guarantee that all its run■ ning thread siblings have executed a core serializing instruc■ tion. This guarantee is provided only for threads in the same process as the calling thread. The "expedited" commands complete faster than the non-expedited ones, they never block, but have the downside of causing extra overhead. A process must register its intent to use the private expedited sync core command prior to using it. This just says that the siblings have executed a serialising instruction, in other words a barrier. It makes no claims concerning cache coherency - and without some form of cache maintenance, there can be no expectation that the I and D streams to be coherent with each other. This description is also weird in another respect. "guarantee that all its running thread siblings have executed a core serializing instruction" ... "The expedited commands ... never block". So, the core executing this call is not allowed to block, but the other part indicates that the other CPUs _have_ executed a serialising instruction before this call returns... one wonders how that happens without blocking. Maybe the CPU spins waiting for completion instead? -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 128AEC433DB for ; Mon, 28 Dec 2020 20:27:00 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 43F56221F8 for ; Mon, 28 Dec 2020 20:26:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 43F56221F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=armlinux.org.uk Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4D4Tc05WjszDqFH for ; Tue, 29 Dec 2020 07:26:56 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=armlinux.org.uk (client-ip=2001:4d48:ad52:32c8:5054:ff:fe00:142; helo=pandora.armlinux.org.uk; envelope-from=linux+linuxppc-dev=lists.ozlabs.org@armlinux.org.uk; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=armlinux.org.uk Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=armlinux.org.uk header.i=@armlinux.org.uk header.a=rsa-sha256 header.s=pandora-2019 header.b=qu34zBWT; dkim-atps=neutral Received: from pandora.armlinux.org.uk (pandora.armlinux.org.uk [IPv6:2001:4d48:ad52:32c8:5054:ff:fe00:142]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4D4TYz2bbVzDq9j for ; Tue, 29 Dec 2020 07:25:05 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Sender:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=aXiztOLVDI1xQ0cgx87QbDrwMbVe0SKDRuOye74fEvg=; b=qu34zBWTNFCGapiNvzkB0lQRr BKwxHW3PEiplLNcr8ZBFrF1PnKkCoS2SrTM7Mas5assUmmgvlu0GDn7DXicumZ8o8HlCBeyew5nPZ cIiXyk9NaJUBGnGpWwZpnurevAG9CwY0q0D77xGnfpsIoLRMVTyEoXY3FZjMQgsrKEzKNzPeqigVE VJxcYb21gQTcj4pZeA3SHCfe0hGy2G/U79AbGUEAjDTY2jn5C8HxuHvp0cV3GGnKbDv3gUK8BCdIj 2mfWeEfNyLbgrvPcoPBmc/0WTtYMuScVaC0I7QllSrp6Za0U4vQfJOG+0DH5PNV+TduGJ0OdLuBwg nn00bsjHA==; Received: from shell.armlinux.org.uk ([fd8f:7570:feb6:1:5054:ff:fe00:4ec]:44614) by pandora.armlinux.org.uk with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ktz4Y-0004g3-6K; Mon, 28 Dec 2020 20:24:54 +0000 Received: from linux by shell.armlinux.org.uk with local (Exim 4.92) (envelope-from ) id 1ktz4V-0000Yw-DR; Mon, 28 Dec 2020 20:24:51 +0000 Date: Mon, 28 Dec 2020 20:24:51 +0000 From: Russell King - ARM Linux admin To: Andy Lutomirski Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() Message-ID: <20201228202451.GJ1551@shell.armlinux.org.uk> References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> <20201228102537.GG1551@shell.armlinux.org.uk> <20201228190852.GI1551@shell.armlinux.org.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linuxppc-dev , Catalin Marinas , Arnd Bergmann , Jann Horn , x86 , linux-kernel , Nicholas Piggin , Mathieu Desnoyers , stable , Paul Mackerras , Will Deacon , linux-arm-kernel Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Mon, Dec 28, 2020 at 11:44:33AM -0800, Andy Lutomirski wrote: > On Mon, Dec 28, 2020 at 11:09 AM Russell King - ARM Linux admin > wrote: > > > > On Mon, Dec 28, 2020 at 07:29:34PM +0100, Jann Horn wrote: > > > After chatting with rmk about this (but without claiming that any of > > > this is his opinion), based on the manpage, I think membarrier() > > > currently doesn't really claim to be synchronizing caches? It just > > > serializes cores. So arguably if userspace wants to use membarrier() > > > to synchronize code changes, userspace should first do the code > > > change, then flush icache as appropriate for the architecture, and > > > then do the membarrier() to ensure that the old code is unused? > > > > > > For 32-bit arm, rmk pointed out that that would be the cacheflush() > > > syscall. That might cause you to end up with two IPIs instead of one > > > in total, but we probably don't care _that_ much about extra IPIs on > > > 32-bit arm? > > > > > > For arm64, I believe userspace can flush icache across the entire > > > system with some instructions from userspace - "DC CVAU" followed by > > > "DSB ISH", or something like that, I think? (See e.g. > > > compat_arm_syscall(), the arm64 compat code that implements the 32-bit > > > arm cacheflush() syscall.) > > > > Note that the ARM cacheflush syscall calls flush_icache_user_range() > > over the range of addresses that userspace has passed - it's intention > > since day one is to support cases where userspace wants to change > > executable code. > > > > It will issue the appropriate write-backs to the data cache (DCCMVAU), > > the invalidates to the instruction cache (ICIMVAU), invalidate the > > branch target buffer (BPIALLIS or BPIALL as appropriate), and issue > > the appropriate barriers (DSB ISHST, ISB). > > > > Note that neither flush_icache_user_range() nor flush_icache_range() > > result in IPIs; cache operations are broadcast across all CPUs (which > > is one of the minimums we require for SMP systems.) > > > > Now, that all said, I think the question that has to be asked is... > > > > What is the basic purpose of membarrier? > > > > Is the purpose of it to provide memory barriers, or is it to provide > > memory coherence? > > > > If it's the former and not the latter, then cache flushes are out of > > scope, and expecting memory written to be visible to the instruction > > stream is totally out of scope of the membarrier interface, whether > > or not the writes happen on the same or a different CPU to the one > > executing the rewritten code. > > > > The documentation in the kernel does not seem to describe what it's > > supposed to be doing - the only thing I could find is this: > > Documentation/features/sched/membarrier-sync-core/arch-support.txt > > which describes it as "arch supports core serializing membarrier" > > whatever that means. > > > > Seems to be the standard and usual case of utterly poor to non-existent > > documentation within the kernel tree, or even a pointer to where any > > useful documentation can be found. > > > > Reading the membarrier(2) man page, I find nothing in there that talks > > about any kind of cache coherency for self-modifying code - it only > > seems to be about _barriers_ and nothing more, and barriers alone do > > precisely nothing to save you from non-coherent Harvard caches. > > > > So, either Andy has a misunderstanding, or the man page is wrong, or > > my rudimentary understanding of what membarrier is supposed to be > > doing is wrong... > > Look at the latest man page: > > https://man7.org/linux/man-pages/man2/membarrier.2.html > > for MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE. The result may not be > all that enlightening. MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE (since Linux 4.16) In addition to providing the memory ordering guarantees de■ scribed in MEMBARRIER_CMD_PRIVATE_EXPEDITED, upon return from system call the calling thread has a guarantee that all its run■ ning thread siblings have executed a core serializing instruc■ tion. This guarantee is provided only for threads in the same process as the calling thread. The "expedited" commands complete faster than the non-expedited ones, they never block, but have the downside of causing extra overhead. A process must register its intent to use the private expedited sync core command prior to using it. This just says that the siblings have executed a serialising instruction, in other words a barrier. It makes no claims concerning cache coherency - and without some form of cache maintenance, there can be no expectation that the I and D streams to be coherent with each other. This description is also weird in another respect. "guarantee that all its running thread siblings have executed a core serializing instruction" ... "The expedited commands ... never block". So, the core executing this call is not allowed to block, but the other part indicates that the other CPUs _have_ executed a serialising instruction before this call returns... one wonders how that happens without blocking. Maybe the CPU spins waiting for completion instead? -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E16ABC433DB for ; Mon, 28 Dec 2020 20:27:05 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 99A4E221F8 for ; Mon, 28 Dec 2020 20:27:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 99A4E221F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=armlinux.org.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tSwg9p//JCEP/fB/daE4H9A8xri/Oy31j297yR1qz4o=; b=gDbSIvNtG2oEOzM1GEjRYMCjP GUfxIBT3M+5rJtrXyE9D9bUZWgG7l8PwNrxFVmRky5RyWfOt/2yqRxmblJAirLIwNXhSNPJavAmSg wrN+vQYSZlwFK/DCPfHGvH/0JjY0Guxd7DpUD0I2LOQPgwcVyOBefKBqisE7GUDFHiCdjL1p/BKEB DqCkkKSFSXOTY/gX/VXcgvT0SeeXB/E587quTIuRE0cAqAxmsnktIaAoQIVn6Bb3Jy/1Uqr0bPw98 ayaP0Iwy+bvAltGh92m7M2cuTt0Cf9Hu/g2sgugjqXEetB+0FqW9cN48KGToTkXIIL/ykpN/1Q46v vi+gpSwWQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ktz4v-0007ah-GJ; Mon, 28 Dec 2020 20:25:17 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1ktz4s-0007a7-FK for linux-arm-kernel@lists.infradead.org; Mon, 28 Dec 2020 20:25:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Sender:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=aXiztOLVDI1xQ0cgx87QbDrwMbVe0SKDRuOye74fEvg=; b=qu34zBWTNFCGapiNvzkB0lQRr BKwxHW3PEiplLNcr8ZBFrF1PnKkCoS2SrTM7Mas5assUmmgvlu0GDn7DXicumZ8o8HlCBeyew5nPZ cIiXyk9NaJUBGnGpWwZpnurevAG9CwY0q0D77xGnfpsIoLRMVTyEoXY3FZjMQgsrKEzKNzPeqigVE VJxcYb21gQTcj4pZeA3SHCfe0hGy2G/U79AbGUEAjDTY2jn5C8HxuHvp0cV3GGnKbDv3gUK8BCdIj 2mfWeEfNyLbgrvPcoPBmc/0WTtYMuScVaC0I7QllSrp6Za0U4vQfJOG+0DH5PNV+TduGJ0OdLuBwg nn00bsjHA==; Received: from shell.armlinux.org.uk ([fd8f:7570:feb6:1:5054:ff:fe00:4ec]:44614) by pandora.armlinux.org.uk with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ktz4Y-0004g3-6K; Mon, 28 Dec 2020 20:24:54 +0000 Received: from linux by shell.armlinux.org.uk with local (Exim 4.92) (envelope-from ) id 1ktz4V-0000Yw-DR; Mon, 28 Dec 2020 20:24:51 +0000 Date: Mon, 28 Dec 2020 20:24:51 +0000 From: Russell King - ARM Linux admin To: Andy Lutomirski Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() Message-ID: <20201228202451.GJ1551@shell.armlinux.org.uk> References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> <20201228102537.GG1551@shell.armlinux.org.uk> <20201228190852.GI1551@shell.armlinux.org.uk> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201228_152514_635780_6F1962D8 X-CRM114-Status: GOOD ( 38.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linuxppc-dev , Catalin Marinas , Arnd Bergmann , Jann Horn , Michael Ellerman , x86 , linux-kernel , Nicholas Piggin , Mathieu Desnoyers , stable , Benjamin Herrenschmidt , Paul Mackerras , Will Deacon , linux-arm-kernel Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gTW9uLCBEZWMgMjgsIDIwMjAgYXQgMTE6NDQ6MzNBTSAtMDgwMCwgQW5keSBMdXRvbWlyc2tp IHdyb3RlOgo+IE9uIE1vbiwgRGVjIDI4LCAyMDIwIGF0IDExOjA5IEFNIFJ1c3NlbGwgS2luZyAt IEFSTSBMaW51eCBhZG1pbgo+IDxsaW51eEBhcm1saW51eC5vcmcudWs+IHdyb3RlOgo+ID4KPiA+ IE9uIE1vbiwgRGVjIDI4LCAyMDIwIGF0IDA3OjI5OjM0UE0gKzAxMDAsIEphbm4gSG9ybiB3cm90 ZToKPiA+ID4gQWZ0ZXIgY2hhdHRpbmcgd2l0aCBybWsgYWJvdXQgdGhpcyAoYnV0IHdpdGhvdXQg Y2xhaW1pbmcgdGhhdCBhbnkgb2YKPiA+ID4gdGhpcyBpcyBoaXMgb3BpbmlvbiksIGJhc2VkIG9u IHRoZSBtYW5wYWdlLCBJIHRoaW5rIG1lbWJhcnJpZXIoKQo+ID4gPiBjdXJyZW50bHkgZG9lc24n dCByZWFsbHkgY2xhaW0gdG8gYmUgc3luY2hyb25pemluZyBjYWNoZXM/IEl0IGp1c3QKPiA+ID4g c2VyaWFsaXplcyBjb3Jlcy4gU28gYXJndWFibHkgaWYgdXNlcnNwYWNlIHdhbnRzIHRvIHVzZSBt ZW1iYXJyaWVyKCkKPiA+ID4gdG8gc3luY2hyb25pemUgY29kZSBjaGFuZ2VzLCB1c2Vyc3BhY2Ug c2hvdWxkIGZpcnN0IGRvIHRoZSBjb2RlCj4gPiA+IGNoYW5nZSwgdGhlbiBmbHVzaCBpY2FjaGUg YXMgYXBwcm9wcmlhdGUgZm9yIHRoZSBhcmNoaXRlY3R1cmUsIGFuZAo+ID4gPiB0aGVuIGRvIHRo ZSBtZW1iYXJyaWVyKCkgdG8gZW5zdXJlIHRoYXQgdGhlIG9sZCBjb2RlIGlzIHVudXNlZD8KPiA+ ID4KPiA+ID4gRm9yIDMyLWJpdCBhcm0sIHJtayBwb2ludGVkIG91dCB0aGF0IHRoYXQgd291bGQg YmUgdGhlIGNhY2hlZmx1c2goKQo+ID4gPiBzeXNjYWxsLiBUaGF0IG1pZ2h0IGNhdXNlIHlvdSB0 byBlbmQgdXAgd2l0aCB0d28gSVBJcyBpbnN0ZWFkIG9mIG9uZQo+ID4gPiBpbiB0b3RhbCwgYnV0 IHdlIHByb2JhYmx5IGRvbid0IGNhcmUgX3RoYXRfIG11Y2ggYWJvdXQgZXh0cmEgSVBJcyBvbgo+ ID4gPiAzMi1iaXQgYXJtPwo+ID4gPgo+ID4gPiBGb3IgYXJtNjQsIEkgYmVsaWV2ZSB1c2Vyc3Bh Y2UgY2FuIGZsdXNoIGljYWNoZSBhY3Jvc3MgdGhlIGVudGlyZQo+ID4gPiBzeXN0ZW0gd2l0aCBz b21lIGluc3RydWN0aW9ucyBmcm9tIHVzZXJzcGFjZSAtICJEQyBDVkFVIiBmb2xsb3dlZCBieQo+ ID4gPiAiRFNCIElTSCIsIG9yIHNvbWV0aGluZyBsaWtlIHRoYXQsIEkgdGhpbms/IChTZWUgZS5n Lgo+ID4gPiBjb21wYXRfYXJtX3N5c2NhbGwoKSwgdGhlIGFybTY0IGNvbXBhdCBjb2RlIHRoYXQg aW1wbGVtZW50cyB0aGUgMzItYml0Cj4gPiA+IGFybSBjYWNoZWZsdXNoKCkgc3lzY2FsbC4pCj4g Pgo+ID4gTm90ZSB0aGF0IHRoZSBBUk0gY2FjaGVmbHVzaCBzeXNjYWxsIGNhbGxzIGZsdXNoX2lj YWNoZV91c2VyX3JhbmdlKCkKPiA+IG92ZXIgdGhlIHJhbmdlIG9mIGFkZHJlc3NlcyB0aGF0IHVz ZXJzcGFjZSBoYXMgcGFzc2VkIC0gaXQncyBpbnRlbnRpb24KPiA+IHNpbmNlIGRheSBvbmUgaXMg dG8gc3VwcG9ydCBjYXNlcyB3aGVyZSB1c2Vyc3BhY2Ugd2FudHMgdG8gY2hhbmdlCj4gPiBleGVj dXRhYmxlIGNvZGUuCj4gPgo+ID4gSXQgd2lsbCBpc3N1ZSB0aGUgYXBwcm9wcmlhdGUgd3JpdGUt YmFja3MgdG8gdGhlIGRhdGEgY2FjaGUgKERDQ01WQVUpLAo+ID4gdGhlIGludmFsaWRhdGVzIHRv IHRoZSBpbnN0cnVjdGlvbiBjYWNoZSAoSUNJTVZBVSksIGludmFsaWRhdGUgdGhlCj4gPiBicmFu Y2ggdGFyZ2V0IGJ1ZmZlciAoQlBJQUxMSVMgb3IgQlBJQUxMIGFzIGFwcHJvcHJpYXRlKSwgYW5k IGlzc3VlCj4gPiB0aGUgYXBwcm9wcmlhdGUgYmFycmllcnMgKERTQiBJU0hTVCwgSVNCKS4KPiA+ Cj4gPiBOb3RlIHRoYXQgbmVpdGhlciBmbHVzaF9pY2FjaGVfdXNlcl9yYW5nZSgpIG5vciBmbHVz aF9pY2FjaGVfcmFuZ2UoKQo+ID4gcmVzdWx0IGluIElQSXM7IGNhY2hlIG9wZXJhdGlvbnMgYXJl IGJyb2FkY2FzdCBhY3Jvc3MgYWxsIENQVXMgKHdoaWNoCj4gPiBpcyBvbmUgb2YgdGhlIG1pbmlt dW1zIHdlIHJlcXVpcmUgZm9yIFNNUCBzeXN0ZW1zLikKPiA+Cj4gPiBOb3csIHRoYXQgYWxsIHNh aWQsIEkgdGhpbmsgdGhlIHF1ZXN0aW9uIHRoYXQgaGFzIHRvIGJlIGFza2VkIGlzLi4uCj4gPgo+ ID4gICAgICAgICBXaGF0IGlzIHRoZSBiYXNpYyBwdXJwb3NlIG9mIG1lbWJhcnJpZXI/Cj4gPgo+ ID4gSXMgdGhlIHB1cnBvc2Ugb2YgaXQgdG8gcHJvdmlkZSBtZW1vcnkgYmFycmllcnMsIG9yIGlz IGl0IHRvIHByb3ZpZGUKPiA+IG1lbW9yeSBjb2hlcmVuY2U/Cj4gPgo+ID4gSWYgaXQncyB0aGUg Zm9ybWVyIGFuZCBub3QgdGhlIGxhdHRlciwgdGhlbiBjYWNoZSBmbHVzaGVzIGFyZSBvdXQgb2YK PiA+IHNjb3BlLCBhbmQgZXhwZWN0aW5nIG1lbW9yeSB3cml0dGVuIHRvIGJlIHZpc2libGUgdG8g dGhlIGluc3RydWN0aW9uCj4gPiBzdHJlYW0gaXMgdG90YWxseSBvdXQgb2Ygc2NvcGUgb2YgdGhl IG1lbWJhcnJpZXIgaW50ZXJmYWNlLCB3aGV0aGVyCj4gPiBvciBub3QgdGhlIHdyaXRlcyBoYXBw ZW4gb24gdGhlIHNhbWUgb3IgYSBkaWZmZXJlbnQgQ1BVIHRvIHRoZSBvbmUKPiA+IGV4ZWN1dGlu ZyB0aGUgcmV3cml0dGVuIGNvZGUuCj4gPgo+ID4gVGhlIGRvY3VtZW50YXRpb24gaW4gdGhlIGtl cm5lbCBkb2VzIG5vdCBzZWVtIHRvIGRlc2NyaWJlIHdoYXQgaXQncwo+ID4gc3VwcG9zZWQgdG8g YmUgZG9pbmcgLSB0aGUgb25seSB0aGluZyBJIGNvdWxkIGZpbmQgaXMgdGhpczoKPiA+IERvY3Vt ZW50YXRpb24vZmVhdHVyZXMvc2NoZWQvbWVtYmFycmllci1zeW5jLWNvcmUvYXJjaC1zdXBwb3J0 LnR4dAo+ID4gd2hpY2ggZGVzY3JpYmVzIGl0IGFzICJhcmNoIHN1cHBvcnRzIGNvcmUgc2VyaWFs aXppbmcgbWVtYmFycmllciIKPiA+IHdoYXRldmVyIHRoYXQgbWVhbnMuCj4gPgo+ID4gU2VlbXMg dG8gYmUgdGhlIHN0YW5kYXJkIGFuZCB1c3VhbCBjYXNlIG9mIHV0dGVybHkgcG9vciB0byBub24t ZXhpc3RlbnQKPiA+IGRvY3VtZW50YXRpb24gd2l0aGluIHRoZSBrZXJuZWwgdHJlZSwgb3IgZXZl biBhIHBvaW50ZXIgdG8gd2hlcmUgYW55Cj4gPiB1c2VmdWwgZG9jdW1lbnRhdGlvbiBjYW4gYmUg Zm91bmQuCj4gPgo+ID4gUmVhZGluZyB0aGUgbWVtYmFycmllcigyKSBtYW4gcGFnZSwgSSBmaW5k IG5vdGhpbmcgaW4gdGhlcmUgdGhhdCB0YWxrcwo+ID4gYWJvdXQgYW55IGtpbmQgb2YgY2FjaGUg Y29oZXJlbmN5IGZvciBzZWxmLW1vZGlmeWluZyBjb2RlIC0gaXQgb25seQo+ID4gc2VlbXMgdG8g YmUgYWJvdXQgX2JhcnJpZXJzXyBhbmQgbm90aGluZyBtb3JlLCBhbmQgYmFycmllcnMgYWxvbmUg ZG8KPiA+IHByZWNpc2VseSBub3RoaW5nIHRvIHNhdmUgeW91IGZyb20gbm9uLWNvaGVyZW50IEhh cnZhcmQgY2FjaGVzLgo+ID4KPiA+IFNvLCBlaXRoZXIgQW5keSBoYXMgYSBtaXN1bmRlcnN0YW5k aW5nLCBvciB0aGUgbWFuIHBhZ2UgaXMgd3JvbmcsIG9yCj4gPiBteSBydWRpbWVudGFyeSB1bmRl cnN0YW5kaW5nIG9mIHdoYXQgbWVtYmFycmllciBpcyBzdXBwb3NlZCB0byBiZQo+ID4gZG9pbmcg aXMgd3JvbmcuLi4KPiAKPiBMb29rIGF0IHRoZSBsYXRlc3QgbWFuIHBhZ2U6Cj4gCj4gaHR0cHM6 Ly9tYW43Lm9yZy9saW51eC9tYW4tcGFnZXMvbWFuMi9tZW1iYXJyaWVyLjIuaHRtbAo+IAo+IGZv ciBNRU1CQVJSSUVSX0NNRF9QUklWQVRFX0VYUEVESVRFRF9TWU5DX0NPUkUuICBUaGUgcmVzdWx0 IG1heSBub3QgYmUKPiBhbGwgdGhhdCBlbmxpZ2h0ZW5pbmcuCgogICAgICAgTUVNQkFSUklFUl9D TURfUFJJVkFURV9FWFBFRElURURfU1lOQ19DT1JFIChzaW5jZSBMaW51eCA0LjE2KQogICAgICAg ICAgICAgIEluICBhZGRpdGlvbiAgdG8gIHByb3ZpZGluZyAgdGhlICBtZW1vcnkgb3JkZXJpbmcg Z3VhcmFudGVlcyBkZeKWoAogICAgICAgICAgICAgIHNjcmliZWQgaW4gTUVNQkFSUklFUl9DTURf UFJJVkFURV9FWFBFRElURUQsICB1cG9uICByZXR1cm4gIGZyb20KICAgICAgICAgICAgICBzeXN0 ZW0gY2FsbCB0aGUgY2FsbGluZyB0aHJlYWQgaGFzIGEgZ3VhcmFudGVlIHRoYXQgYWxsIGl0cyBy dW7ilqAKICAgICAgICAgICAgICBuaW5nIHRocmVhZCBzaWJsaW5ncyBoYXZlIGV4ZWN1dGVkIGEg Y29yZSAgc2VyaWFsaXppbmcgIGluc3RydWPilqAKICAgICAgICAgICAgICB0aW9uLiAgIFRoaXMg IGd1YXJhbnRlZSBpcyBwcm92aWRlZCBvbmx5IGZvciB0aHJlYWRzIGluIHRoZSBzYW1lCiAgICAg ICAgICAgICAgcHJvY2VzcyBhcyB0aGUgY2FsbGluZyB0aHJlYWQuCgogICAgICAgICAgICAgIFRo ZSAiZXhwZWRpdGVkIiBjb21tYW5kcyBjb21wbGV0ZSBmYXN0ZXIgdGhhbiB0aGUgIG5vbi1leHBl ZGl0ZWQKICAgICAgICAgICAgICBvbmVzLCAgdGhleSAgbmV2ZXIgYmxvY2ssIGJ1dCBoYXZlIHRo ZSBkb3duc2lkZSBvZiBjYXVzaW5nIGV4dHJhCiAgICAgICAgICAgICAgb3ZlcmhlYWQuCgogICAg ICAgICAgICAgIEEgcHJvY2VzcyBtdXN0IHJlZ2lzdGVyIGl0cyBpbnRlbnQgdG8gdXNlIHRoZSBw cml2YXRlICBleHBlZGl0ZWQKICAgICAgICAgICAgICBzeW5jIGNvcmUgY29tbWFuZCBwcmlvciB0 byB1c2luZyBpdC4KClRoaXMganVzdCBzYXlzIHRoYXQgdGhlIHNpYmxpbmdzIGhhdmUgZXhlY3V0 ZWQgYSBzZXJpYWxpc2luZwppbnN0cnVjdGlvbiwgaW4gb3RoZXIgd29yZHMgYSBiYXJyaWVyLiBJ dCBtYWtlcyBubyBjbGFpbXMgY29uY2VybmluZwpjYWNoZSBjb2hlcmVuY3kgLSBhbmQgd2l0aG91 dCBzb21lIGZvcm0gb2YgY2FjaGUgbWFpbnRlbmFuY2UsIHRoZXJlCmNhbiBiZSBubyBleHBlY3Rh dGlvbiB0aGF0IHRoZSBJIGFuZCBEIHN0cmVhbXMgdG8gYmUgY29oZXJlbnQgd2l0aAplYWNoIG90 aGVyLgoKVGhpcyBkZXNjcmlwdGlvbiBpcyBhbHNvIHdlaXJkIGluIGFub3RoZXIgcmVzcGVjdC4g Imd1YXJhbnRlZSB0aGF0CmFsbCBpdHMgcnVubmluZyB0aHJlYWQgc2libGluZ3MgaGF2ZSBleGVj dXRlZCBhIGNvcmUgc2VyaWFsaXppbmcKaW5zdHJ1Y3Rpb24iIC4uLiAiVGhlIGV4cGVkaXRlZCBj b21tYW5kcyAuLi4gbmV2ZXIgYmxvY2siLgoKU28sIHRoZSBjb3JlIGV4ZWN1dGluZyB0aGlzIGNh bGwgaXMgbm90IGFsbG93ZWQgdG8gYmxvY2ssIGJ1dCB0aGUKb3RoZXIgcGFydCBpbmRpY2F0ZXMg dGhhdCB0aGUgb3RoZXIgQ1BVcyBfaGF2ZV8gZXhlY3V0ZWQgYSBzZXJpYWxpc2luZwppbnN0cnVj dGlvbiBiZWZvcmUgdGhpcyBjYWxsIHJldHVybnMuLi4gb25lIHdvbmRlcnMgaG93IHRoYXQgaGFw cGVucwp3aXRob3V0IGJsb2NraW5nLiBNYXliZSB0aGUgQ1BVIHNwaW5zIHdhaXRpbmcgZm9yIGNv bXBsZXRpb24gaW5zdGVhZD8KCi0tIApSTUsncyBQYXRjaCBzeXN0ZW06IGh0dHBzOi8vd3d3LmFy bWxpbnV4Lm9yZy51ay9kZXZlbG9wZXIvcGF0Y2hlcy8KRlRUUCBpcyBoZXJlISA0ME1icHMgZG93 biAxME1icHMgdXAuIERlY2VudCBjb25uZWN0aXZpdHkgYXQgbGFzdCEKCl9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGlu ZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMu aW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK