From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62CF5C433E6 for ; Mon, 28 Dec 2020 23:03:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 36541223C8 for ; Mon, 28 Dec 2020 23:03:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731508AbgL1W4K (ORCPT ); Mon, 28 Dec 2020 17:56:10 -0500 Received: from mail.efficios.com ([167.114.26.124]:40400 "EHLO mail.efficios.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729494AbgL1VKV (ORCPT ); Mon, 28 Dec 2020 16:10:21 -0500 Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 063922761C7; Mon, 28 Dec 2020 16:09:40 -0500 (EST) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Xl2HNlqr6BSs; Mon, 28 Dec 2020 16:09:39 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 8512E2761C5; Mon, 28 Dec 2020 16:09:39 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 8512E2761C5 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1609189779; bh=A+A/Blx9QD1hoONN5Jxqqa59/BnSFAuRhzOnDZD8md8=; h=Date:From:To:Message-ID:MIME-Version; b=p/szlAxKXnb8sMCXgDpGdE6is/o1rjODbnzdRovA9owiEjuUPlM3+dte9oFxY/+Hd pvcRHGvBLx8S5o5ANGCMKBwMqzBmFMfuacgLtZuDT1vBtUVUSIdSFMD3lwBW0Mm8Sq XkulYMWharVGPEE76YIKlwesiZMP7dKhpXj/gToE1q4CiEAv+QDgP6f1MTue34+EDL yC7abPFDn6sQU4EVRsjnLUgN48sbY+nRlQq9VYRasErERk2As+18wk5Q0JRbIc/lci ltQp4+4byni+b6Qb7IBjB/By21MOffNXi7Ev/3OOnq+SzZHcRZHXttXzb3ZGlGI4S3 +WkZTiCC1H5Hw== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id BWu1pC_47JoY; Mon, 28 Dec 2020 16:09:39 -0500 (EST) Received: from mail03.efficios.com (mail03.efficios.com [167.114.26.124]) by mail.efficios.com (Postfix) with ESMTP id 725FE27640E; Mon, 28 Dec 2020 16:09:39 -0500 (EST) Date: Mon, 28 Dec 2020 16:09:39 -0500 (EST) From: Mathieu Desnoyers To: Andy Lutomirski , paulmck , Peter Zijlstra Cc: "Russell King, ARM Linux" , x86 , linux-kernel , Nicholas Piggin , Arnd Bergmann , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , linuxppc-dev , Catalin Marinas , Will Deacon , linux-arm-kernel , stable Message-ID: <1670059472.3671.1609189779376.JavaMail.zimbra@efficios.com> In-Reply-To: References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [167.114.26.124] X-Mailer: Zimbra 8.8.15_GA_3991 (ZimbraWebClient - FF84 (Linux)/8.8.15_GA_3980) Thread-Topic: membarrier: Rewrite sync_core_before_usermode() Thread-Index: UfUbgHPNYlsGbr5X+VxS/QBaGieq4w== Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org ----- On Dec 27, 2020, at 4:36 PM, Andy Lutomirski luto@kernel.org wrote: [...] >> You seem to have noticed odd cases on arm64 where this guarantee does not >> match reality. Where exactly can we find this in the code, and which part >> of the architecture manual can you point us to which supports your concern ? >> >> Based on the notes I have, use of `eret` on aarch64 guarantees a context >> synchronizing >> instruction when returning to user-space. > > Based on my reading of the manual, ERET on ARM doesn't synchronize > anything at all. I can't find any evidence that it synchronizes data > or instructions, and I've seen reports that the CPU will happily > speculate right past it. Reading [1] there appears to be 3 kind of context synchronization events: - Taking an exception, - Returning from an exception, - ISB. This other source [2] adds (search for Context synchronization operation): - Exit from Debug state - Executing a DCPS instruction - Executing a DRPS instruction "ERET" falls into the second kind of events, and AFAIU should be context synchronizing. That was confirmed to me by Will Deacon when membarrier sync-core was implemented for aarch64. If the architecture reference manuals are wrong, is there an errata ? As for the algorithm to use on ARMv8 to update instructions, see [2] B2.3.4 Implication of caches for the application programmer "Synchronization and coherency issues between data and instruction accesses" Membarrier only takes care of making sure the "ISB" part of the algorithm can be done easily and efficiently on multiprocessor systems. Thanks, Mathieu [1] https://developer.arm.com/documentation/den0024/a/Memory-Ordering/Barriers/ISB-in-more-detail [2] https://montcs.bloomu.edu/Information/ARMv8/ARMv8-A_Architecture_Reference_Manual_(Issue_A.a).pdf -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77A69C433E0 for ; Mon, 28 Dec 2020 21:11:22 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 73A45207C9 for ; Mon, 28 Dec 2020 21:11:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 73A45207C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=efficios.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4D4VbC1TKGzDqDM for ; Tue, 29 Dec 2020 08:11:19 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=efficios.com (client-ip=167.114.26.124; helo=mail.efficios.com; envelope-from=compudj@efficios.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=efficios.com header.i=@efficios.com header.a=rsa-sha256 header.s=default header.b=p/szlAxK; dkim-atps=neutral Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4D4VYM4cFszDqCp for ; Tue, 29 Dec 2020 08:09:42 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 063922761C7; Mon, 28 Dec 2020 16:09:40 -0500 (EST) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Xl2HNlqr6BSs; Mon, 28 Dec 2020 16:09:39 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 8512E2761C5; Mon, 28 Dec 2020 16:09:39 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 8512E2761C5 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1609189779; bh=A+A/Blx9QD1hoONN5Jxqqa59/BnSFAuRhzOnDZD8md8=; h=Date:From:To:Message-ID:MIME-Version; b=p/szlAxKXnb8sMCXgDpGdE6is/o1rjODbnzdRovA9owiEjuUPlM3+dte9oFxY/+Hd pvcRHGvBLx8S5o5ANGCMKBwMqzBmFMfuacgLtZuDT1vBtUVUSIdSFMD3lwBW0Mm8Sq XkulYMWharVGPEE76YIKlwesiZMP7dKhpXj/gToE1q4CiEAv+QDgP6f1MTue34+EDL yC7abPFDn6sQU4EVRsjnLUgN48sbY+nRlQq9VYRasErERk2As+18wk5Q0JRbIc/lci ltQp4+4byni+b6Qb7IBjB/By21MOffNXi7Ev/3OOnq+SzZHcRZHXttXzb3ZGlGI4S3 +WkZTiCC1H5Hw== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id BWu1pC_47JoY; Mon, 28 Dec 2020 16:09:39 -0500 (EST) Received: from mail03.efficios.com (mail03.efficios.com [167.114.26.124]) by mail.efficios.com (Postfix) with ESMTP id 725FE27640E; Mon, 28 Dec 2020 16:09:39 -0500 (EST) Date: Mon, 28 Dec 2020 16:09:39 -0500 (EST) From: Mathieu Desnoyers To: Andy Lutomirski , paulmck , Peter Zijlstra Message-ID: <1670059472.3671.1609189779376.JavaMail.zimbra@efficios.com> In-Reply-To: References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [167.114.26.124] X-Mailer: Zimbra 8.8.15_GA_3991 (ZimbraWebClient - FF84 (Linux)/8.8.15_GA_3980) Thread-Topic: membarrier: Rewrite sync_core_before_usermode() Thread-Index: UfUbgHPNYlsGbr5X+VxS/QBaGieq4w== X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Catalin Marinas , Arnd Bergmann , x86 , "Russell King, ARM Linux" , Nicholas Piggin , linux-kernel , Will Deacon , Paul Mackerras , stable , linuxppc-dev , linux-arm-kernel Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" ----- On Dec 27, 2020, at 4:36 PM, Andy Lutomirski luto@kernel.org wrote: [...] >> You seem to have noticed odd cases on arm64 where this guarantee does not >> match reality. Where exactly can we find this in the code, and which part >> of the architecture manual can you point us to which supports your concern ? >> >> Based on the notes I have, use of `eret` on aarch64 guarantees a context >> synchronizing >> instruction when returning to user-space. > > Based on my reading of the manual, ERET on ARM doesn't synchronize > anything at all. I can't find any evidence that it synchronizes data > or instructions, and I've seen reports that the CPU will happily > speculate right past it. Reading [1] there appears to be 3 kind of context synchronization events: - Taking an exception, - Returning from an exception, - ISB. This other source [2] adds (search for Context synchronization operation): - Exit from Debug state - Executing a DCPS instruction - Executing a DRPS instruction "ERET" falls into the second kind of events, and AFAIU should be context synchronizing. That was confirmed to me by Will Deacon when membarrier sync-core was implemented for aarch64. If the architecture reference manuals are wrong, is there an errata ? As for the algorithm to use on ARMv8 to update instructions, see [2] B2.3.4 Implication of caches for the application programmer "Synchronization and coherency issues between data and instruction accesses" Membarrier only takes care of making sure the "ISB" part of the algorithm can be done easily and efficiently on multiprocessor systems. Thanks, Mathieu [1] https://developer.arm.com/documentation/den0024/a/Memory-Ordering/Barriers/ISB-in-more-detail [2] https://montcs.bloomu.edu/Information/ARMv8/ARMv8-A_Architecture_Reference_Manual_(Issue_A.a).pdf -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com