From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 843E9C433E0 for ; Sun, 27 Dec 2020 21:37:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 59F79207FF for ; Sun, 27 Dec 2020 21:37:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726391AbgL0VhI (ORCPT ); Sun, 27 Dec 2020 16:37:08 -0500 Received: from mail.kernel.org ([198.145.29.99]:38922 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726260AbgL0VhI (ORCPT ); Sun, 27 Dec 2020 16:37:08 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 27CC0229F0 for ; Sun, 27 Dec 2020 21:36:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1609104987; bh=xf1xhuCSbIvU6yFstK8GVJEOxP3WIu96vlXc4UxPWjg=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=AqlCdXKHE+ITIKKRlqblXCSPB1fZi8/UDAb1RDKCM1LvBmfFUzWwvmRaxDmO2t3hX Nk6pMZGldD5DaO605Rtt8TjjpOD/gSk58ws/PQ9FMfNSuHWbyzwf58epdI6g3zk8xZ Igbu0xStOGQWdGd9Kogu2VJnNRAWN3zsxS3vBuHZhFn096dU225m4SpqgJdSX6aMov EBd0pc/BjZrshn7beSWmBBUXcEIKmk+336nrsvKpQCdQceJupZ7MtSF8mohlT+MVwl hKrhBGpC7tm8GB4+e/xrihyvUg8Ly26YAqTnggMTfP4n7AumuBvcriGH+eaPw+YB2h jGdZuFrFFMGPQ== Received: by mail-wr1-f44.google.com with SMTP id a12so8945668wrv.8 for ; Sun, 27 Dec 2020 13:36:27 -0800 (PST) X-Gm-Message-State: AOAM532S44puFBjJsT5CiDxGSXybxTuus3QTzlYFZo5sz84sg1EyILzo 9T70TECrfZNDU6onDXIm0X+aL2iNz3aUyrdd4O0o5A== X-Google-Smtp-Source: ABdhPJwL0d4jxly4xZo1ozG4f8+9aazYql5ttlOL3TT+E+3PWhEM6R+ijPQOdmXIla5FWQwjhuFdendweCd9l/QAEJQ= X-Received: by 2002:a5d:4905:: with SMTP id x5mr47749724wrq.75.1609104985693; Sun, 27 Dec 2020 13:36:25 -0800 (PST) MIME-Version: 1.0 References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> In-Reply-To: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> From: Andy Lutomirski Date: Sun, 27 Dec 2020 13:36:13 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() To: Mathieu Desnoyers , Russell King Cc: Andy Lutomirski , x86 , linux-kernel , Nicholas Piggin , Arnd Bergmann , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , linuxppc-dev , Catalin Marinas , Will Deacon , linux-arm-kernel , stable Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, Dec 27, 2020 at 12:18 PM Mathieu Desnoyers wrote: > > ----- On Dec 27, 2020, at 1:28 PM, Andy Lutomirski luto@kernel.org wrote: > > > > > I admit that I'm rather surprised that the code worked at all on arm64, > > and I'm suspicious that it has never been very well tested. My apologies > > for not reviewing this more carefully in the first place. > > Please refer to Documentation/features/sched/membarrier-sync-core/arch-support.txt > > It clearly states that only arm, arm64, powerpc and x86 support the membarrier > sync core feature as of now: Sigh, I missed arm (32). Russell or ARM folks, what's the right incantation to make the CPU notice instruction changes initiated by other cores on 32-bit ARM? > > > # Architecture requirements > # > # * arm/arm64/powerpc > # > # Rely on implicit context synchronization as a result of exception return > # when returning from IPI handler, and when returning to user-space. > # > # * x86 > # > # x86-32 uses IRET as return from interrupt, which takes care of the IPI. > # However, it uses both IRET and SYSEXIT to go back to user-space. The IRET > # instruction is core serializing, but not SYSEXIT. > # > # x86-64 uses IRET as return from interrupt, which takes care of the IPI. > # However, it can return to user-space through either SYSRETL (compat code), > # SYSRETQ, or IRET. > # > # Given that neither SYSRET{L,Q}, nor SYSEXIT, are core serializing, we rely > # instead on write_cr3() performed by switch_mm() to provide core serialization > # after changing the current mm, and deal with the special case of kthread -> > # uthread (temporarily keeping current mm into active_mm) by issuing a > # sync_core_before_usermode() in that specific case. > I need to update that document as part of my series. > This is based on direct feedback from the architecture maintainers. > > You seem to have noticed odd cases on arm64 where this guarantee does not > match reality. Where exactly can we find this in the code, and which part > of the architecture manual can you point us to which supports your concern ? > > Based on the notes I have, use of `eret` on aarch64 guarantees a context synchronizing > instruction when returning to user-space. Based on my reading of the manual, ERET on ARM doesn't synchronize anything at all. I can't find any evidence that it synchronizes data or instructions, and I've seen reports that the CPU will happily speculate right past it. --Andy From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF11FC433E0 for ; Sun, 27 Dec 2020 21:38:11 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 00A87207B1 for ; Sun, 27 Dec 2020 21:38:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 00A87207B1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4D3vDd10lwzDqCK for ; Mon, 28 Dec 2020 08:38:09 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=198.145.29.99; helo=mail.kernel.org; envelope-from=luto@kernel.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=kernel.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=AqlCdXKH; dkim-atps=neutral Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4D3vBk563XzDqC7 for ; Mon, 28 Dec 2020 08:36:30 +1100 (AEDT) Received: by mail.kernel.org (Postfix) with ESMTPSA id 174EC207FF for ; Sun, 27 Dec 2020 21:36:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1609104987; bh=xf1xhuCSbIvU6yFstK8GVJEOxP3WIu96vlXc4UxPWjg=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=AqlCdXKHE+ITIKKRlqblXCSPB1fZi8/UDAb1RDKCM1LvBmfFUzWwvmRaxDmO2t3hX Nk6pMZGldD5DaO605Rtt8TjjpOD/gSk58ws/PQ9FMfNSuHWbyzwf58epdI6g3zk8xZ Igbu0xStOGQWdGd9Kogu2VJnNRAWN3zsxS3vBuHZhFn096dU225m4SpqgJdSX6aMov EBd0pc/BjZrshn7beSWmBBUXcEIKmk+336nrsvKpQCdQceJupZ7MtSF8mohlT+MVwl hKrhBGpC7tm8GB4+e/xrihyvUg8Ly26YAqTnggMTfP4n7AumuBvcriGH+eaPw+YB2h jGdZuFrFFMGPQ== Received: by mail-wr1-f53.google.com with SMTP id y17so8935648wrr.10 for ; Sun, 27 Dec 2020 13:36:27 -0800 (PST) X-Gm-Message-State: AOAM530Cz5RaQlvbo95+4OzKZO8ivATUhX2j2AB1lr0Jzb4+21O2GSI3 EYK1LCqt1sSuqMdtyL2dg+OT02TFhje38rS+QNZZUQ== X-Google-Smtp-Source: ABdhPJwL0d4jxly4xZo1ozG4f8+9aazYql5ttlOL3TT+E+3PWhEM6R+ijPQOdmXIla5FWQwjhuFdendweCd9l/QAEJQ= X-Received: by 2002:a5d:4905:: with SMTP id x5mr47749724wrq.75.1609104985693; Sun, 27 Dec 2020 13:36:25 -0800 (PST) MIME-Version: 1.0 References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> In-Reply-To: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> From: Andy Lutomirski Date: Sun, 27 Dec 2020 13:36:13 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() To: Mathieu Desnoyers , Russell King Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Catalin Marinas , Arnd Bergmann , x86 , linux-kernel , Nicholas Piggin , Will Deacon , Paul Mackerras , stable , Andy Lutomirski , linuxppc-dev , linux-arm-kernel Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Sun, Dec 27, 2020 at 12:18 PM Mathieu Desnoyers wrote: > > ----- On Dec 27, 2020, at 1:28 PM, Andy Lutomirski luto@kernel.org wrote: > > > > > I admit that I'm rather surprised that the code worked at all on arm64, > > and I'm suspicious that it has never been very well tested. My apologies > > for not reviewing this more carefully in the first place. > > Please refer to Documentation/features/sched/membarrier-sync-core/arch-support.txt > > It clearly states that only arm, arm64, powerpc and x86 support the membarrier > sync core feature as of now: Sigh, I missed arm (32). Russell or ARM folks, what's the right incantation to make the CPU notice instruction changes initiated by other cores on 32-bit ARM? > > > # Architecture requirements > # > # * arm/arm64/powerpc > # > # Rely on implicit context synchronization as a result of exception return > # when returning from IPI handler, and when returning to user-space. > # > # * x86 > # > # x86-32 uses IRET as return from interrupt, which takes care of the IPI. > # However, it uses both IRET and SYSEXIT to go back to user-space. The IRET > # instruction is core serializing, but not SYSEXIT. > # > # x86-64 uses IRET as return from interrupt, which takes care of the IPI. > # However, it can return to user-space through either SYSRETL (compat code), > # SYSRETQ, or IRET. > # > # Given that neither SYSRET{L,Q}, nor SYSEXIT, are core serializing, we rely > # instead on write_cr3() performed by switch_mm() to provide core serialization > # after changing the current mm, and deal with the special case of kthread -> > # uthread (temporarily keeping current mm into active_mm) by issuing a > # sync_core_before_usermode() in that specific case. > I need to update that document as part of my series. > This is based on direct feedback from the architecture maintainers. > > You seem to have noticed odd cases on arm64 where this guarantee does not > match reality. Where exactly can we find this in the code, and which part > of the architecture manual can you point us to which supports your concern ? > > Based on the notes I have, use of `eret` on aarch64 guarantees a context synchronizing > instruction when returning to user-space. Based on my reading of the manual, ERET on ARM doesn't synchronize anything at all. I can't find any evidence that it synchronizes data or instructions, and I've seen reports that the CPU will happily speculate right past it. --Andy From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2FF9C433E0 for ; Sun, 27 Dec 2020 21:38:21 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 89370207B1 for ; Sun, 27 Dec 2020 21:38:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 89370207B1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Y35zqqAR2QRF1WvGNg+cuIAxWSzYzNPGXD0J8CIs2c0=; b=Un5ZvDOC7G6zdv4jlregtZpk7 2q6hWk1PveUBm74IdK84/Y6NUitqUggUO6vrJVS2HIG7sE9rF0jAThUAwl5rTYtu5oJyfNs1rsBnn F0RgY7emK2EHH3SmQo58wwwq+RudJjYTaWGt5/orw0CTnkeSQc00fxilT9t6Auhc5ZPc5AHby5z4H ELFzHvGChtFVO9EEXBbDUY4I+ARUPLS3XW0xDo1OhlWqYu4sbnPoYrmNcum02PmN1rBI8unAQHEvE 0rRex9Y5DqTaGkrI0rEZyBzBDoFBprdlpEDkIjWNgxXZwcwKfFQA7VhdZq/MQDce9sfGk0qUc2zuR 5Vdq82+UA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ktdiJ-0002I6-V7; Sun, 27 Dec 2020 21:36:32 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1ktdiH-0002HG-3K for linux-arm-kernel@lists.infradead.org; Sun, 27 Dec 2020 21:36:30 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 38F3122AEC for ; Sun, 27 Dec 2020 21:36:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1609104987; bh=xf1xhuCSbIvU6yFstK8GVJEOxP3WIu96vlXc4UxPWjg=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=AqlCdXKHE+ITIKKRlqblXCSPB1fZi8/UDAb1RDKCM1LvBmfFUzWwvmRaxDmO2t3hX Nk6pMZGldD5DaO605Rtt8TjjpOD/gSk58ws/PQ9FMfNSuHWbyzwf58epdI6g3zk8xZ Igbu0xStOGQWdGd9Kogu2VJnNRAWN3zsxS3vBuHZhFn096dU225m4SpqgJdSX6aMov EBd0pc/BjZrshn7beSWmBBUXcEIKmk+336nrsvKpQCdQceJupZ7MtSF8mohlT+MVwl hKrhBGpC7tm8GB4+e/xrihyvUg8Ly26YAqTnggMTfP4n7AumuBvcriGH+eaPw+YB2h jGdZuFrFFMGPQ== Received: by mail-wr1-f47.google.com with SMTP id t16so8976487wra.3 for ; Sun, 27 Dec 2020 13:36:27 -0800 (PST) X-Gm-Message-State: AOAM533jcT1Cvsf/S4YnV43H9uuuqjHiJzsXUx3kIT8Hh1QJmDXl11GD ZLfjiFrRzlnZ/0n3iUkHx3d5xwSTvF+r4U80BLaevA== X-Google-Smtp-Source: ABdhPJwL0d4jxly4xZo1ozG4f8+9aazYql5ttlOL3TT+E+3PWhEM6R+ijPQOdmXIla5FWQwjhuFdendweCd9l/QAEJQ= X-Received: by 2002:a5d:4905:: with SMTP id x5mr47749724wrq.75.1609104985693; Sun, 27 Dec 2020 13:36:25 -0800 (PST) MIME-Version: 1.0 References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> In-Reply-To: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> From: Andy Lutomirski Date: Sun, 27 Dec 2020 13:36:13 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() To: Mathieu Desnoyers , Russell King X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201227_163629_354822_724EEE71 X-CRM114-Status: GOOD ( 23.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Catalin Marinas , Arnd Bergmann , Michael Ellerman , x86 , linux-kernel , Nicholas Piggin , Will Deacon , Paul Mackerras , stable , Andy Lutomirski , Benjamin Herrenschmidt , linuxppc-dev , linux-arm-kernel Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Sun, Dec 27, 2020 at 12:18 PM Mathieu Desnoyers wrote: > > ----- On Dec 27, 2020, at 1:28 PM, Andy Lutomirski luto@kernel.org wrote: > > > > > I admit that I'm rather surprised that the code worked at all on arm64, > > and I'm suspicious that it has never been very well tested. My apologies > > for not reviewing this more carefully in the first place. > > Please refer to Documentation/features/sched/membarrier-sync-core/arch-support.txt > > It clearly states that only arm, arm64, powerpc and x86 support the membarrier > sync core feature as of now: Sigh, I missed arm (32). Russell or ARM folks, what's the right incantation to make the CPU notice instruction changes initiated by other cores on 32-bit ARM? > > > # Architecture requirements > # > # * arm/arm64/powerpc > # > # Rely on implicit context synchronization as a result of exception return > # when returning from IPI handler, and when returning to user-space. > # > # * x86 > # > # x86-32 uses IRET as return from interrupt, which takes care of the IPI. > # However, it uses both IRET and SYSEXIT to go back to user-space. The IRET > # instruction is core serializing, but not SYSEXIT. > # > # x86-64 uses IRET as return from interrupt, which takes care of the IPI. > # However, it can return to user-space through either SYSRETL (compat code), > # SYSRETQ, or IRET. > # > # Given that neither SYSRET{L,Q}, nor SYSEXIT, are core serializing, we rely > # instead on write_cr3() performed by switch_mm() to provide core serialization > # after changing the current mm, and deal with the special case of kthread -> > # uthread (temporarily keeping current mm into active_mm) by issuing a > # sync_core_before_usermode() in that specific case. > I need to update that document as part of my series. > This is based on direct feedback from the architecture maintainers. > > You seem to have noticed odd cases on arm64 where this guarantee does not > match reality. Where exactly can we find this in the code, and which part > of the architecture manual can you point us to which supports your concern ? > > Based on the notes I have, use of `eret` on aarch64 guarantees a context synchronizing > instruction when returning to user-space. Based on my reading of the manual, ERET on ARM doesn't synchronize anything at all. I can't find any evidence that it synchronizes data or instructions, and I've seen reports that the CPU will happily speculate right past it. --Andy _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel