From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A7A9C433E0 for ; Mon, 28 Dec 2020 18:51:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 44306229EF for ; Mon, 28 Dec 2020 18:51:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727471AbgL1SvQ (ORCPT ); Mon, 28 Dec 2020 13:51:16 -0500 Received: from mail.kernel.org ([198.145.29.99]:37982 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726420AbgL1SvP (ORCPT ); Mon, 28 Dec 2020 13:51:15 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 4694222B2B for ; Mon, 28 Dec 2020 18:50:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1609181434; bh=6YqwLVWGZ7ybOF3Ak7Rutxzpv/0P68frB/ABbtNRx2A=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=AaTJQGDx58ZoPUwIflwplFjWLnIFD807YZxgy/wYiFKWT1laWnPttoNEf6fcvtHHU +1CZ1iOYgjTejeFWS++DiH29gBHSy8svPa+lVVWyGWKVVVrLK0b4deZrIMXHHkUZav Cbh7w6P49o2HymZPH6nKYAV5V/DIz4ohgdngS6R+6FETrskhWBi87AJduXHvoFDd99 f4+7QHkuPbqsmYqNWe6Q41r1iNa231xtvKSnlNbahIH/zwC+CDwxNnAeD2oXFPcWSf M9A+zrh7bnLWxYVHHSprHpra4oKbHl/w+xYPE2jSLv3Y/SWoptfm2qEu1lf6V4xPqh 2uJEMxqBjqKBw== Received: by mail-wm1-f42.google.com with SMTP id e25so271046wme.0 for ; Mon, 28 Dec 2020 10:50:34 -0800 (PST) X-Gm-Message-State: AOAM533BcUcxTCi1yN6c/YUR1VxWfH0gXeMSVGByfkdrwa9X78RguDtw ruwKQ/uDTodjdZVva6CeaYZUj7ryI/MEoAPOaIgHGA== X-Google-Smtp-Source: ABdhPJy2pGQl+qZycpaVqhSpXhzX+qb6KsnFF26UHOwhZZYqp85PUh6OZSCBMuyYlamRKZ+/qeujbpFqUc9sglmMa5s= X-Received: by 2002:a7b:c40b:: with SMTP id k11mr257558wmi.36.1609181432795; Mon, 28 Dec 2020 10:50:32 -0800 (PST) MIME-Version: 1.0 References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> <20201228102537.GG1551@shell.armlinux.org.uk> In-Reply-To: From: Andy Lutomirski Date: Mon, 28 Dec 2020 10:50:20 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() To: Jann Horn Cc: Andy Lutomirski , Will Deacon , Russell King - ARM Linux admin , Mathieu Desnoyers , x86 , linux-kernel , Nicholas Piggin , Arnd Bergmann , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , linuxppc-dev , Catalin Marinas , linux-arm-kernel , stable Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Dec 28, 2020 at 10:30 AM Jann Horn wrote: > > On Mon, Dec 28, 2020 at 6:14 PM Andy Lutomirski wrote: > > On Mon, Dec 28, 2020 at 2:25 AM Russell King - ARM Linux admin > > wrote: > > > > > > On Sun, Dec 27, 2020 at 01:36:13PM -0800, Andy Lutomirski wrote: > > > > On Sun, Dec 27, 2020 at 12:18 PM Mathieu Desnoyers > > > > wrote: > > > > > > > > > > ----- On Dec 27, 2020, at 1:28 PM, Andy Lutomirski luto@kernel.org wrote: > > > > > > > > > > > > > > > > > > > > > I admit that I'm rather surprised that the code worked at all on arm64, > > > > > > and I'm suspicious that it has never been very well tested. My apologies > > > > > > for not reviewing this more carefully in the first place. > > > > > > > > > > Please refer to Documentation/features/sched/membarrier-sync-core/arch-support.txt > > > > > > > > > > It clearly states that only arm, arm64, powerpc and x86 support the membarrier > > > > > sync core feature as of now: > > > > > > > > Sigh, I missed arm (32). Russell or ARM folks, what's the right > > > > incantation to make the CPU notice instruction changes initiated by > > > > other cores on 32-bit ARM? > > > > > > You need to call flush_icache_range(), since the changes need to be > > > flushed from the data cache to the point of unification (of the Harvard > > > I and D), and the instruction cache needs to be invalidated so it can > > > then see those updated instructions. This will also take care of the > > > necessary barriers that the CPU requires for you. > > > > With what parameters? From looking at the header, this is for the > > case in which the kernel writes some memory and then intends to > > execute it. That's not what membarrier() does at all. membarrier() > > works like this: > > > > User thread 1: > > > > write to RWX memory *or* write to an RW alias of an X region. > > membarrier(...); > > somehow tell thread 2 that we're ready (with a store release, perhaps, > > or even just a relaxed store.) > > > > User thread 2: > > > > wait for the indication from thread 1. > > barrier(); > > jump to the code. > > > > membarrier() is, for better or for worse, not given a range of addresses. > > > > On x86, the documentation is a bit weak, but a strict reading > > indicates that thread 2 must execute a serializing instruction at some > > point after thread 1 writes the code and before thread 2 runs it. > > membarrier() does this by sending an IPI and ensuring that a > > "serializing" instruction (thanks for great terminology, Intel) is > > executed. Note that flush_icache_range() is a no-op on x86, and I've > > asked Intel's architects to please clarify their precise rules. No > > response yet. > > > > On arm64, flush_icache_range() seems to send an IPI, and that's not > > what I want. membarrier() already does an IPI. > > After chatting with rmk about this (but without claiming that any of > this is his opinion), based on the manpage, I think membarrier() > currently doesn't really claim to be synchronizing caches? It just > serializes cores. So arguably if userspace wants to use membarrier() > to synchronize code changes, userspace should first do the code > change, then flush icache as appropriate for the architecture, and > then do the membarrier() to ensure that the old code is unused? I haven't the faintest clue what "serializes cores" means. It seems to be a bit of a mishmash of x86 SDM terminology and Linux x86 "sync_core" terminology. The latter means very little to me, even as an x86 person. I'm moderately confident that the *intent* is that a multithreaded program can write some JIT code to memory, do this membarrier() operation, and then execute the code, and it will work. Maybe it's even intended to work cross-architecture without any additional help from the program. But maybe the existing API is simply incorrect for this. > > For 32-bit arm, rmk pointed out that that would be the cacheflush() > syscall. That might cause you to end up with two IPIs instead of one > in total, but we probably don't care _that_ much about extra IPIs on > 32-bit arm? > > For arm64, I believe userspace can flush icache across the entire > system with some instructions from userspace - "DC CVAU" followed by > "DSB ISH", or something like that, I think? (See e.g. > compat_arm_syscall(), the arm64 compat code that implements the 32-bit > arm cacheflush() syscall.) I have no idea what DC anything does. Based on my very cursory reading of the manual, ISB is the right approach, but I don't pretend I understand all the details. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0066AC433E0 for ; Mon, 28 Dec 2020 18:52:16 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2D1E4221F8 for ; Mon, 28 Dec 2020 18:52:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2D1E4221F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4D4RVj0vPQzDqFD for ; Tue, 29 Dec 2020 05:52:13 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=198.145.29.99; helo=mail.kernel.org; envelope-from=luto@kernel.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=kernel.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=AaTJQGDx; dkim-atps=neutral Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4D4RSs6pQyzDqC3 for ; Tue, 29 Dec 2020 05:50:37 +1100 (AEDT) Received: by mail.kernel.org (Postfix) with ESMTPSA id 55C5822B3B for ; Mon, 28 Dec 2020 18:50:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1609181434; bh=6YqwLVWGZ7ybOF3Ak7Rutxzpv/0P68frB/ABbtNRx2A=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=AaTJQGDx58ZoPUwIflwplFjWLnIFD807YZxgy/wYiFKWT1laWnPttoNEf6fcvtHHU +1CZ1iOYgjTejeFWS++DiH29gBHSy8svPa+lVVWyGWKVVVrLK0b4deZrIMXHHkUZav Cbh7w6P49o2HymZPH6nKYAV5V/DIz4ohgdngS6R+6FETrskhWBi87AJduXHvoFDd99 f4+7QHkuPbqsmYqNWe6Q41r1iNa231xtvKSnlNbahIH/zwC+CDwxNnAeD2oXFPcWSf M9A+zrh7bnLWxYVHHSprHpra4oKbHl/w+xYPE2jSLv3Y/SWoptfm2qEu1lf6V4xPqh 2uJEMxqBjqKBw== Received: by mail-wm1-f44.google.com with SMTP id c133so231716wme.4 for ; Mon, 28 Dec 2020 10:50:34 -0800 (PST) X-Gm-Message-State: AOAM530PEr+687hWmbCasW8VBC3kPB55FVXKreHrv4zkuitzPAFvSxV6 0s4EH/nBaz/CeZzg6sWJxbs1Kh1uQPC8k4RCGxaqPw== X-Google-Smtp-Source: ABdhPJy2pGQl+qZycpaVqhSpXhzX+qb6KsnFF26UHOwhZZYqp85PUh6OZSCBMuyYlamRKZ+/qeujbpFqUc9sglmMa5s= X-Received: by 2002:a7b:c40b:: with SMTP id k11mr257558wmi.36.1609181432795; Mon, 28 Dec 2020 10:50:32 -0800 (PST) MIME-Version: 1.0 References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> <20201228102537.GG1551@shell.armlinux.org.uk> In-Reply-To: From: Andy Lutomirski Date: Mon, 28 Dec 2020 10:50:20 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() To: Jann Horn Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linuxppc-dev , Catalin Marinas , Arnd Bergmann , x86 , Russell King - ARM Linux admin , Nicholas Piggin , linux-kernel , Mathieu Desnoyers , stable , Andy Lutomirski , Paul Mackerras , Will Deacon , linux-arm-kernel Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Mon, Dec 28, 2020 at 10:30 AM Jann Horn wrote: > > On Mon, Dec 28, 2020 at 6:14 PM Andy Lutomirski wrote: > > On Mon, Dec 28, 2020 at 2:25 AM Russell King - ARM Linux admin > > wrote: > > > > > > On Sun, Dec 27, 2020 at 01:36:13PM -0800, Andy Lutomirski wrote: > > > > On Sun, Dec 27, 2020 at 12:18 PM Mathieu Desnoyers > > > > wrote: > > > > > > > > > > ----- On Dec 27, 2020, at 1:28 PM, Andy Lutomirski luto@kernel.org wrote: > > > > > > > > > > > > > > > > > > > > > I admit that I'm rather surprised that the code worked at all on arm64, > > > > > > and I'm suspicious that it has never been very well tested. My apologies > > > > > > for not reviewing this more carefully in the first place. > > > > > > > > > > Please refer to Documentation/features/sched/membarrier-sync-core/arch-support.txt > > > > > > > > > > It clearly states that only arm, arm64, powerpc and x86 support the membarrier > > > > > sync core feature as of now: > > > > > > > > Sigh, I missed arm (32). Russell or ARM folks, what's the right > > > > incantation to make the CPU notice instruction changes initiated by > > > > other cores on 32-bit ARM? > > > > > > You need to call flush_icache_range(), since the changes need to be > > > flushed from the data cache to the point of unification (of the Harvard > > > I and D), and the instruction cache needs to be invalidated so it can > > > then see those updated instructions. This will also take care of the > > > necessary barriers that the CPU requires for you. > > > > With what parameters? From looking at the header, this is for the > > case in which the kernel writes some memory and then intends to > > execute it. That's not what membarrier() does at all. membarrier() > > works like this: > > > > User thread 1: > > > > write to RWX memory *or* write to an RW alias of an X region. > > membarrier(...); > > somehow tell thread 2 that we're ready (with a store release, perhaps, > > or even just a relaxed store.) > > > > User thread 2: > > > > wait for the indication from thread 1. > > barrier(); > > jump to the code. > > > > membarrier() is, for better or for worse, not given a range of addresses. > > > > On x86, the documentation is a bit weak, but a strict reading > > indicates that thread 2 must execute a serializing instruction at some > > point after thread 1 writes the code and before thread 2 runs it. > > membarrier() does this by sending an IPI and ensuring that a > > "serializing" instruction (thanks for great terminology, Intel) is > > executed. Note that flush_icache_range() is a no-op on x86, and I've > > asked Intel's architects to please clarify their precise rules. No > > response yet. > > > > On arm64, flush_icache_range() seems to send an IPI, and that's not > > what I want. membarrier() already does an IPI. > > After chatting with rmk about this (but without claiming that any of > this is his opinion), based on the manpage, I think membarrier() > currently doesn't really claim to be synchronizing caches? It just > serializes cores. So arguably if userspace wants to use membarrier() > to synchronize code changes, userspace should first do the code > change, then flush icache as appropriate for the architecture, and > then do the membarrier() to ensure that the old code is unused? I haven't the faintest clue what "serializes cores" means. It seems to be a bit of a mishmash of x86 SDM terminology and Linux x86 "sync_core" terminology. The latter means very little to me, even as an x86 person. I'm moderately confident that the *intent* is that a multithreaded program can write some JIT code to memory, do this membarrier() operation, and then execute the code, and it will work. Maybe it's even intended to work cross-architecture without any additional help from the program. But maybe the existing API is simply incorrect for this. > > For 32-bit arm, rmk pointed out that that would be the cacheflush() > syscall. That might cause you to end up with two IPIs instead of one > in total, but we probably don't care _that_ much about extra IPIs on > 32-bit arm? > > For arm64, I believe userspace can flush icache across the entire > system with some instructions from userspace - "DC CVAU" followed by > "DSB ISH", or something like that, I think? (See e.g. > compat_arm_syscall(), the arm64 compat code that implements the 32-bit > arm cacheflush() syscall.) I have no idea what DC anything does. Based on my very cursory reading of the manual, ISB is the right approach, but I don't pretend I understand all the details. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B54B4C433DB for ; Mon, 28 Dec 2020 18:52:04 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7379A221F8 for ; Mon, 28 Dec 2020 18:52:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7379A221F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1EAingvbsSr03Htwfb3817EqF2DsyFMisFMQPft/2TA=; b=U6gcsHgkaEuuoxp5kYJtqqXqr 9oYBQgz7O0xMZ5wFl6gj5A6qUyefDfWCMbg9AprRBaWXAVkAun/kMxTg7Bb0xuSZbC9HlSoqOk629 rscwvphZ4Ax3FuuXoIXo5SJm5uIcd1/tzJuh3zaZ0ir/JF3R/u3OVMGCorHL9XE6T8TAgg3AnNnFj wZQ78xJZxcQaU1+KIDvhdgkABcKnjlYl0IpWgXgm4PXqP2O9Px8mFnTzE5G1sFK2bPLREPktJ2q/r /2J5qR9+KX0TOeRRhs4j8ky9kP1EOTJomCIBStMCGZA4AXkgJfUOPtVJuZN4TwOfQo+lEjCX3V1KP wSjbjNb3A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ktxbM-0006pz-9l; Mon, 28 Dec 2020 18:50:40 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1ktxbJ-0006pF-Ms for linux-arm-kernel@lists.infradead.org; Mon, 28 Dec 2020 18:50:38 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 2A9C2221F8 for ; Mon, 28 Dec 2020 18:50:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1609181434; bh=6YqwLVWGZ7ybOF3Ak7Rutxzpv/0P68frB/ABbtNRx2A=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=AaTJQGDx58ZoPUwIflwplFjWLnIFD807YZxgy/wYiFKWT1laWnPttoNEf6fcvtHHU +1CZ1iOYgjTejeFWS++DiH29gBHSy8svPa+lVVWyGWKVVVrLK0b4deZrIMXHHkUZav Cbh7w6P49o2HymZPH6nKYAV5V/DIz4ohgdngS6R+6FETrskhWBi87AJduXHvoFDd99 f4+7QHkuPbqsmYqNWe6Q41r1iNa231xtvKSnlNbahIH/zwC+CDwxNnAeD2oXFPcWSf M9A+zrh7bnLWxYVHHSprHpra4oKbHl/w+xYPE2jSLv3Y/SWoptfm2qEu1lf6V4xPqh 2uJEMxqBjqKBw== Received: by mail-wm1-f44.google.com with SMTP id v14so241178wml.1 for ; Mon, 28 Dec 2020 10:50:34 -0800 (PST) X-Gm-Message-State: AOAM533sVYlQ4TOhWU6tbPSM3YFjvEwPrmCRquCuZR4zNZvCKPTV4T9r g5MMohp1b5UDzB79kTfOU3m6UoyTMT/f9DNNjPaYmQ== X-Google-Smtp-Source: ABdhPJy2pGQl+qZycpaVqhSpXhzX+qb6KsnFF26UHOwhZZYqp85PUh6OZSCBMuyYlamRKZ+/qeujbpFqUc9sglmMa5s= X-Received: by 2002:a7b:c40b:: with SMTP id k11mr257558wmi.36.1609181432795; Mon, 28 Dec 2020 10:50:32 -0800 (PST) MIME-Version: 1.0 References: <1836294649.3345.1609100294833.JavaMail.zimbra@efficios.com> <20201228102537.GG1551@shell.armlinux.org.uk> In-Reply-To: From: Andy Lutomirski Date: Mon, 28 Dec 2020 10:50:20 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode() To: Jann Horn X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201228_135037_894833_7CE34187 X-CRM114-Status: GOOD ( 39.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linuxppc-dev , Catalin Marinas , Arnd Bergmann , Michael Ellerman , x86 , Russell King - ARM Linux admin , Nicholas Piggin , linux-kernel , Mathieu Desnoyers , stable , Andy Lutomirski , Benjamin Herrenschmidt , Paul Mackerras , Will Deacon , linux-arm-kernel Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Dec 28, 2020 at 10:30 AM Jann Horn wrote: > > On Mon, Dec 28, 2020 at 6:14 PM Andy Lutomirski wrote: > > On Mon, Dec 28, 2020 at 2:25 AM Russell King - ARM Linux admin > > wrote: > > > > > > On Sun, Dec 27, 2020 at 01:36:13PM -0800, Andy Lutomirski wrote: > > > > On Sun, Dec 27, 2020 at 12:18 PM Mathieu Desnoyers > > > > wrote: > > > > > > > > > > ----- On Dec 27, 2020, at 1:28 PM, Andy Lutomirski luto@kernel.org wrote: > > > > > > > > > > > > > > > > > > > > > I admit that I'm rather surprised that the code worked at all on arm64, > > > > > > and I'm suspicious that it has never been very well tested. My apologies > > > > > > for not reviewing this more carefully in the first place. > > > > > > > > > > Please refer to Documentation/features/sched/membarrier-sync-core/arch-support.txt > > > > > > > > > > It clearly states that only arm, arm64, powerpc and x86 support the membarrier > > > > > sync core feature as of now: > > > > > > > > Sigh, I missed arm (32). Russell or ARM folks, what's the right > > > > incantation to make the CPU notice instruction changes initiated by > > > > other cores on 32-bit ARM? > > > > > > You need to call flush_icache_range(), since the changes need to be > > > flushed from the data cache to the point of unification (of the Harvard > > > I and D), and the instruction cache needs to be invalidated so it can > > > then see those updated instructions. This will also take care of the > > > necessary barriers that the CPU requires for you. > > > > With what parameters? From looking at the header, this is for the > > case in which the kernel writes some memory and then intends to > > execute it. That's not what membarrier() does at all. membarrier() > > works like this: > > > > User thread 1: > > > > write to RWX memory *or* write to an RW alias of an X region. > > membarrier(...); > > somehow tell thread 2 that we're ready (with a store release, perhaps, > > or even just a relaxed store.) > > > > User thread 2: > > > > wait for the indication from thread 1. > > barrier(); > > jump to the code. > > > > membarrier() is, for better or for worse, not given a range of addresses. > > > > On x86, the documentation is a bit weak, but a strict reading > > indicates that thread 2 must execute a serializing instruction at some > > point after thread 1 writes the code and before thread 2 runs it. > > membarrier() does this by sending an IPI and ensuring that a > > "serializing" instruction (thanks for great terminology, Intel) is > > executed. Note that flush_icache_range() is a no-op on x86, and I've > > asked Intel's architects to please clarify their precise rules. No > > response yet. > > > > On arm64, flush_icache_range() seems to send an IPI, and that's not > > what I want. membarrier() already does an IPI. > > After chatting with rmk about this (but without claiming that any of > this is his opinion), based on the manpage, I think membarrier() > currently doesn't really claim to be synchronizing caches? It just > serializes cores. So arguably if userspace wants to use membarrier() > to synchronize code changes, userspace should first do the code > change, then flush icache as appropriate for the architecture, and > then do the membarrier() to ensure that the old code is unused? I haven't the faintest clue what "serializes cores" means. It seems to be a bit of a mishmash of x86 SDM terminology and Linux x86 "sync_core" terminology. The latter means very little to me, even as an x86 person. I'm moderately confident that the *intent* is that a multithreaded program can write some JIT code to memory, do this membarrier() operation, and then execute the code, and it will work. Maybe it's even intended to work cross-architecture without any additional help from the program. But maybe the existing API is simply incorrect for this. > > For 32-bit arm, rmk pointed out that that would be the cacheflush() > syscall. That might cause you to end up with two IPIs instead of one > in total, but we probably don't care _that_ much about extra IPIs on > 32-bit arm? > > For arm64, I believe userspace can flush icache across the entire > system with some instructions from userspace - "DC CVAU" followed by > "DSB ISH", or something like that, I think? (See e.g. > compat_arm_syscall(), the arm64 compat code that implements the 32-bit > arm cacheflush() syscall.) I have no idea what DC anything does. Based on my very cursory reading of the manual, ISB is the right approach, but I don't pretend I understand all the details. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel