From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755075AbdKNOxv (ORCPT ); Tue, 14 Nov 2017 09:53:51 -0500 Received: from mail-wm0-f48.google.com ([74.125.82.48]:52493 "EHLO mail-wm0-f48.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753869AbdKNOxl (ORCPT ); Tue, 14 Nov 2017 09:53:41 -0500 X-Google-Smtp-Source: AGs4zMaBJUhTVZ1TDqaNMEpDxUKvLgrs1YP4o2gpwXBtJVJKZmYBeyvmHTwW6snWZsATmHr7liE+oA== Subject: Re: [RFC PATCH 0/2] x86: Fix missing core serialization on migration To: Mathieu Desnoyers , Linus Torvalds Cc: Andy Lutomirski , linux-kernel , linux-api , Peter Zijlstra , "Paul E. McKenney" , Boqun Feng , Andrew Hunter , maged michael , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Dave Watson , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , Andrea Parri , "Russell King, ARM Linux" , Greg Hackmann , Will Deacon , David Sehr , x86 References: <20171110211249.10742-1-mathieu.desnoyers@efficios.com> <885227610.13045.1510351034488.JavaMail.zimbra@efficios.com> <617343212.13932.1510592207202.JavaMail.zimbra@efficios.com> From: Avi Kivity Organization: ScyllaDB Message-ID: <4d47fbb8-8f99-19d3-a9cf-66841aeffac3@scylladb.com> Date: Tue, 14 Nov 2017 16:53:34 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <617343212.13932.1510592207202.JavaMail.zimbra@efficios.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 11/13/2017 06:56 PM, Mathieu Desnoyers wrote: > ----- On Nov 10, 2017, at 4:57 PM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote: > >> ----- On Nov 10, 2017, at 4:36 PM, Linus Torvalds torvalds@linux-foundation.org >> wrote: >> >>> On Fri, Nov 10, 2017 at 1:12 PM, Mathieu Desnoyers >>> wrote: >>>> x86 can return to user-space through sysexit and sysretq, which are not >>>> core serializing. This breaks expectations from user-space about >>>> sequential consistency from a single-threaded self-modifying program >>>> point of view in specific migration patterns. >>>> >>>> Feedback is welcome, >>> We should check with Intel. I would actually be surprised if the I$ >>> can be out of sync with the D$ after a sysretq. It would actually >>> break things like "read code from disk" too in theory. >> That core serializing instruction is not that much about I$ vs D$ >> consistency, but rather about the processor speculatively executing code >> ahead of its retirement point. Ref. Intel Architecture Software Developer's >> Manual, Volume 3: System Programming. >> >> 7.1.3. "Handling Self- and Cross-Modifying Code": >> >> "The act of a processor writing data into a currently executing code segment >> with the intent of >> executing that data as code is called self-modifying code. Intel Architecture >> processors exhibit >> model-specific behavior when executing self-modified code, depending upon how >> far ahead of >> the current execution pointer the code has been modified. As processor >> architectures become >> more complex and start to speculatively execute code ahead of the retirement >> point (as in the P6 >> family processors), the rules regarding which code should execute, pre- or >> post-modification, >> become blurred. [...]" >> >> AFAIU, this core serializing instruction seems to be needed for use-cases of >> self-modifying code, but not for the initial load of a program from disk, >> as the processor has no way to have speculatively executed any of its >> instructions. > I figured out what you're pointing to: if exec() is executed by a previously > running thread, and there is no core serializing instruction between program > load and return to user-space, the kernel ends up acting like a JIT, indeed. I think that's safe. The kernel has to execute a MOV CR3 instruction before it can execute code loaded by exec, and that is a serializing instruction. Loading and unloading shared libraries is made safe by the IRET executed by page faults (loading) and TLB shootdown IPIs (unloading). Directly modifying code in userspace is unsafe if there is some non-coherent instruction cache. Instruction fetch and speculative execution are non-coherent, but they're probably too short (in current processors) to matter. Trace caches are probably large enough, but I don't know whether they are coherent or not. > > Therefore, we'd also need to invoke sync_core_before_usermode() after loading > the program. > > Let's wait to hear back from hpa, > > Thanks, > > Mathieu > > >> Hopefully hpa can tell us more about this, >> >> Thanks, >> >> Mathieu >> >> >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> http://www.efficios.com From mboxrd@z Thu Jan 1 00:00:00 1970 From: Avi Kivity Subject: Re: [RFC PATCH 0/2] x86: Fix missing core serialization on migration Date: Tue, 14 Nov 2017 16:53:34 +0200 Message-ID: <4d47fbb8-8f99-19d3-a9cf-66841aeffac3@scylladb.com> References: <20171110211249.10742-1-mathieu.desnoyers@efficios.com> <885227610.13045.1510351034488.JavaMail.zimbra@efficios.com> <617343212.13932.1510592207202.JavaMail.zimbra@efficios.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <617343212.13932.1510592207202.JavaMail.zimbra-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org> Content-Language: en-US Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Mathieu Desnoyers , Linus Torvalds Cc: Andy Lutomirski , linux-kernel , linux-api , Peter Zijlstra , "Paul E. McKenney" , Boqun Feng , Andrew Hunter , maged michael , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Dave Watson , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , Andrea Parri , "Russell King, ARM Linux" , Greg Hackmann , Will Deacon Dav List-Id: linux-api@vger.kernel.org On 11/13/2017 06:56 PM, Mathieu Desnoyers wrote: > ----- On Nov 10, 2017, at 4:57 PM, Mathieu Desnoyers mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org wrote: > >> ----- On Nov 10, 2017, at 4:36 PM, Linus Torvalds torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org >> wrote: >> >>> On Fri, Nov 10, 2017 at 1:12 PM, Mathieu Desnoyers >>> wrote: >>>> x86 can return to user-space through sysexit and sysretq, which are not >>>> core serializing. This breaks expectations from user-space about >>>> sequential consistency from a single-threaded self-modifying program >>>> point of view in specific migration patterns. >>>> >>>> Feedback is welcome, >>> We should check with Intel. I would actually be surprised if the I$ >>> can be out of sync with the D$ after a sysretq. It would actually >>> break things like "read code from disk" too in theory. >> That core serializing instruction is not that much about I$ vs D$ >> consistency, but rather about the processor speculatively executing code >> ahead of its retirement point. Ref. Intel Architecture Software Developer's >> Manual, Volume 3: System Programming. >> >> 7.1.3. "Handling Self- and Cross-Modifying Code": >> >> "The act of a processor writing data into a currently executing code segment >> with the intent of >> executing that data as code is called self-modifying code. Intel Architecture >> processors exhibit >> model-specific behavior when executing self-modified code, depending upon how >> far ahead of >> the current execution pointer the code has been modified. As processor >> architectures become >> more complex and start to speculatively execute code ahead of the retirement >> point (as in the P6 >> family processors), the rules regarding which code should execute, pre- or >> post-modification, >> become blurred. [...]" >> >> AFAIU, this core serializing instruction seems to be needed for use-cases of >> self-modifying code, but not for the initial load of a program from disk, >> as the processor has no way to have speculatively executed any of its >> instructions. > I figured out what you're pointing to: if exec() is executed by a previously > running thread, and there is no core serializing instruction between program > load and return to user-space, the kernel ends up acting like a JIT, indeed. I think that's safe. The kernel has to execute a MOV CR3 instruction before it can execute code loaded by exec, and that is a serializing instruction. Loading and unloading shared libraries is made safe by the IRET executed by page faults (loading) and TLB shootdown IPIs (unloading). Directly modifying code in userspace is unsafe if there is some non-coherent instruction cache. Instruction fetch and speculative execution are non-coherent, but they're probably too short (in current processors) to matter. Trace caches are probably large enough, but I don't know whether they are coherent or not. > > Therefore, we'd also need to invoke sync_core_before_usermode() after loading > the program. > > Let's wait to hear back from hpa, > > Thanks, > > Mathieu > > >> Hopefully hpa can tell us more about this, >> >> Thanks, >> >> Mathieu >> >> >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> http://www.efficios.com