From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF3D9C10F29 for ; Mon, 9 Mar 2020 23:59:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9D57D2146E for ; Mon, 9 Mar 2020 23:59:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=amacapital-net.20150623.gappssmtp.com header.i=@amacapital-net.20150623.gappssmtp.com header.b="bbtke5oP" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727498AbgCIX7v (ORCPT ); Mon, 9 Mar 2020 19:59:51 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:34498 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727287AbgCIX7v (ORCPT ); Mon, 9 Mar 2020 19:59:51 -0400 Received: by mail-wr1-f68.google.com with SMTP id z15so13550812wrl.1 for ; Mon, 09 Mar 2020 16:59:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amacapital-net.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0PcIMvT9Ges60/RCEcv9P9n3V3wEXCUl81zN9+mizqE=; b=bbtke5oP3UVppLVojB1atGqhDMEwwoZMv55eTr8ZnNZrgn0YDHohNAJDV+eqnhKipa 4aN9G0lOac5v18XPEriKg/5/D/0L6WNLOtgk+WPAkKIYHDuW4f7I6O8D05xQO9vTC/CI gvOE7yjLpbAqy2xs5nRCImHAI6w7kO6KCMC/FuGLSpEa/Cn8g0BDVHDycc6ierC/ThYB AZO4cFtCs7Q4J2LJBF7s+XfM3ycj5kw+gP7SQ2AYP7yKhuTikOByPG5P52PIBB6xvitH 8x8bbkXarJqpP2ySfHwYKIFOqsbvw5VjToymh99dZ/T1/NHd5AjR6yR4BXjfAUqy2tR0 52gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0PcIMvT9Ges60/RCEcv9P9n3V3wEXCUl81zN9+mizqE=; b=Uz32g+OoSTJrY7zm4QMktr2oEjjJifndsBvIXu9anFYt0wsQo7DAOT+oWter8Q85xF Ir1+I1w5r37Z94B1z407so3Ocf/L4GxB9OfWcGGgyCnN10HuCsAqjqGKOakJAWvPeRwe xrBHtqU29S05R/ZZcpEmtImCThL9fgQoUTKvgrI8ZYaGs1X3tJs8if0GypEXiVr4ajRp 9sOtluil4ADk0YciVbWU7vAiERpZ+S+OjpcXG9lRv/4UIgfcq46S4D0CyD97saMDxzwi YJt//HNs5ObDkzi65wWTy2LU/PKudWQMphE6W5H/Xrr3iLui3lMG3IzVWqSUjeFl3WcA +zag== X-Gm-Message-State: ANhLgQ02Bgz8iUualSRuy/mfS+x+H/lEQLFLDvXK6FDx0nxfu0JBKAyK lu7lBLHPfPemOfiqkTAj25hACC3Aqk1cOdPAV7YXcQ== X-Google-Smtp-Source: ADFU+vt2iiohYPhJ2F/0p/lc9pYIMGQoNqww6lZPdFqjkGInjDqr9voSFlL20LHhzIQclFgUMk+afkKjVc/hC1ajvTA= X-Received: by 2002:adf:b641:: with SMTP id i1mr23638629wre.18.1583798389063; Mon, 09 Mar 2020 16:59:49 -0700 (PDT) MIME-Version: 1.0 References: <0088001c-0b12-a7dc-ff2a-9d5c282fa36b@intel.com> <56ab33ac-865b-b37e-75f2-a489424566c3@intel.com> In-Reply-To: From: Andy Lutomirski Date: Mon, 9 Mar 2020 16:59:37 -0700 Message-ID: Subject: Re: [RFC PATCH v9 01/27] Documentation/x86: Add CET description To: "H.J. Lu" Cc: Dave Hansen , Yu-cheng Yu , "the arch/x86 maintainers" , "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , LKML , "open list:DOCUMENTATION" , Linux-MM , linux-arch , Linux API , Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin , x86-patch-review@intel.com Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Mar 9, 2020 at 4:52 PM H.J. Lu wrote: > > On Mon, Mar 9, 2020 at 4:21 PM Dave Hansen wrote: > > > > On 3/9/20 4:11 PM, H.J. Lu wrote: > > > A threaded application is loaded from disk. The object file on disk is > > > either CET enabled or not CET enabled. > > > > Huh. Are you saying that all instructions executed on userspace on > > Linux come off of object files on the disk? That's an interesting > > assertion. You might want to go take a look at the processes on your > > systems. Here's my browser for example: > > > > # for p in $(ps aux | grep chromium | awk '{print $2}' ); do cat > > /proc/$p/maps; done | grep ' r-xp 00000000 00:00 0' > > ... > > 202f00082000-202f000bf000 r-xp 00000000 00:00 0 > > 202f000c2000-202f000c3000 r-xp 00000000 00:00 0 > > 202f00102000-202f00103000 r-xp 00000000 00:00 0 > > 202f00142000-202f00143000 r-xp 00000000 00:00 0 > > 202f00182000-202f001bf000 r-xp 00000000 00:00 0 > > > > Lots of funny looking memory areas which are anonymous and executable! > > Those didn't come off the disk. Same thing in firefox. Weird. Any > > idea what those are? > > > > One guess: https://en.wikipedia.org/wiki/Just-in-time_compilation > > jitted code belongs to a process loaded from disk. Enable CET in > an application which uses JIT engine means to also enable CET in > JIT engine. Take git as an example, "git grep" crashed for me on Tiger > Lake. It turned out that git itself was compiled with -fcf-protection and > git was linked against libpcre2-8.so.0 also compiled with -fcf-protection, > which has a JIT, sljit, which was not CET enabled. git crashed in the > jitted codes due to missing ENDBR. I had to enable CET in sljit to make > git working on CET enabled Tiger Lake. So we need to enable CET in > JIT engine before enabling CET in applications which use JIT engine. This could presumably have been fixed by having libpcre or sljit disable IBT before calling into JIT code or by running the JIT code in another thread. In the other direction, a non-CET libpcre build could build IBT-capable JITted code and enable JIT (by syscall if we allow that or by creating a thread?) when calling it. And IBT has this fancy legacy bitmap to allow non-instrumented code to run with IBT on, although SHSTK doesn't have hardware support for a similar feature. So, sure, the glibc-linked ELF ecosystem needs some degree of CET coordination, but it is absolutely not the case that a process MUST have all CET or no CET. Let's please support the complicated cases in the kernel and the ABI too. If glibc wants to make it annoying to do complicated things, so be it. People work behind glibc's back all the time. --Andy From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andy Lutomirski Subject: Re: [RFC PATCH v9 01/27] Documentation/x86: Add CET description Date: Mon, 9 Mar 2020 16:59:37 -0700 Message-ID: References: <0088001c-0b12-a7dc-ff2a-9d5c282fa36b@intel.com> <56ab33ac-865b-b37e-75f2-a489424566c3@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: "H.J. Lu" Cc: Dave Hansen , Yu-cheng Yu , the arch/x86 maintainers , "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , LKML , "open list:DOCUMENTATION" , Linux-MM , linux-arch , Linux API , Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , Jann Horn List-Id: linux-arch.vger.kernel.org On Mon, Mar 9, 2020 at 4:52 PM H.J. Lu wrote: > > On Mon, Mar 9, 2020 at 4:21 PM Dave Hansen wrote: > > > > On 3/9/20 4:11 PM, H.J. Lu wrote: > > > A threaded application is loaded from disk. The object file on disk is > > > either CET enabled or not CET enabled. > > > > Huh. Are you saying that all instructions executed on userspace on > > Linux come off of object files on the disk? That's an interesting > > assertion. You might want to go take a look at the processes on your > > systems. Here's my browser for example: > > > > # for p in $(ps aux | grep chromium | awk '{print $2}' ); do cat > > /proc/$p/maps; done | grep ' r-xp 00000000 00:00 0' > > ... > > 202f00082000-202f000bf000 r-xp 00000000 00:00 0 > > 202f000c2000-202f000c3000 r-xp 00000000 00:00 0 > > 202f00102000-202f00103000 r-xp 00000000 00:00 0 > > 202f00142000-202f00143000 r-xp 00000000 00:00 0 > > 202f00182000-202f001bf000 r-xp 00000000 00:00 0 > > > > Lots of funny looking memory areas which are anonymous and executable! > > Those didn't come off the disk. Same thing in firefox. Weird. Any > > idea what those are? > > > > One guess: https://en.wikipedia.org/wiki/Just-in-time_compilation > > jitted code belongs to a process loaded from disk. Enable CET in > an application which uses JIT engine means to also enable CET in > JIT engine. Take git as an example, "git grep" crashed for me on Tiger > Lake. It turned out that git itself was compiled with -fcf-protection and > git was linked against libpcre2-8.so.0 also compiled with -fcf-protection, > which has a JIT, sljit, which was not CET enabled. git crashed in the > jitted codes due to missing ENDBR. I had to enable CET in sljit to make > git working on CET enabled Tiger Lake. So we need to enable CET in > JIT engine before enabling CET in applications which use JIT engine. This could presumably have been fixed by having libpcre or sljit disable IBT before calling into JIT code or by running the JIT code in another thread. In the other direction, a non-CET libpcre build could build IBT-capable JITted code and enable JIT (by syscall if we allow that or by creating a thread?) when calling it. And IBT has this fancy legacy bitmap to allow non-instrumented code to run with IBT on, although SHSTK doesn't have hardware support for a similar feature. So, sure, the glibc-linked ELF ecosystem needs some degree of CET coordination, but it is absolutely not the case that a process MUST have all CET or no CET. Let's please support the complicated cases in the kernel and the ABI too. If glibc wants to make it annoying to do complicated things, so be it. People work behind glibc's back all the time. --Andy From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23A08C10F25 for ; Mon, 9 Mar 2020 23:59:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C49222146E for ; Mon, 9 Mar 2020 23:59:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=amacapital-net.20150623.gappssmtp.com header.i=@amacapital-net.20150623.gappssmtp.com header.b="bbtke5oP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C49222146E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amacapital.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4A6FA6B0074; Mon, 9 Mar 2020 19:59:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 431936B0075; Mon, 9 Mar 2020 19:59:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2AA346B0078; Mon, 9 Mar 2020 19:59:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0124.hostedemail.com [216.40.44.124]) by kanga.kvack.org (Postfix) with ESMTP id 0CABC6B0074 for ; Mon, 9 Mar 2020 19:59:51 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A052F8140 for ; Mon, 9 Mar 2020 23:59:50 +0000 (UTC) X-FDA: 76577494140.30.burst14_38cf058b61a07 X-HE-Tag: burst14_38cf058b61a07 X-Filterd-Recvd-Size: 7064 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Mon, 9 Mar 2020 23:59:50 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id r7so13565818wro.2 for ; Mon, 09 Mar 2020 16:59:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amacapital-net.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0PcIMvT9Ges60/RCEcv9P9n3V3wEXCUl81zN9+mizqE=; b=bbtke5oP3UVppLVojB1atGqhDMEwwoZMv55eTr8ZnNZrgn0YDHohNAJDV+eqnhKipa 4aN9G0lOac5v18XPEriKg/5/D/0L6WNLOtgk+WPAkKIYHDuW4f7I6O8D05xQO9vTC/CI gvOE7yjLpbAqy2xs5nRCImHAI6w7kO6KCMC/FuGLSpEa/Cn8g0BDVHDycc6ierC/ThYB AZO4cFtCs7Q4J2LJBF7s+XfM3ycj5kw+gP7SQ2AYP7yKhuTikOByPG5P52PIBB6xvitH 8x8bbkXarJqpP2ySfHwYKIFOqsbvw5VjToymh99dZ/T1/NHd5AjR6yR4BXjfAUqy2tR0 52gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0PcIMvT9Ges60/RCEcv9P9n3V3wEXCUl81zN9+mizqE=; b=VJFFW6oZCGJCUBt+JKnKyJpxsbBGPld0KD25NEUGsvtOv8fmY053GJtWhpQukqFosB 738I8/RI46J4WMyX4KdYA3IKitHKOpf23JdflMpRWyXLogT3FqA+hYPx5OdBR411rDwO WjtdgtkBdxKJXJprn6Y0h+kfdpcCjjwCvPYW6yBjIZTG17AnVOB35jNREsaNAEKbR1Z2 zKYvvS7ashpdBM2k3uSCv9Qe2OtvxfEOrdIOAl1oOpBnTX1BwAbJ+CtcTI11FL3mc0Eq 3sDyO0kL/EdS+9axapQIf4tmq1lNDg19yNbA8gzxCT/Xch2AOQJuWNZSBo8/cMoDzM84 0yrA== X-Gm-Message-State: ANhLgQ1DaA2zIhKMtm1/SgrJzo36Rf8SHKfR6PJsOp3DP4IuEqwfBDM1 bKCU4+M0BMm2yGn4uUL4o4aple7RQBMp3ndHnDdZrA== X-Google-Smtp-Source: ADFU+vt2iiohYPhJ2F/0p/lc9pYIMGQoNqww6lZPdFqjkGInjDqr9voSFlL20LHhzIQclFgUMk+afkKjVc/hC1ajvTA= X-Received: by 2002:adf:b641:: with SMTP id i1mr23638629wre.18.1583798389063; Mon, 09 Mar 2020 16:59:49 -0700 (PDT) MIME-Version: 1.0 References: <0088001c-0b12-a7dc-ff2a-9d5c282fa36b@intel.com> <56ab33ac-865b-b37e-75f2-a489424566c3@intel.com> In-Reply-To: From: Andy Lutomirski Date: Mon, 9 Mar 2020 16:59:37 -0700 Message-ID: Subject: Re: [RFC PATCH v9 01/27] Documentation/x86: Add CET description To: "H.J. Lu" Cc: Dave Hansen , Yu-cheng Yu , "the arch/x86 maintainers" , "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , LKML , "open list:DOCUMENTATION" , Linux-MM , linux-arch , Linux API , Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin , x86-patch-review@intel.com Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Mar 9, 2020 at 4:52 PM H.J. Lu wrote: > > On Mon, Mar 9, 2020 at 4:21 PM Dave Hansen wrote: > > > > On 3/9/20 4:11 PM, H.J. Lu wrote: > > > A threaded application is loaded from disk. The object file on disk is > > > either CET enabled or not CET enabled. > > > > Huh. Are you saying that all instructions executed on userspace on > > Linux come off of object files on the disk? That's an interesting > > assertion. You might want to go take a look at the processes on your > > systems. Here's my browser for example: > > > > # for p in $(ps aux | grep chromium | awk '{print $2}' ); do cat > > /proc/$p/maps; done | grep ' r-xp 00000000 00:00 0' > > ... > > 202f00082000-202f000bf000 r-xp 00000000 00:00 0 > > 202f000c2000-202f000c3000 r-xp 00000000 00:00 0 > > 202f00102000-202f00103000 r-xp 00000000 00:00 0 > > 202f00142000-202f00143000 r-xp 00000000 00:00 0 > > 202f00182000-202f001bf000 r-xp 00000000 00:00 0 > > > > Lots of funny looking memory areas which are anonymous and executable! > > Those didn't come off the disk. Same thing in firefox. Weird. Any > > idea what those are? > > > > One guess: https://en.wikipedia.org/wiki/Just-in-time_compilation > > jitted code belongs to a process loaded from disk. Enable CET in > an application which uses JIT engine means to also enable CET in > JIT engine. Take git as an example, "git grep" crashed for me on Tiger > Lake. It turned out that git itself was compiled with -fcf-protection and > git was linked against libpcre2-8.so.0 also compiled with -fcf-protection, > which has a JIT, sljit, which was not CET enabled. git crashed in the > jitted codes due to missing ENDBR. I had to enable CET in sljit to make > git working on CET enabled Tiger Lake. So we need to enable CET in > JIT engine before enabling CET in applications which use JIT engine. This could presumably have been fixed by having libpcre or sljit disable IBT before calling into JIT code or by running the JIT code in another thread. In the other direction, a non-CET libpcre build could build IBT-capable JITted code and enable JIT (by syscall if we allow that or by creating a thread?) when calling it. And IBT has this fancy legacy bitmap to allow non-instrumented code to run with IBT on, although SHSTK doesn't have hardware support for a similar feature. So, sure, the glibc-linked ELF ecosystem needs some degree of CET coordination, but it is absolutely not the case that a process MUST have all CET or no CET. Let's please support the complicated cases in the kernel and the ABI too. If glibc wants to make it annoying to do complicated things, so be it. People work behind glibc's back all the time. --Andy