From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59CF1C43381 for ; Tue, 2 Feb 2021 00:03:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2BCDC64ECE for ; Tue, 2 Feb 2021 00:03:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231261AbhBBADq (ORCPT ); Mon, 1 Feb 2021 19:03:46 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:36410 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231219AbhBBADo (ORCPT ); Mon, 1 Feb 2021 19:03:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612224137; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xMnypghb7eobehgvuvyvI6ftY8R6Uotx581HxzZ88OM=; b=PH3/2qYJi+892n3f4Kl1DvlRZOJ0ysXroiLmgij1NutsFKr10VSRkIdnomTU4Q3VCY6VSq +dXFlMBvdNhTIg0LCxx+46B5Tpru6cHjz0q+r+l54SjxGG7SotPXFUkvak/bZ2n9U8yKJN WpkrPrkxy7IBJoggWcXSBR5rqStfnbY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-507-PjoyMbgUNiSDiR5nWhS47g-1; Mon, 01 Feb 2021 19:02:10 -0500 X-MC-Unique: PjoyMbgUNiSDiR5nWhS47g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9FE43107ACE6; Tue, 2 Feb 2021 00:02:07 +0000 (UTC) Received: from treble (ovpn-120-118.rdu2.redhat.com [10.10.120.118]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4250E5D9CA; Tue, 2 Feb 2021 00:02:05 +0000 (UTC) Date: Mon, 1 Feb 2021 18:02:03 -0600 From: Josh Poimboeuf To: Nick Desaulniers Cc: Julien Thierry , Ard Biesheuvel , Mark Brown , Catalin Marinas , Kees Cook , Linux ARM , linux-efi , linux-hardening@vger.kernel.org, LKML , Mark Rutland , Masahiro Yamada , Michal Marek , Peter Zijlstra , raphael.gault@arm.com, Will Deacon , clang-built-linux , Bill Wendling Subject: Re: [RFC PATCH 12/17] gcc-plugins: objtool: Add plugin to detect switch table on arm64 Message-ID: <20210202000203.rk7lh5mx4aflgkwr@treble> References: <20210120173800.1660730-13-jthierry@redhat.com> <20210127221557.1119744-1-ndesaulniers@google.com> <20210127232651.rj3mo7c2oqh4ytsr@treble> <20210201214423.dhsma73k7ccscovm@treble> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Feb 01, 2021 at 03:17:40PM -0800, Nick Desaulniers wrote: > > > > > Rather than tightly couple this feature to a particular toolchain via > > > > > plugin, it might be nice to consider what features could be spec'ed out > > > > > for toolchains to implement (perhaps via a -f flag). > > > > > > > > The problem is being able to detect switch statement jump table vectors. > > > > > > > > For a given indirect branch (due to a switch statement), what are all > > > > the corresponding jump targets? > > > > > > > > We would need the compiler to annotate that information somehow. > > > > > > Makes sense, the compiler should have this information. How is this > > > problem solved on x86? > > > > Thus far we've been able to successfully reverse engineer it on x86, > > though it hasn't been easy. > > > > There were some particulars for arm64 which made doing so impossible. > > (I don't remember the details.) > > I think the details are pertinent to finding a portable solution. The > commit message of this commit in particular doesn't document such > details, such as why such an approach is necessary or how the data is > laid out for objtool to consume it. Agreed, the commit message needs more details. > > > > > Distributions (like Android, CrOS) wont be able to use such a feature as > > > > > is. > > > > > > > > Would a Clang plugin be out of the question? > > > > > > Generally, we frown on out of tree kernel modules for a couple reasons. > > > > > > Maintaining ABI compatibility when the core kernel changes is > > > generally not instantaneous; someone has to notice the ABI has changed > > > which will be more delayed than if the module was in tree. Worse is > > > when semantics subtly change. While we must not break userspace, we > > > provide no such guarantees within the kernel proper. > > > > > > Also, it's less likely that out of tree kernel modules have been > > > reviewed by kernel developers. They may not have the same quality, > > > use the recommended interfaces, follow coding conventions, etc.. > > > > > > Oh, did I say "out of tree kernel modules?" I meant "compiler > > > plugins." But it's two different sides of the same coin to me. > > > > I thought Android already relied on OOT modules. > > Android Common Kernel does not *rely* on OOT modules or compiler > plugins. Android doesn't forbid vendors from using OOT modules, > though, just as the mainline kernel does not prevent modules from > being built out of tree, or users from insmod'ing them. It's > certainly a pragmatic approach; idealism doesn't work for all OEMs. No judgement, RHEL is similar. Linux is nothing if not pragmatic. > Personally, I lean more towards idealistic; I prefer in-tree modules, > dislike compiler plugins (for much the same reasons), and idealize > loose coupling of software components. This series looks to me like it > tightly couples arm64 live patching to objtool and GCC. > > On the earlier thread, Julien writes: > > >> I think most people interested in livepatching are using GCC built > >> kernels, but I could be mistaken (althought in the long run, both > >> compilers should be supported, and yes, I realize the objtool solution > >> currently only would support GCC). > > Google's production kernels are using livepatching and are built with > Clang. Getting similar functionality working for arm64 would be of > interest. Well, that's cool. I had no idea. I'm curious how they're generating livepatch modules? Because kpatch-build doesn't support Clang (AFAIK), and if they're not using kpatch-build then there are some traps to look out for. > > GCC plugins generally enforce the exact same GCC version for OOT > > modules. So there's no ABI to worry about. I assume Clang does the > > same? > > > > Or did I miss your point? > > I think so. It would seem that the current solution for stack > validation would require multiple different compiler plugins to work > reliably. If jump tables are an issue, I don't see why you wouldn't > pursue a more portable option like -fno-jump-tables first, then work > to claw back any performance loss from that, if relevant? Then > there's no complaint about toolchain specific implementations or tight > coupling. > > Objtool support on arm64 is interesting to me though, because it has > found bugs in LLVM codegen. That alone is extremely valuable. But not > it's not helpful if it's predicated or tightly coupled to GCC, as this > series appears to do. I agree 100%, if there are actual Clang livepatch users (which it sounds like there are) then we should target both compilers. And yes, objtool has been pretty good at finding compiler bugs, so the more coverage the better. I like the -fno-jump-tables suggestion, assuming it doesn't affect performance significantly. > The idea of rebuilding control flow from binary analysis and using > that to find codegen bugs is a really cool idea (novel, even? idk), > and I wish we had some analog for userspace binaries that could > perform similar checks. Objtool is generic in many ways -- in fact I recently heard from a PhD candidate who used it successfully on another kernel for an ORC unwinder. It could probably be used on user space without much effort. That was an early original stated goal but I definitely don't have the bandwidth or incentive to work on it. > > > FWIW, I think the approach taken by -mstack-protector-guard-reg= is a > > > useful case study. It was prototyped as a GCC extension, then added > > > to GCC proper, then added to LLVM (currently only x86, but most of the > > > machinery is in place in the compiler to get it running on arm64). My > > > recommendation is to skip the plugin part and work on a standard > > > interface for compilers to implement, with input from compiler > > > developers. > > > > I like the idea. Is there a recommended forum for such discussions? > > Just an email to GCC/Clang development lists? > > linux-toolchains@vger.kernel.org is probably a good start. Thanks, I'll write something up (maybe a more specific proposal) and post it. -- Josh From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4481CC433DB for ; Tue, 2 Feb 2021 00:03:43 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D09F864EDC for ; Tue, 2 Feb 2021 00:03:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D09F864EDC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=z0E1x5wkC3iJGFL8/2TaIfwdjblqZ8f9f32AEKj5bso=; b=zh6yVIld0j6v+vzLMvOlgBjY2 8kwODjUQg1nZQ3OyAEMdousgLX+lPMAZT6CqOBxcgfiWSj0jmNF7XrrlI/y5A7LaSpYcDp4KNWsVe Ae0yMcKR1XsXGfvDsNlwXn77MrXCjQvxS1R5EvQIzlRO4KkfBlKJ4DYCSQ08H7qs5q55+ZJgPcnB7 AbnPhrUHEHnawg8qq2ovtGWJ6MIQTr3ogM5TkQEol3p75z6f1Bw40Fp9OBOdlismi+gDNjZl8WaAm /a7EPVSRC7gHpGHTS6p5XlPK045owbXqTc2FJHI5Mt2AiIuwvbflAT5tpLOF2qFcUwL2iVzjMyJta 3GQez6o6g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6j98-0004Cp-QS; Tue, 02 Feb 2021 00:02:18 +0000 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6j95-0004CM-Ry for linux-arm-kernel@lists.infradead.org; Tue, 02 Feb 2021 00:02:16 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612224135; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xMnypghb7eobehgvuvyvI6ftY8R6Uotx581HxzZ88OM=; b=Lf+nmG/lGzFzImzYqhu2vqC7GH2JyW4fgEvgb5SQ6YBXgpEYo8CaoZZEKdl8aKI++uA76Z 9Sa9WMIXUz8/PJK4IBFh1nLR1W4hNHsmo8jfz4dBM52ZIUJaUpqXk7cljOhjmwI0u+QEAk 27WA9dgJSoG+pjB1dFxdEAczMrFDbgI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-507-PjoyMbgUNiSDiR5nWhS47g-1; Mon, 01 Feb 2021 19:02:10 -0500 X-MC-Unique: PjoyMbgUNiSDiR5nWhS47g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9FE43107ACE6; Tue, 2 Feb 2021 00:02:07 +0000 (UTC) Received: from treble (ovpn-120-118.rdu2.redhat.com [10.10.120.118]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4250E5D9CA; Tue, 2 Feb 2021 00:02:05 +0000 (UTC) Date: Mon, 1 Feb 2021 18:02:03 -0600 From: Josh Poimboeuf To: Nick Desaulniers Subject: Re: [RFC PATCH 12/17] gcc-plugins: objtool: Add plugin to detect switch table on arm64 Message-ID: <20210202000203.rk7lh5mx4aflgkwr@treble> References: <20210120173800.1660730-13-jthierry@redhat.com> <20210127221557.1119744-1-ndesaulniers@google.com> <20210127232651.rj3mo7c2oqh4ytsr@treble> <20210201214423.dhsma73k7ccscovm@treble> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_190215_947952_F2F950DA X-CRM114-Status: GOOD ( 49.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Bill Wendling , linux-efi , Julien Thierry , clang-built-linux , Peter Zijlstra , Catalin Marinas , Masahiro Yamada , LKML , Michal Marek , raphael.gault@arm.com, Mark Brown , linux-hardening@vger.kernel.org, Will Deacon , Ard Biesheuvel , Linux ARM , Kees Cook Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Feb 01, 2021 at 03:17:40PM -0800, Nick Desaulniers wrote: > > > > > Rather than tightly couple this feature to a particular toolchain via > > > > > plugin, it might be nice to consider what features could be spec'ed out > > > > > for toolchains to implement (perhaps via a -f flag). > > > > > > > > The problem is being able to detect switch statement jump table vectors. > > > > > > > > For a given indirect branch (due to a switch statement), what are all > > > > the corresponding jump targets? > > > > > > > > We would need the compiler to annotate that information somehow. > > > > > > Makes sense, the compiler should have this information. How is this > > > problem solved on x86? > > > > Thus far we've been able to successfully reverse engineer it on x86, > > though it hasn't been easy. > > > > There were some particulars for arm64 which made doing so impossible. > > (I don't remember the details.) > > I think the details are pertinent to finding a portable solution. The > commit message of this commit in particular doesn't document such > details, such as why such an approach is necessary or how the data is > laid out for objtool to consume it. Agreed, the commit message needs more details. > > > > > Distributions (like Android, CrOS) wont be able to use such a feature as > > > > > is. > > > > > > > > Would a Clang plugin be out of the question? > > > > > > Generally, we frown on out of tree kernel modules for a couple reasons. > > > > > > Maintaining ABI compatibility when the core kernel changes is > > > generally not instantaneous; someone has to notice the ABI has changed > > > which will be more delayed than if the module was in tree. Worse is > > > when semantics subtly change. While we must not break userspace, we > > > provide no such guarantees within the kernel proper. > > > > > > Also, it's less likely that out of tree kernel modules have been > > > reviewed by kernel developers. They may not have the same quality, > > > use the recommended interfaces, follow coding conventions, etc.. > > > > > > Oh, did I say "out of tree kernel modules?" I meant "compiler > > > plugins." But it's two different sides of the same coin to me. > > > > I thought Android already relied on OOT modules. > > Android Common Kernel does not *rely* on OOT modules or compiler > plugins. Android doesn't forbid vendors from using OOT modules, > though, just as the mainline kernel does not prevent modules from > being built out of tree, or users from insmod'ing them. It's > certainly a pragmatic approach; idealism doesn't work for all OEMs. No judgement, RHEL is similar. Linux is nothing if not pragmatic. > Personally, I lean more towards idealistic; I prefer in-tree modules, > dislike compiler plugins (for much the same reasons), and idealize > loose coupling of software components. This series looks to me like it > tightly couples arm64 live patching to objtool and GCC. > > On the earlier thread, Julien writes: > > >> I think most people interested in livepatching are using GCC built > >> kernels, but I could be mistaken (althought in the long run, both > >> compilers should be supported, and yes, I realize the objtool solution > >> currently only would support GCC). > > Google's production kernels are using livepatching and are built with > Clang. Getting similar functionality working for arm64 would be of > interest. Well, that's cool. I had no idea. I'm curious how they're generating livepatch modules? Because kpatch-build doesn't support Clang (AFAIK), and if they're not using kpatch-build then there are some traps to look out for. > > GCC plugins generally enforce the exact same GCC version for OOT > > modules. So there's no ABI to worry about. I assume Clang does the > > same? > > > > Or did I miss your point? > > I think so. It would seem that the current solution for stack > validation would require multiple different compiler plugins to work > reliably. If jump tables are an issue, I don't see why you wouldn't > pursue a more portable option like -fno-jump-tables first, then work > to claw back any performance loss from that, if relevant? Then > there's no complaint about toolchain specific implementations or tight > coupling. > > Objtool support on arm64 is interesting to me though, because it has > found bugs in LLVM codegen. That alone is extremely valuable. But not > it's not helpful if it's predicated or tightly coupled to GCC, as this > series appears to do. I agree 100%, if there are actual Clang livepatch users (which it sounds like there are) then we should target both compilers. And yes, objtool has been pretty good at finding compiler bugs, so the more coverage the better. I like the -fno-jump-tables suggestion, assuming it doesn't affect performance significantly. > The idea of rebuilding control flow from binary analysis and using > that to find codegen bugs is a really cool idea (novel, even? idk), > and I wish we had some analog for userspace binaries that could > perform similar checks. Objtool is generic in many ways -- in fact I recently heard from a PhD candidate who used it successfully on another kernel for an ORC unwinder. It could probably be used on user space without much effort. That was an early original stated goal but I definitely don't have the bandwidth or incentive to work on it. > > > FWIW, I think the approach taken by -mstack-protector-guard-reg= is a > > > useful case study. It was prototyped as a GCC extension, then added > > > to GCC proper, then added to LLVM (currently only x86, but most of the > > > machinery is in place in the compiler to get it running on arm64). My > > > recommendation is to skip the plugin part and work on a standard > > > interface for compilers to implement, with input from compiler > > > developers. > > > > I like the idea. Is there a recommended forum for such discussions? > > Just an email to GCC/Clang development lists? > > linux-toolchains@vger.kernel.org is probably a good start. Thanks, I'll write something up (maybe a more specific proposal) and post it. -- Josh _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel