From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB2E51376 for ; Mon, 16 May 2022 19:39:56 +0000 (UTC) Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-2fb9a85a124so161870837b3.13 for ; Mon, 16 May 2022 12:39:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=99BWpQJkm6AArPPk7WzhPOholNHloGIEKeqo/lgHTyY=; b=HKQpIxEMC4GFsVb7vDVuur9BVqohtM/mXn0p1tH0Gcc2mwIz+1Tr22I3UprAMWSABk 6xud03AVA7BWC54cAD97RMffWEB5L35VrUnFHlscjrrErL5W0hPpOCAJcTZL+Y007vUV jdFL+PVaXOmUgIAWJVRG3vPFljY9tUW2cVk9+nJ/5mSJqi/GSAR6t476Q0FiYvI2y42m sZvXIuXVVO4FakgC30CPAMbCdlNMeUf5MKxgMsgUOtKqD62rCULW3pro/bRAgZbvVwXM 0cq3BLN4+NkCWUUcEedGxECqGNRpPkkusw29hef/tZVDtRTdG0FbBo5OE1bTREoX9qYF nAtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=99BWpQJkm6AArPPk7WzhPOholNHloGIEKeqo/lgHTyY=; b=hSVOtr4xCBXY1WmIppAitiPGcslsa+aovtq40fRpQU+rKeVkqN27YlopunhbUjvAwD +H5UQ7CSxDNYojhoHjjSHTSebuxxPXWfRQQZRbjthzgAm4dra6yon39UrW3nr/zmZbr5 cdLEydsAOwWpZUmjATXQ/fo7T4BsHf184UsTnL0ABbZMNPhdmT8B1+O/cyfhSXQ8pKes YJ/XPqQ8OzpvFPaa3sWRh8YKEDHZKuBzlFKqz8D5Vc2+NfNGQZwuUvdHaGI/Fom/Kdai KnP80glV1wdJidJNYgxr+vhUIngcHSztKYM8Xod4+9+1gJSOt6dCupX6A5MWdrDWuJKB VBTQ== X-Gm-Message-State: AOAM5314WR7P/6sa3aXUUUKJPiLttwhsxzt9hK9cwdejbwcycbHVAfrz Roih13yHzuS51A/FBORYWdxAAggvnlFbs9UUzihmbg== X-Google-Smtp-Source: ABdhPJx5GvWkBqAu2Nm1IG231I1soql0HIjYv2G9Ay0REk/aLK0lMgQQ0OdtHS8Nmsne1YoyXIZh5EuOk5GnANhybGQ= X-Received: by 2002:a81:9188:0:b0:2fe:d01a:6b09 with SMTP id i130-20020a819188000000b002fed01a6b09mr12922227ywg.243.1652729995504; Mon, 16 May 2022 12:39:55 -0700 (PDT) Precedence: bulk X-Mailing-List: llvm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20220513202159.1550547-1-samitolvanen@google.com> <20220513202159.1550547-21-samitolvanen@google.com> <20220516183047.GM76023@worktop.programming.kicks-ass.net> In-Reply-To: <20220516183047.GM76023@worktop.programming.kicks-ass.net> From: Sami Tolvanen Date: Mon, 16 May 2022 12:39:19 -0700 Message-ID: Subject: Re: [RFC PATCH v2 20/21] x86: Add support for CONFIG_CFI_CLANG To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Kees Cook , Josh Poimboeuf , x86@kernel.org, Catalin Marinas , Will Deacon , Mark Rutland , Nathan Chancellor , Nick Desaulniers , Joao Moreira , Sedat Dilek , Steven Rostedt , linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, llvm@lists.linux.dev Content-Type: text/plain; charset="UTF-8" On Mon, May 16, 2022 at 11:30 AM Peter Zijlstra wrote: > > On Mon, May 16, 2022 at 10:15:00AM -0700, Sami Tolvanen wrote: > > On Mon, May 16, 2022 at 2:54 AM Peter Zijlstra wrote: > > > > > > On Fri, May 13, 2022 at 01:21:58PM -0700, Sami Tolvanen wrote: > > > > With CONFIG_CFI_CLANG, the compiler injects a type preamble > > > > immediately before each function and a check to validate the target > > > > function type before indirect calls: > > > > > > > > ; type preamble > > > > __cfi_function: > > > > int3 > > > > int3 > > > > mov , %eax > > > > int3 > > > > int3 > > > > function: > > > > ... > > > > > > When I enable CFI_CLANG and X86_KERNEL_IBT I get: > > > > > > 0000000000000c80 <__cfi_io_schedule_timeout>: > > > c80: cc int3 > > > c81: cc int3 > > > c82: b8 b5 b1 39 b3 mov $0xb339b1b5,%eax > > > c87: cc int3 > > > c88: cc int3 > > > > > > 0000000000000c89 : > > > c89: f3 0f 1e fa endbr64 > > > > > > > > > That seems unfortunate. Would it be possible to get an additional > > > compiler option to suppress the endbr for all symbols that get a __cfi_ > > > preaamble? > > > > What's the concern with the endbr? Dropping it would currently break > > the CFI+IBT combination on newer hardware, no? > > Well, yes, but also that combination isn't very interesting. See, > > https://lore.kernel.org/all/20220420004241.2093-1-joao@overdrivepizza.com/T/#m5d67fb010d488b2f8eee33f1eb39d12f769e4ad2 > > and the patch I did down-thread: > > https://lkml.kernel.org/r/YoJKhHluN4n0kZDm@hirez.programming.kicks-ass.net > > If we have IBT, then FineIBT is a much better option than kCFI+IBT. > Removing that superfluous endbr also shrinks the whole thing by 4 bytes. > > So I'm fine with the compiler generating working code for that > combination; but please get me an option to supress it in order to save > those pointless bytes. All this CFI stuff is enough bloat as it is. Sure, I'll take a look at what's the best way to accomplish this. > > > > ; indirect call check > > > > cmpl , -6(%r11) > > > > je .Ltmp1 > > > > ud2 > > > > .Ltmp1: > > > > call __x86_indirect_thunk_r11 > > > > > > The first one I try and find looks like: > > > > > > 26: 41 81 7b fa a6 96 9e 38 cmpl $0x389e96a6,-0x6(%r11) > > > 2e: 74 02 je 32 <__traceiter_sched_kthread_stop+0x29> > > > 30: 0f 0b ud2 > > > 32: 4c 89 f6 mov %r14,%rsi > > > 35: e8 00 00 00 00 call 3a <__traceiter_sched_kthread_stop+0x31> 36: R_X86_64_PLT32 __x86_indirect_thunk_r11-0x4 > > > > > > This must not be. If I'm to rewrite that lot to: > > > > > > movl $\hash, %r10d > > > sub $9, %r11 > > > call *%r11 > > > .nop 4 > > > > > > Then there must not be spurious instruction in between the ud2 and the > > > indirect call/retpoline thing. > > > > With the current compiler patch, LLVM sets up function arguments after > > the CFI check. if it's a problem, we can look into changing that. > > Yes, please fix that. Again see that same patch for why this is a > problem. Objtool can trivially find retpoline calls, but finding this > kCFI gadget is going to be hard work. If you ensure they're > unconditionally stuck together, then the problem goes away find one, > finds the other. You can use .kcfi_traps to locate the check right now, but I agree, it's not quite ideal. Sami From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 32588C433EF for ; Mon, 16 May 2022 19:41:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=G/60tdOgU9tu9+zWLzqVOUs4OANfhYXFp0czZ6Wx8pg=; b=2itWJBQUhsgrPF 1AipeP9mjgrcDbbFpg0BwbIidP8Kzoscd9OvTTVDG1tNc03eKogu7/cPhwY5ziSXYC+cOUp2Duyjv 945uzaXqgREGlVraj0/HR0FdJaBYPea4loMTDS/SgdBzuF9Bm+9Vjvv6WzgLAh9lVwT3zIqg2buzg eV/3Raf5yjPDjmmPR3Q3FNwGBwhzVqqBJqNvOmXqYtcwLseD4gyMYfuYuyquoUjXONtfZ+PPcOq0H W4O/O4Tot7rtZVff4IK/ik6J2b+XzvyY7ipXOUv0VyNyZRvB8e5QXq9Ak11rgu+C9I2ooJNGjTQsk dnO3Zc4YGwp+rMfEl9fw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nqgZW-009qiJ-Ig; Mon, 16 May 2022 19:40:02 +0000 Received: from mail-yw1-x112e.google.com ([2607:f8b0:4864:20::112e]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nqgZS-009qdC-Tw for linux-arm-kernel@lists.infradead.org; Mon, 16 May 2022 19:40:00 +0000 Received: by mail-yw1-x112e.google.com with SMTP id 00721157ae682-2f16645872fso165027457b3.4 for ; Mon, 16 May 2022 12:39:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=99BWpQJkm6AArPPk7WzhPOholNHloGIEKeqo/lgHTyY=; b=HKQpIxEMC4GFsVb7vDVuur9BVqohtM/mXn0p1tH0Gcc2mwIz+1Tr22I3UprAMWSABk 6xud03AVA7BWC54cAD97RMffWEB5L35VrUnFHlscjrrErL5W0hPpOCAJcTZL+Y007vUV jdFL+PVaXOmUgIAWJVRG3vPFljY9tUW2cVk9+nJ/5mSJqi/GSAR6t476Q0FiYvI2y42m sZvXIuXVVO4FakgC30CPAMbCdlNMeUf5MKxgMsgUOtKqD62rCULW3pro/bRAgZbvVwXM 0cq3BLN4+NkCWUUcEedGxECqGNRpPkkusw29hef/tZVDtRTdG0FbBo5OE1bTREoX9qYF nAtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=99BWpQJkm6AArPPk7WzhPOholNHloGIEKeqo/lgHTyY=; b=XP2IVA/FFpG5xfxAkcS4BfZ+XoLF/TCwO8+n2NZBhRgTfP2Ze0cPskZ8S3sWstm7E6 oITUsE8p8iuePqaZ4YBlJ20HlR7amacNuGj+tm1mPeNK2w4ykAwYrTCtaEZ1UH5/45J2 zD84ZwVZkdOEHCbAGEW0c8q4HyCPZzPTztiwdv1Fg/H03p9+g2J4g8p/hH/XYlRhE7uo y0vQowoGN3w6ca67n1B3l2CEL4gkKU6ZCTkaVWU2KYQlXfNMT4ihet2jlawR+f5fOpeb tgwIevYpi5YvdHOTnf3ujJEAnuM/jP4u1X9J2ZyaJFAK0oHm72jeIGxzofNxwlrtVoj4 k6OA== X-Gm-Message-State: AOAM5335EGLDZHdnjrOvIJP9/WAIr3EK16XIqjy8IeqwLfOpKdi56i/S NZ+RFydjgSAKXPzD49bTeUd2xc100WRMxXNzsnDFAw== X-Google-Smtp-Source: ABdhPJx5GvWkBqAu2Nm1IG231I1soql0HIjYv2G9Ay0REk/aLK0lMgQQ0OdtHS8Nmsne1YoyXIZh5EuOk5GnANhybGQ= X-Received: by 2002:a81:9188:0:b0:2fe:d01a:6b09 with SMTP id i130-20020a819188000000b002fed01a6b09mr12922227ywg.243.1652729995504; Mon, 16 May 2022 12:39:55 -0700 (PDT) MIME-Version: 1.0 References: <20220513202159.1550547-1-samitolvanen@google.com> <20220513202159.1550547-21-samitolvanen@google.com> <20220516183047.GM76023@worktop.programming.kicks-ass.net> In-Reply-To: <20220516183047.GM76023@worktop.programming.kicks-ass.net> From: Sami Tolvanen Date: Mon, 16 May 2022 12:39:19 -0700 Message-ID: Subject: Re: [RFC PATCH v2 20/21] x86: Add support for CONFIG_CFI_CLANG To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Kees Cook , Josh Poimboeuf , x86@kernel.org, Catalin Marinas , Will Deacon , Mark Rutland , Nathan Chancellor , Nick Desaulniers , Joao Moreira , Sedat Dilek , Steven Rostedt , linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, llvm@lists.linux.dev X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220516_123959_009004_7A20409C X-CRM114-Status: GOOD ( 32.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, May 16, 2022 at 11:30 AM Peter Zijlstra wrote: > > On Mon, May 16, 2022 at 10:15:00AM -0700, Sami Tolvanen wrote: > > On Mon, May 16, 2022 at 2:54 AM Peter Zijlstra wrote: > > > > > > On Fri, May 13, 2022 at 01:21:58PM -0700, Sami Tolvanen wrote: > > > > With CONFIG_CFI_CLANG, the compiler injects a type preamble > > > > immediately before each function and a check to validate the target > > > > function type before indirect calls: > > > > > > > > ; type preamble > > > > __cfi_function: > > > > int3 > > > > int3 > > > > mov , %eax > > > > int3 > > > > int3 > > > > function: > > > > ... > > > > > > When I enable CFI_CLANG and X86_KERNEL_IBT I get: > > > > > > 0000000000000c80 <__cfi_io_schedule_timeout>: > > > c80: cc int3 > > > c81: cc int3 > > > c82: b8 b5 b1 39 b3 mov $0xb339b1b5,%eax > > > c87: cc int3 > > > c88: cc int3 > > > > > > 0000000000000c89 : > > > c89: f3 0f 1e fa endbr64 > > > > > > > > > That seems unfortunate. Would it be possible to get an additional > > > compiler option to suppress the endbr for all symbols that get a __cfi_ > > > preaamble? > > > > What's the concern with the endbr? Dropping it would currently break > > the CFI+IBT combination on newer hardware, no? > > Well, yes, but also that combination isn't very interesting. See, > > https://lore.kernel.org/all/20220420004241.2093-1-joao@overdrivepizza.com/T/#m5d67fb010d488b2f8eee33f1eb39d12f769e4ad2 > > and the patch I did down-thread: > > https://lkml.kernel.org/r/YoJKhHluN4n0kZDm@hirez.programming.kicks-ass.net > > If we have IBT, then FineIBT is a much better option than kCFI+IBT. > Removing that superfluous endbr also shrinks the whole thing by 4 bytes. > > So I'm fine with the compiler generating working code for that > combination; but please get me an option to supress it in order to save > those pointless bytes. All this CFI stuff is enough bloat as it is. Sure, I'll take a look at what's the best way to accomplish this. > > > > ; indirect call check > > > > cmpl , -6(%r11) > > > > je .Ltmp1 > > > > ud2 > > > > .Ltmp1: > > > > call __x86_indirect_thunk_r11 > > > > > > The first one I try and find looks like: > > > > > > 26: 41 81 7b fa a6 96 9e 38 cmpl $0x389e96a6,-0x6(%r11) > > > 2e: 74 02 je 32 <__traceiter_sched_kthread_stop+0x29> > > > 30: 0f 0b ud2 > > > 32: 4c 89 f6 mov %r14,%rsi > > > 35: e8 00 00 00 00 call 3a <__traceiter_sched_kthread_stop+0x31> 36: R_X86_64_PLT32 __x86_indirect_thunk_r11-0x4 > > > > > > This must not be. If I'm to rewrite that lot to: > > > > > > movl $\hash, %r10d > > > sub $9, %r11 > > > call *%r11 > > > .nop 4 > > > > > > Then there must not be spurious instruction in between the ud2 and the > > > indirect call/retpoline thing. > > > > With the current compiler patch, LLVM sets up function arguments after > > the CFI check. if it's a problem, we can look into changing that. > > Yes, please fix that. Again see that same patch for why this is a > problem. Objtool can trivially find retpoline calls, but finding this > kCFI gadget is going to be hard work. If you ensure they're > unconditionally stuck together, then the problem goes away find one, > finds the other. You can use .kcfi_traps to locate the check right now, but I agree, it's not quite ideal. Sami _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel