From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFF3DC433DB for ; Fri, 5 Mar 2021 02:38:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A3BF564E5F for ; Fri, 5 Mar 2021 02:38:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229516AbhCECib (ORCPT ); Thu, 4 Mar 2021 21:38:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229436AbhCECia (ORCPT ); Thu, 4 Mar 2021 21:38:30 -0500 Received: from mail-ua1-x929.google.com (mail-ua1-x929.google.com [IPv6:2607:f8b0:4864:20::929]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8988CC061574 for ; Thu, 4 Mar 2021 18:38:30 -0800 (PST) Received: by mail-ua1-x929.google.com with SMTP id c44so258559uad.12 for ; Thu, 04 Mar 2021 18:38:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QS2dc4BRJJgr8umUp9EpdfHdtGHxUWTC0395zWXgRQ0=; b=LSH6iwuk+SlxyAgzUKOCThatNBJjTF63wj9pl7/rv3WvWASJdjJ5g08bUmf4+LU4nr hUsYKepzYVnbq2ahvA2yinYFF1yOI+u7nlH+XozKJBVt/hGCDT0EI6B51/JnJ4hrDbXI lcFb/vEF3ijhC35zMtJnWQNY1R0xMoIaRGfErQzQSiAN3iNmMjUjLN14hzICPK81xQCI Pb9/lKaEvaWgRm3xCbQqMpivxmjtVOCDebEA+INWDAkDqUPrMMxty/lQQDEK2+Db3xdC CwiHANsN8spPAlFQkX7yqbhlTWfSY1RcjoyC/niyBpki7rRNL3tK4aZU20flB8KcWp5R C83g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QS2dc4BRJJgr8umUp9EpdfHdtGHxUWTC0395zWXgRQ0=; b=OND6Tmm8zkoxZoFvy4vpgUzGxZNS2m4xLpP6aqqZdu3QTykfmtv+FyCeSo0Cs5WUZC n6bQj9Xihm3AG468RVVMV6ztuv4oiXHprisTh57XzjqQl38Hd3r6KD+Hcek+kNgBJvTj GnDCp5+xrhVJMiMJwZ++cLxZuhAVZ2bz/wQ/hKi4Bxj8TVKQpCR6U+ZSHRW1XCmuIYNT kJwv3GhMvKs4F6y7Gt0igYxEiMRZ+TsihjdLnIR0v3m484pLYJOSPo/FjRZoNpYn1+WZ PsgeFPxa88WLWzD92XNpi9P2YpkqbyJbiGEs/Hoo5ixJMK0w03q3It/S8/4AO4QmvRAn cKUw== X-Gm-Message-State: AOAM533x/1XyVsq3ahQWuScl1PewuFK/W2qEAsiep7jPkYp/NIyV7oyO YyeSfucEKESIdPFGtbu26DJe802Em46aAys/VViiQw== X-Google-Smtp-Source: ABdhPJxMZd/afyVrbW4edXEQyCei1a86gfA8o3+MYDlHubXJYjQzSdCnAYOppQlxL7/Vjn59qH+xxqJciQKFnen8Po0= X-Received: by 2002:a9f:374d:: with SMTP id a13mr4660885uae.122.1614911909135; Thu, 04 Mar 2021 18:38:29 -0800 (PST) MIME-Version: 1.0 References: <20210304184544.2014171-1-samitolvanen@google.com> <87k0qmzq5u.wl-maz@kernel.org> <878s72sgwt.wl-maz@kernel.org> In-Reply-To: From: Sami Tolvanen Date: Thu, 4 Mar 2021 18:38:17 -0800 Message-ID: Subject: Re: [PATCH] KVM: arm64: Disable LTO in hyp To: Marc Zyngier Cc: Fangrui Song , Nick Desaulniers , James Morse , Nathan Chancellor , Kees Cook , Julien Thierry , Suzuki K Poulose , Catalin Marinas , Will Deacon , linux-arm-kernel , kvmarm@lists.cs.columbia.edu, LKML Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Mar 4, 2021 at 2:34 PM Sami Tolvanen wrote: > > On Thu, Mar 4, 2021 at 2:17 PM Marc Zyngier wrote: > > > > On Thu, 04 Mar 2021 21:25:41 +0000, > > Sami Tolvanen wrote: > > > > > > On Thu, Mar 4, 2021 at 11:15 AM Marc Zyngier wrote: > > > > > > > > On Thu, 04 Mar 2021 18:45:44 +0000, > > > > Sami Tolvanen wrote: > > > > > > > > > > allmodconfig + CONFIG_LTO_CLANG_THIN=y fails to build due to following > > > > > linker errors: > > > > > > > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21CC): > > > > > > > > I assume this message is only an oddity, right? Because > > > > __guest_enter() is as far as you can imagine from irqbypass.c... > > > > > > I'm not sure what's up with the filename in the error message. Fangrui > > > or Nick probably have a better idea. > > > > > > > > relocation R_AARCH64_CONDBR19 out of range: 2031220 is not in > > > > > [-1048576, 1048575]; references hyp_panic > > > > > >>> defined in vmlinux.o > > > > > > > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21E0): > > > > > relocation R_AARCH64_ADR_PREL_LO21 out of range: 2031200 is not in > > > > > [-1048576, 1048575]; references hyp_panic > > > > > >>> defined in vmlinux.o > > > > > > > > > > As LTO is not really necessary for the hypervisor code, disable it for > > > > > the hyp directory to fix the build. > > > > > > > > Can you shed some light on what the problem is exactly? > > > > > > I assume hyp_panic() ends up being placed too far from __guest_enter() > > > when the kernel is large enough. Possibly something to do with LLVM > > > always splitting functions into separate sections with LTO. I'm not > > > sure why the linker cannot shuffle things around to make everyone > > > happy in this case, but I confirmed that this patch also fixes the > > > build issue for me: > > > > > > diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c > > > index af8e940d0f03..128197b7c794 100644 > > > --- a/arch/arm64/kvm/hyp/vhe/switch.c > > > +++ b/arch/arm64/kvm/hyp/vhe/switch.c > > > @@ -214,7 +214,7 @@ static void __hyp_call_panic(u64 spsr, u64 elr, u64 par) > > > } > > > NOKPROBE_SYMBOL(__hyp_call_panic); > > > > > > -void __noreturn hyp_panic(void) > > > +void __noreturn hyp_panic(void) __section(".text") > > > { > > > u64 spsr = read_sysreg_el2(SYS_SPSR); > > > u64 elr = read_sysreg_el2(SYS_ELR); > > > > > > > We're getting into black-magic territory here. Why wouldn't hyp_panic > > be in the .text section already? > > It's not quite black magic. LLVM essentially flips on > -ffunction-sections with LTO and therefore, hyp_panic() will be in > .text.hyp_panic in vmlinux.o, while __guest_enter() will be in .text. > Everything ends up in .text when we link vmlinux, of course. > > $ readelf --sections vmlinux.o | grep hyp_panic > [3936] .text.hyp_panic PROGBITS 0000000000000000 004b56e4 Note that disabling LTO here has essentially the same effect as using __section(".text"). It stops the compiler from splitting these functions into .text.* sections and makes it less likely that hyp_panic() ends up too far away from __guest_enter(). If neither of these workarounds sound appealing, I suppose we could alternatively change hyp/entry.S to use adr_l for hyp_panic. Thoughts? Sami From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72FD4C433DB for ; Fri, 5 Mar 2021 18:58:38 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id CA0C1650A3 for ; Fri, 5 Mar 2021 18:58:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA0C1650A3 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 4E9A84B382; Fri, 5 Mar 2021 13:58:37 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id obLxYSCtGvPV; Fri, 5 Mar 2021 13:58:36 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 6B10D4B3D9; Fri, 5 Mar 2021 13:58:34 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 638B64B198 for ; Thu, 4 Mar 2021 21:38:31 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LbuCl-m9RSFB for ; Thu, 4 Mar 2021 21:38:30 -0500 (EST) Received: from mail-ua1-f46.google.com (mail-ua1-f46.google.com [209.85.222.46]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 580934B16B for ; Thu, 4 Mar 2021 21:38:30 -0500 (EST) Received: by mail-ua1-f46.google.com with SMTP id 62so259013uar.13 for ; Thu, 04 Mar 2021 18:38:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QS2dc4BRJJgr8umUp9EpdfHdtGHxUWTC0395zWXgRQ0=; b=LSH6iwuk+SlxyAgzUKOCThatNBJjTF63wj9pl7/rv3WvWASJdjJ5g08bUmf4+LU4nr hUsYKepzYVnbq2ahvA2yinYFF1yOI+u7nlH+XozKJBVt/hGCDT0EI6B51/JnJ4hrDbXI lcFb/vEF3ijhC35zMtJnWQNY1R0xMoIaRGfErQzQSiAN3iNmMjUjLN14hzICPK81xQCI Pb9/lKaEvaWgRm3xCbQqMpivxmjtVOCDebEA+INWDAkDqUPrMMxty/lQQDEK2+Db3xdC CwiHANsN8spPAlFQkX7yqbhlTWfSY1RcjoyC/niyBpki7rRNL3tK4aZU20flB8KcWp5R C83g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QS2dc4BRJJgr8umUp9EpdfHdtGHxUWTC0395zWXgRQ0=; b=ldCxRAJr7amZoILal9jg/C21EiYFUbC5oh2Snqsi/IbT7L3kRDBjGoYQWkBiV4sTsG gFHV22I7rOTnjldI9uuKTfplGmUET91n4r2GGrigmGdarxDB6eEgBaM3ZojFTnbm2yC0 ovnauc43NyWdsmEAqzz+OaBzST+/U6VOyLUI70Fjw6NgtbKa7Hpm9jdf+hieRorSA/nz km8R5AL73MVjxoFj2IQotIdISDTXUXrSDaa41apM1hr/02x6YOILHpOfmRMae1voVMZe JL9BIKaXgqPFOBrZMHqRGzO4JMgSSa/o+yF+F+skL24njRqdtSBWT2Cqrg7Z7rw6Q4TA qTIw== X-Gm-Message-State: AOAM531VD453QjEbJxWoyxgCBwcki83UoLTPupvNFgWpo7pOYXDBX/eq mUNWXxdX20XLgfcdXb5zmSF04c9gluie44tr7mOJZw== X-Google-Smtp-Source: ABdhPJxMZd/afyVrbW4edXEQyCei1a86gfA8o3+MYDlHubXJYjQzSdCnAYOppQlxL7/Vjn59qH+xxqJciQKFnen8Po0= X-Received: by 2002:a9f:374d:: with SMTP id a13mr4660885uae.122.1614911909135; Thu, 04 Mar 2021 18:38:29 -0800 (PST) MIME-Version: 1.0 References: <20210304184544.2014171-1-samitolvanen@google.com> <87k0qmzq5u.wl-maz@kernel.org> <878s72sgwt.wl-maz@kernel.org> In-Reply-To: From: Sami Tolvanen Date: Thu, 4 Mar 2021 18:38:17 -0800 Message-ID: Subject: Re: [PATCH] KVM: arm64: Disable LTO in hyp To: Marc Zyngier X-Mailman-Approved-At: Fri, 05 Mar 2021 13:58:33 -0500 Cc: Kees Cook , Fangrui Song , Catalin Marinas , Nick Desaulniers , LKML , Nathan Chancellor , linux-arm-kernel , Will Deacon , kvmarm@lists.cs.columbia.edu X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Thu, Mar 4, 2021 at 2:34 PM Sami Tolvanen wrote: > > On Thu, Mar 4, 2021 at 2:17 PM Marc Zyngier wrote: > > > > On Thu, 04 Mar 2021 21:25:41 +0000, > > Sami Tolvanen wrote: > > > > > > On Thu, Mar 4, 2021 at 11:15 AM Marc Zyngier wrote: > > > > > > > > On Thu, 04 Mar 2021 18:45:44 +0000, > > > > Sami Tolvanen wrote: > > > > > > > > > > allmodconfig + CONFIG_LTO_CLANG_THIN=y fails to build due to following > > > > > linker errors: > > > > > > > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21CC): > > > > > > > > I assume this message is only an oddity, right? Because > > > > __guest_enter() is as far as you can imagine from irqbypass.c... > > > > > > I'm not sure what's up with the filename in the error message. Fangrui > > > or Nick probably have a better idea. > > > > > > > > relocation R_AARCH64_CONDBR19 out of range: 2031220 is not in > > > > > [-1048576, 1048575]; references hyp_panic > > > > > >>> defined in vmlinux.o > > > > > > > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21E0): > > > > > relocation R_AARCH64_ADR_PREL_LO21 out of range: 2031200 is not in > > > > > [-1048576, 1048575]; references hyp_panic > > > > > >>> defined in vmlinux.o > > > > > > > > > > As LTO is not really necessary for the hypervisor code, disable it for > > > > > the hyp directory to fix the build. > > > > > > > > Can you shed some light on what the problem is exactly? > > > > > > I assume hyp_panic() ends up being placed too far from __guest_enter() > > > when the kernel is large enough. Possibly something to do with LLVM > > > always splitting functions into separate sections with LTO. I'm not > > > sure why the linker cannot shuffle things around to make everyone > > > happy in this case, but I confirmed that this patch also fixes the > > > build issue for me: > > > > > > diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c > > > index af8e940d0f03..128197b7c794 100644 > > > --- a/arch/arm64/kvm/hyp/vhe/switch.c > > > +++ b/arch/arm64/kvm/hyp/vhe/switch.c > > > @@ -214,7 +214,7 @@ static void __hyp_call_panic(u64 spsr, u64 elr, u64 par) > > > } > > > NOKPROBE_SYMBOL(__hyp_call_panic); > > > > > > -void __noreturn hyp_panic(void) > > > +void __noreturn hyp_panic(void) __section(".text") > > > { > > > u64 spsr = read_sysreg_el2(SYS_SPSR); > > > u64 elr = read_sysreg_el2(SYS_ELR); > > > > > > > We're getting into black-magic territory here. Why wouldn't hyp_panic > > be in the .text section already? > > It's not quite black magic. LLVM essentially flips on > -ffunction-sections with LTO and therefore, hyp_panic() will be in > .text.hyp_panic in vmlinux.o, while __guest_enter() will be in .text. > Everything ends up in .text when we link vmlinux, of course. > > $ readelf --sections vmlinux.o | grep hyp_panic > [3936] .text.hyp_panic PROGBITS 0000000000000000 004b56e4 Note that disabling LTO here has essentially the same effect as using __section(".text"). It stops the compiler from splitting these functions into .text.* sections and makes it less likely that hyp_panic() ends up too far away from __guest_enter(). If neither of these workarounds sound appealing, I suppose we could alternatively change hyp/entry.S to use adr_l for hyp_panic. Thoughts? Sami _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03200C433E0 for ; Fri, 5 Mar 2021 02:40:21 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7875A64E60 for ; Fri, 5 Mar 2021 02:40:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7875A64E60 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MFdm8Y95iUx2tXfHHrW9PHgNuzBE+UBuWfhUSFsSuEQ=; b=l4H77Gc08tJBNNOtUIGnrgFzY k4zdcit32DZiiJfDnxBh2n93eIkEHW6lJKMZEjtUr07Uql4N6sxCJrA1o34NGSKpjnNtxBJ0kL95j SE8YLf8x8T5h+3NnSOSl+H17/nSI04Dnp8Jd+VtCznwZWy2+eag09r1jZrOivpv0aTqoQZr4of4O+ 1xYzuxKQIJ0GDdwHZ0RYf0wICyQhDoFm/Tv7eRmJIfy9mKGoeZHMrGI3j9aV5gt7HSEx5cJ1Q9Ac8 Qk2+axMDVvPVrZtZT4zovFv2DUVnyXb1GHwfzp+EVgNK03W7UT89oWrj6eq08b2IVykvlRzBvNfoJ YGP9BKOgA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lI0MT-00Bl9r-Bw; Fri, 05 Mar 2021 02:38:41 +0000 Received: from mail-ua1-x92d.google.com ([2607:f8b0:4864:20::92d]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lI0MN-00Bl3A-HA for linux-arm-kernel@lists.infradead.org; Fri, 05 Mar 2021 02:38:37 +0000 Received: by mail-ua1-x92d.google.com with SMTP id y35so273595uad.5 for ; Thu, 04 Mar 2021 18:38:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QS2dc4BRJJgr8umUp9EpdfHdtGHxUWTC0395zWXgRQ0=; b=LSH6iwuk+SlxyAgzUKOCThatNBJjTF63wj9pl7/rv3WvWASJdjJ5g08bUmf4+LU4nr hUsYKepzYVnbq2ahvA2yinYFF1yOI+u7nlH+XozKJBVt/hGCDT0EI6B51/JnJ4hrDbXI lcFb/vEF3ijhC35zMtJnWQNY1R0xMoIaRGfErQzQSiAN3iNmMjUjLN14hzICPK81xQCI Pb9/lKaEvaWgRm3xCbQqMpivxmjtVOCDebEA+INWDAkDqUPrMMxty/lQQDEK2+Db3xdC CwiHANsN8spPAlFQkX7yqbhlTWfSY1RcjoyC/niyBpki7rRNL3tK4aZU20flB8KcWp5R C83g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QS2dc4BRJJgr8umUp9EpdfHdtGHxUWTC0395zWXgRQ0=; b=lne2mcTMX3X1CayqZSzErsk9CsIMwnPMsey7p33dJejvMIuP3/NCXl2jHvJZCUFVkT O2jzsVkrb7+XNhSDXlpViZFT4Nj+SCjKa6iHwygMPrgJX9NxfCLSscreqn4ED6Nr9QnR Iy8nqJTsfBGC0dfBEVID+pqsQiRILdG0rLJCLAnImuqmm//X7glZbjN79rl+hyqYSH1A 1tQFo+GhUIiyIDAhdCr8v62SxRWWkZBVcXuCt/KHmLncdWCefT+J8deYxv/gweKnilzC uNhZXme9Jc4deT/s9LJSbNRwWtTWxXMZD7/8NKiqiXuVReBRH4LFs3VTnSaVg8vdEhO0 Oa5w== X-Gm-Message-State: AOAM532NdQKcpRDdyPjWooUHQmmngO4NxR0f4ws8pMDxkmJr39Z5auZc R+T1Z9JeGx6t5dDhDqiE3B61qtNgHuq/Dfxgc/UWUg== X-Google-Smtp-Source: ABdhPJxMZd/afyVrbW4edXEQyCei1a86gfA8o3+MYDlHubXJYjQzSdCnAYOppQlxL7/Vjn59qH+xxqJciQKFnen8Po0= X-Received: by 2002:a9f:374d:: with SMTP id a13mr4660885uae.122.1614911909135; Thu, 04 Mar 2021 18:38:29 -0800 (PST) MIME-Version: 1.0 References: <20210304184544.2014171-1-samitolvanen@google.com> <87k0qmzq5u.wl-maz@kernel.org> <878s72sgwt.wl-maz@kernel.org> In-Reply-To: From: Sami Tolvanen Date: Thu, 4 Mar 2021 18:38:17 -0800 Message-ID: Subject: Re: [PATCH] KVM: arm64: Disable LTO in hyp To: Marc Zyngier Cc: Fangrui Song , Nick Desaulniers , James Morse , Nathan Chancellor , Kees Cook , Julien Thierry , Suzuki K Poulose , Catalin Marinas , Will Deacon , linux-arm-kernel , kvmarm@lists.cs.columbia.edu, LKML X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210305_023835_804475_B04E1811 X-CRM114-Status: GOOD ( 30.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Mar 4, 2021 at 2:34 PM Sami Tolvanen wrote: > > On Thu, Mar 4, 2021 at 2:17 PM Marc Zyngier wrote: > > > > On Thu, 04 Mar 2021 21:25:41 +0000, > > Sami Tolvanen wrote: > > > > > > On Thu, Mar 4, 2021 at 11:15 AM Marc Zyngier wrote: > > > > > > > > On Thu, 04 Mar 2021 18:45:44 +0000, > > > > Sami Tolvanen wrote: > > > > > > > > > > allmodconfig + CONFIG_LTO_CLANG_THIN=y fails to build due to following > > > > > linker errors: > > > > > > > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21CC): > > > > > > > > I assume this message is only an oddity, right? Because > > > > __guest_enter() is as far as you can imagine from irqbypass.c... > > > > > > I'm not sure what's up with the filename in the error message. Fangrui > > > or Nick probably have a better idea. > > > > > > > > relocation R_AARCH64_CONDBR19 out of range: 2031220 is not in > > > > > [-1048576, 1048575]; references hyp_panic > > > > > >>> defined in vmlinux.o > > > > > > > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21E0): > > > > > relocation R_AARCH64_ADR_PREL_LO21 out of range: 2031200 is not in > > > > > [-1048576, 1048575]; references hyp_panic > > > > > >>> defined in vmlinux.o > > > > > > > > > > As LTO is not really necessary for the hypervisor code, disable it for > > > > > the hyp directory to fix the build. > > > > > > > > Can you shed some light on what the problem is exactly? > > > > > > I assume hyp_panic() ends up being placed too far from __guest_enter() > > > when the kernel is large enough. Possibly something to do with LLVM > > > always splitting functions into separate sections with LTO. I'm not > > > sure why the linker cannot shuffle things around to make everyone > > > happy in this case, but I confirmed that this patch also fixes the > > > build issue for me: > > > > > > diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c > > > index af8e940d0f03..128197b7c794 100644 > > > --- a/arch/arm64/kvm/hyp/vhe/switch.c > > > +++ b/arch/arm64/kvm/hyp/vhe/switch.c > > > @@ -214,7 +214,7 @@ static void __hyp_call_panic(u64 spsr, u64 elr, u64 par) > > > } > > > NOKPROBE_SYMBOL(__hyp_call_panic); > > > > > > -void __noreturn hyp_panic(void) > > > +void __noreturn hyp_panic(void) __section(".text") > > > { > > > u64 spsr = read_sysreg_el2(SYS_SPSR); > > > u64 elr = read_sysreg_el2(SYS_ELR); > > > > > > > We're getting into black-magic territory here. Why wouldn't hyp_panic > > be in the .text section already? > > It's not quite black magic. LLVM essentially flips on > -ffunction-sections with LTO and therefore, hyp_panic() will be in > .text.hyp_panic in vmlinux.o, while __guest_enter() will be in .text. > Everything ends up in .text when we link vmlinux, of course. > > $ readelf --sections vmlinux.o | grep hyp_panic > [3936] .text.hyp_panic PROGBITS 0000000000000000 004b56e4 Note that disabling LTO here has essentially the same effect as using __section(".text"). It stops the compiler from splitting these functions into .text.* sections and makes it less likely that hyp_panic() ends up too far away from __guest_enter(). If neither of these workarounds sound appealing, I suppose we could alternatively change hyp/entry.S to use adr_l for hyp_panic. Thoughts? Sami _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel