From: Sami Tolvanen <samitolvanen@google.com>
To: Marc Zyngier <maz@kernel.org>
Cc: Fangrui Song <maskray@google.com>,
Nick Desaulniers <ndesaulniers@google.com>,
James Morse <james.morse@arm.com>,
Nathan Chancellor <nathan@kernel.org>,
Kees Cook <keescook@chromium.org>,
Julien Thierry <julien.thierry.kdev@gmail.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
Catalin Marinas <catalin.marinas@arm.com>,
Will Deacon <will@kernel.org>,
linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
kvmarm@lists.cs.columbia.edu, LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] KVM: arm64: Disable LTO in hyp
Date: Thu, 4 Mar 2021 18:38:17 -0800 [thread overview]
Message-ID: <CABCJKudvzBggE7AZQERto5Wo_LoL0G2sNee7_1R7h2TnGhjq8A@mail.gmail.com> (raw)
In-Reply-To: <CABCJKud1EmXmmQj-YOUNCFhE3P1W6Uhqpwe1G0zcR5zw71ksJA@mail.gmail.com>
On Thu, Mar 4, 2021 at 2:34 PM Sami Tolvanen <samitolvanen@google.com> wrote:
>
> On Thu, Mar 4, 2021 at 2:17 PM Marc Zyngier <maz@kernel.org> wrote:
> >
> > On Thu, 04 Mar 2021 21:25:41 +0000,
> > Sami Tolvanen <samitolvanen@google.com> wrote:
> > >
> > > On Thu, Mar 4, 2021 at 11:15 AM Marc Zyngier <maz@kernel.org> wrote:
> > > >
> > > > On Thu, 04 Mar 2021 18:45:44 +0000,
> > > > Sami Tolvanen <samitolvanen@google.com> wrote:
> > > > >
> > > > > allmodconfig + CONFIG_LTO_CLANG_THIN=y fails to build due to following
> > > > > linker errors:
> > > > >
> > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21CC):
> > > >
> > > > I assume this message is only an oddity, right? Because
> > > > __guest_enter() is as far as you can imagine from irqbypass.c...
> > >
> > > I'm not sure what's up with the filename in the error message. Fangrui
> > > or Nick probably have a better idea.
> > >
> > > > > relocation R_AARCH64_CONDBR19 out of range: 2031220 is not in
> > > > > [-1048576, 1048575]; references hyp_panic
> > > > > >>> defined in vmlinux.o
> > > > >
> > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21E0):
> > > > > relocation R_AARCH64_ADR_PREL_LO21 out of range: 2031200 is not in
> > > > > [-1048576, 1048575]; references hyp_panic
> > > > > >>> defined in vmlinux.o
> > > > >
> > > > > As LTO is not really necessary for the hypervisor code, disable it for
> > > > > the hyp directory to fix the build.
> > > >
> > > > Can you shed some light on what the problem is exactly?
> > >
> > > I assume hyp_panic() ends up being placed too far from __guest_enter()
> > > when the kernel is large enough. Possibly something to do with LLVM
> > > always splitting functions into separate sections with LTO. I'm not
> > > sure why the linker cannot shuffle things around to make everyone
> > > happy in this case, but I confirmed that this patch also fixes the
> > > build issue for me:
> > >
> > > diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
> > > index af8e940d0f03..128197b7c794 100644
> > > --- a/arch/arm64/kvm/hyp/vhe/switch.c
> > > +++ b/arch/arm64/kvm/hyp/vhe/switch.c
> > > @@ -214,7 +214,7 @@ static void __hyp_call_panic(u64 spsr, u64 elr, u64 par)
> > > }
> > > NOKPROBE_SYMBOL(__hyp_call_panic);
> > >
> > > -void __noreturn hyp_panic(void)
> > > +void __noreturn hyp_panic(void) __section(".text")
> > > {
> > > u64 spsr = read_sysreg_el2(SYS_SPSR);
> > > u64 elr = read_sysreg_el2(SYS_ELR);
> > >
> >
> > We're getting into black-magic territory here. Why wouldn't hyp_panic
> > be in the .text section already?
>
> It's not quite black magic. LLVM essentially flips on
> -ffunction-sections with LTO and therefore, hyp_panic() will be in
> .text.hyp_panic in vmlinux.o, while __guest_enter() will be in .text.
> Everything ends up in .text when we link vmlinux, of course.
>
> $ readelf --sections vmlinux.o | grep hyp_panic
> [3936] .text.hyp_panic PROGBITS 0000000000000000 004b56e4
Note that disabling LTO here has essentially the same effect as using
__section(".text"). It stops the compiler from splitting these
functions into .text.* sections and makes it less likely that
hyp_panic() ends up too far away from __guest_enter().
If neither of these workarounds sound appealing, I suppose we could
alternatively change hyp/entry.S to use adr_l for hyp_panic. Thoughts?
Sami
WARNING: multiple messages have this Message-ID (diff)
From: Sami Tolvanen <samitolvanen@google.com>
To: Marc Zyngier <maz@kernel.org>
Cc: Kees Cook <keescook@chromium.org>,
Fangrui Song <maskray@google.com>,
Catalin Marinas <catalin.marinas@arm.com>,
Nick Desaulniers <ndesaulniers@google.com>,
LKML <linux-kernel@vger.kernel.org>,
Nathan Chancellor <nathan@kernel.org>,
linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
Will Deacon <will@kernel.org>,
kvmarm@lists.cs.columbia.edu
Subject: Re: [PATCH] KVM: arm64: Disable LTO in hyp
Date: Thu, 4 Mar 2021 18:38:17 -0800 [thread overview]
Message-ID: <CABCJKudvzBggE7AZQERto5Wo_LoL0G2sNee7_1R7h2TnGhjq8A@mail.gmail.com> (raw)
In-Reply-To: <CABCJKud1EmXmmQj-YOUNCFhE3P1W6Uhqpwe1G0zcR5zw71ksJA@mail.gmail.com>
On Thu, Mar 4, 2021 at 2:34 PM Sami Tolvanen <samitolvanen@google.com> wrote:
>
> On Thu, Mar 4, 2021 at 2:17 PM Marc Zyngier <maz@kernel.org> wrote:
> >
> > On Thu, 04 Mar 2021 21:25:41 +0000,
> > Sami Tolvanen <samitolvanen@google.com> wrote:
> > >
> > > On Thu, Mar 4, 2021 at 11:15 AM Marc Zyngier <maz@kernel.org> wrote:
> > > >
> > > > On Thu, 04 Mar 2021 18:45:44 +0000,
> > > > Sami Tolvanen <samitolvanen@google.com> wrote:
> > > > >
> > > > > allmodconfig + CONFIG_LTO_CLANG_THIN=y fails to build due to following
> > > > > linker errors:
> > > > >
> > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21CC):
> > > >
> > > > I assume this message is only an oddity, right? Because
> > > > __guest_enter() is as far as you can imagine from irqbypass.c...
> > >
> > > I'm not sure what's up with the filename in the error message. Fangrui
> > > or Nick probably have a better idea.
> > >
> > > > > relocation R_AARCH64_CONDBR19 out of range: 2031220 is not in
> > > > > [-1048576, 1048575]; references hyp_panic
> > > > > >>> defined in vmlinux.o
> > > > >
> > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21E0):
> > > > > relocation R_AARCH64_ADR_PREL_LO21 out of range: 2031200 is not in
> > > > > [-1048576, 1048575]; references hyp_panic
> > > > > >>> defined in vmlinux.o
> > > > >
> > > > > As LTO is not really necessary for the hypervisor code, disable it for
> > > > > the hyp directory to fix the build.
> > > >
> > > > Can you shed some light on what the problem is exactly?
> > >
> > > I assume hyp_panic() ends up being placed too far from __guest_enter()
> > > when the kernel is large enough. Possibly something to do with LLVM
> > > always splitting functions into separate sections with LTO. I'm not
> > > sure why the linker cannot shuffle things around to make everyone
> > > happy in this case, but I confirmed that this patch also fixes the
> > > build issue for me:
> > >
> > > diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
> > > index af8e940d0f03..128197b7c794 100644
> > > --- a/arch/arm64/kvm/hyp/vhe/switch.c
> > > +++ b/arch/arm64/kvm/hyp/vhe/switch.c
> > > @@ -214,7 +214,7 @@ static void __hyp_call_panic(u64 spsr, u64 elr, u64 par)
> > > }
> > > NOKPROBE_SYMBOL(__hyp_call_panic);
> > >
> > > -void __noreturn hyp_panic(void)
> > > +void __noreturn hyp_panic(void) __section(".text")
> > > {
> > > u64 spsr = read_sysreg_el2(SYS_SPSR);
> > > u64 elr = read_sysreg_el2(SYS_ELR);
> > >
> >
> > We're getting into black-magic territory here. Why wouldn't hyp_panic
> > be in the .text section already?
>
> It's not quite black magic. LLVM essentially flips on
> -ffunction-sections with LTO and therefore, hyp_panic() will be in
> .text.hyp_panic in vmlinux.o, while __guest_enter() will be in .text.
> Everything ends up in .text when we link vmlinux, of course.
>
> $ readelf --sections vmlinux.o | grep hyp_panic
> [3936] .text.hyp_panic PROGBITS 0000000000000000 004b56e4
Note that disabling LTO here has essentially the same effect as using
__section(".text"). It stops the compiler from splitting these
functions into .text.* sections and makes it less likely that
hyp_panic() ends up too far away from __guest_enter().
If neither of these workarounds sound appealing, I suppose we could
alternatively change hyp/entry.S to use adr_l for hyp_panic. Thoughts?
Sami
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
WARNING: multiple messages have this Message-ID (diff)
From: Sami Tolvanen <samitolvanen@google.com>
To: Marc Zyngier <maz@kernel.org>
Cc: Fangrui Song <maskray@google.com>,
Nick Desaulniers <ndesaulniers@google.com>,
James Morse <james.morse@arm.com>,
Nathan Chancellor <nathan@kernel.org>,
Kees Cook <keescook@chromium.org>,
Julien Thierry <julien.thierry.kdev@gmail.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
Catalin Marinas <catalin.marinas@arm.com>,
Will Deacon <will@kernel.org>,
linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
kvmarm@lists.cs.columbia.edu,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] KVM: arm64: Disable LTO in hyp
Date: Thu, 4 Mar 2021 18:38:17 -0800 [thread overview]
Message-ID: <CABCJKudvzBggE7AZQERto5Wo_LoL0G2sNee7_1R7h2TnGhjq8A@mail.gmail.com> (raw)
In-Reply-To: <CABCJKud1EmXmmQj-YOUNCFhE3P1W6Uhqpwe1G0zcR5zw71ksJA@mail.gmail.com>
On Thu, Mar 4, 2021 at 2:34 PM Sami Tolvanen <samitolvanen@google.com> wrote:
>
> On Thu, Mar 4, 2021 at 2:17 PM Marc Zyngier <maz@kernel.org> wrote:
> >
> > On Thu, 04 Mar 2021 21:25:41 +0000,
> > Sami Tolvanen <samitolvanen@google.com> wrote:
> > >
> > > On Thu, Mar 4, 2021 at 11:15 AM Marc Zyngier <maz@kernel.org> wrote:
> > > >
> > > > On Thu, 04 Mar 2021 18:45:44 +0000,
> > > > Sami Tolvanen <samitolvanen@google.com> wrote:
> > > > >
> > > > > allmodconfig + CONFIG_LTO_CLANG_THIN=y fails to build due to following
> > > > > linker errors:
> > > > >
> > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21CC):
> > > >
> > > > I assume this message is only an oddity, right? Because
> > > > __guest_enter() is as far as you can imagine from irqbypass.c...
> > >
> > > I'm not sure what's up with the filename in the error message. Fangrui
> > > or Nick probably have a better idea.
> > >
> > > > > relocation R_AARCH64_CONDBR19 out of range: 2031220 is not in
> > > > > [-1048576, 1048575]; references hyp_panic
> > > > > >>> defined in vmlinux.o
> > > > >
> > > > > ld.lld: error: irqbypass.c:(function __guest_enter: .text+0x21E0):
> > > > > relocation R_AARCH64_ADR_PREL_LO21 out of range: 2031200 is not in
> > > > > [-1048576, 1048575]; references hyp_panic
> > > > > >>> defined in vmlinux.o
> > > > >
> > > > > As LTO is not really necessary for the hypervisor code, disable it for
> > > > > the hyp directory to fix the build.
> > > >
> > > > Can you shed some light on what the problem is exactly?
> > >
> > > I assume hyp_panic() ends up being placed too far from __guest_enter()
> > > when the kernel is large enough. Possibly something to do with LLVM
> > > always splitting functions into separate sections with LTO. I'm not
> > > sure why the linker cannot shuffle things around to make everyone
> > > happy in this case, but I confirmed that this patch also fixes the
> > > build issue for me:
> > >
> > > diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
> > > index af8e940d0f03..128197b7c794 100644
> > > --- a/arch/arm64/kvm/hyp/vhe/switch.c
> > > +++ b/arch/arm64/kvm/hyp/vhe/switch.c
> > > @@ -214,7 +214,7 @@ static void __hyp_call_panic(u64 spsr, u64 elr, u64 par)
> > > }
> > > NOKPROBE_SYMBOL(__hyp_call_panic);
> > >
> > > -void __noreturn hyp_panic(void)
> > > +void __noreturn hyp_panic(void) __section(".text")
> > > {
> > > u64 spsr = read_sysreg_el2(SYS_SPSR);
> > > u64 elr = read_sysreg_el2(SYS_ELR);
> > >
> >
> > We're getting into black-magic territory here. Why wouldn't hyp_panic
> > be in the .text section already?
>
> It's not quite black magic. LLVM essentially flips on
> -ffunction-sections with LTO and therefore, hyp_panic() will be in
> .text.hyp_panic in vmlinux.o, while __guest_enter() will be in .text.
> Everything ends up in .text when we link vmlinux, of course.
>
> $ readelf --sections vmlinux.o | grep hyp_panic
> [3936] .text.hyp_panic PROGBITS 0000000000000000 004b56e4
Note that disabling LTO here has essentially the same effect as using
__section(".text"). It stops the compiler from splitting these
functions into .text.* sections and makes it less likely that
hyp_panic() ends up too far away from __guest_enter().
If neither of these workarounds sound appealing, I suppose we could
alternatively change hyp/entry.S to use adr_l for hyp_panic. Thoughts?
Sami
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-03-05 2:38 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-04 18:45 [PATCH] KVM: arm64: Disable LTO in hyp Sami Tolvanen
2021-03-04 18:45 ` Sami Tolvanen
2021-03-04 18:45 ` Sami Tolvanen
2021-03-04 19:15 ` Marc Zyngier
2021-03-04 19:15 ` Marc Zyngier
2021-03-04 19:15 ` Marc Zyngier
2021-03-04 21:25 ` Sami Tolvanen
2021-03-04 21:25 ` Sami Tolvanen
2021-03-04 21:25 ` Sami Tolvanen
2021-03-04 22:17 ` Marc Zyngier
2021-03-04 22:17 ` Marc Zyngier
2021-03-04 22:17 ` Marc Zyngier
2021-03-04 22:34 ` Sami Tolvanen
2021-03-04 22:34 ` Sami Tolvanen
2021-03-04 22:34 ` Sami Tolvanen
2021-03-05 2:38 ` Sami Tolvanen [this message]
2021-03-05 2:38 ` Sami Tolvanen
2021-03-05 2:38 ` Sami Tolvanen
2021-03-05 11:36 ` Marc Zyngier
2021-03-05 11:36 ` Marc Zyngier
2021-03-05 11:36 ` Marc Zyngier
2021-03-05 14:22 ` Ard Biesheuvel
2021-03-05 14:22 ` Ard Biesheuvel
2021-03-05 14:22 ` Ard Biesheuvel
2021-03-05 16:55 ` Sami Tolvanen
2021-03-05 16:55 ` Sami Tolvanen
2021-03-05 16:55 ` Sami Tolvanen
2021-03-05 17:07 ` Marc Zyngier
2021-03-05 17:07 ` Marc Zyngier
2021-03-05 17:07 ` Marc Zyngier
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CABCJKudvzBggE7AZQERto5Wo_LoL0G2sNee7_1R7h2TnGhjq8A@mail.gmail.com \
--to=samitolvanen@google.com \
--cc=catalin.marinas@arm.com \
--cc=james.morse@arm.com \
--cc=julien.thierry.kdev@gmail.com \
--cc=keescook@chromium.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=maskray@google.com \
--cc=maz@kernel.org \
--cc=nathan@kernel.org \
--cc=ndesaulniers@google.com \
--cc=suzuki.poulose@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.