Re: x86 entry perf unwinding failure (missing IRET_REGS annotation on stack switch?)

From: Peter Zijlstra <peterz@infradead.org>
To: Jann Horn <jannh@google.com>
Cc: Andy Lutomirski <luto@kernel.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
	"H. Peter Anvin" <hpa@zytor.com>,
	the arch/x86 maintainers <x86@kernel.org>,
	kernel list <linux-kernel@vger.kernel.org>,
	Josh Poimboeuf <jpoimboe@redhat.com>
Subject: Re: x86 entry perf unwinding failure (missing IRET_REGS annotation on stack switch?)
Date: Mon, 2 Mar 2020 15:58:22 +0100	[thread overview]
Message-ID: <20200302145822.GC2562@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <CAG48ez1rkN0YU-ieBaUZDKFYG5XFnd7dhDjSDdRmVfWyQzsA5g@mail.gmail.com>

On Sun, Mar 01, 2020 at 07:02:15AM +0100, Jann Horn wrote:

> 0000000000000a2f <swapgs_restore_regs_and_return_to_usermode>:
>      a2f: 41 5f                pop    %r15
> #######sp:sp-8 bp:(und) type:regs end:0
>      a31: 41 5e                pop    %r14
> #######sp:sp-16 bp:(und) type:regs end:0
>      a33: 41 5d                pop    %r13
> #######sp:sp-24 bp:(und) type:regs end:0
>      a35: 41 5c                pop    %r12
> #######sp:sp-32 bp:(und) type:regs end:0
>      a37: 5d                    pop    %rbp
> #######sp:sp-40 bp:(und) type:regs end:0
>      a38: 5b                    pop    %rbx
> #######sp:sp-48 bp:(und) type:regs end:0
>      a39: 41 5b                pop    %r11
> #######sp:sp-56 bp:(und) type:regs end:0
>      a3b: 41 5a                pop    %r10
> #######sp:sp-64 bp:(und) type:regs end:0
>      a3d: 41 59                pop    %r9
> #######sp:sp-72 bp:(und) type:regs end:0
>      a3f: 41 58                pop    %r8
> #######sp:sp-80 bp:(und) type:regs end:0
>      a41: 58                    pop    %rax
> #######sp:sp-88 bp:(und) type:regs end:0
>      a42: 59                    pop    %rcx
> #######sp:sp-96 bp:(und) type:regs end:0
>      a43: 5a                    pop    %rdx
> #######sp:sp-104 bp:(und) type:regs end:0
>      a44: 5e                    pop    %rsi
> #######sp:sp-112 bp:(und) type:regs end:0
>      a45: 48 89 e7              mov    %rsp,%rdi
>      a48: 65 48 8b 24 25 00 00 mov    %gs:0x0,%rsp
>      a4f: 00 00

Right, so here we flip stacks,

>      a51: ff 77 30              pushq  0x30(%rdi)
> #######sp:sp-104 bp:(und) type:regs end:0
>      a54: ff 77 28              pushq  0x28(%rdi)
> #######sp:sp-96 bp:(und) type:regs end:0
>      a57: ff 77 20              pushq  0x20(%rdi)
> #######sp:sp-88 bp:(und) type:regs end:0
>      a5a: ff 77 18              pushq  0x18(%rdi)
> #######sp:sp-80 bp:(und) type:regs end:0
>      a5d: ff 77 10              pushq  0x10(%rdi)

And here we've pushed an IRET frame

> #######sp:sp-72 bp:(und) type:regs end:0
>      a60: ff 37                pushq  (%rdi)

> It looks to me like things go wrong at the point where we switch over
> to the trampoline stack? The ORC info claims that we have full user
> registers on the trampoline stack (and that we're clobbering them with
> our pushes - apparently objtool is not smart enough to realize that
> that looks bogus), but at that point we should probably actually use
> something like UNWIND_HINT_IRET_REGS, right?

I _think_ you've nailed it, but I'm somewhat new to this part of
objtool.

Josh?