Re: [PATCH] kvm: add proper frame pointer logic for vmx

[Qian Cai <cai@lca.pw>]

On 1/15/19 11:34 AM, Sean Christopherson wrote:
> On Tue, Jan 15, 2019 at 08:13:22AM +0100, Paolo Bonzini wrote:
>> On 15/01/19 08:04, Qian Cai wrote:
>>>
>>>
>>> On 1/15/19 1:44 AM, Qian Cai wrote:
>>>> compilation warning since v5.0-rc1,
>>>>
>>>> arch/x86/kvm/vmx/vmx.o: warning: objtool: vmx_vcpu_run.part.17()+0x3171:
>>>> call without frame pointer save/setup
> 
> The warning is complaining about vmx_vcpu_run() in vmx.c, not vmenter.S.
> The rule being "broken" is that a call is made without creating a stack
> frame, and vmx_vmenter() obviously makes no calls.
> 
> E.g., manually running objtool check:
> 
>     $ tools/objtool/objtool check arch/x86/kvm/vmx/vmenter.o
>     $ tools/objtool/objtool check arch/x86/kvm/vmx/vmx.o
>     arch/x86/kvm/vmx/vmx.o: warning: objtool: vmx_vcpu_run.part.19()+0x83e: call without frame pointer save/setup
> 
> I put "broken" in quotes because AFAICT we're not actually violating the
> rule.  From tools/objtool/Documentation/stack-validation.txt:
> 
>    If it's a GCC-compiled .c file, the error may be because the function
>    uses an inline asm() statement which has a "call" instruction.  An
>    asm() statement with a call instruction must declare the use of the
>    stack pointer in its output operand.  On x86_64, this means adding
>    the ASM_CALL_CONSTRAINT as an output constraint:
> 
>      asm volatile("call func" : ASM_CALL_CONSTRAINT);
> 
>    Otherwise the stack frame may not get created before the call.
> 
> 
> The asm() blob that calls vmx_vmenter() uses ASM_CALL_CONSTRAINT, and
> the resulting asm output generates a frame pointer, e.g. this is from
> the vmx.o that objtool warns on:
> 
> Dump of assembler code for function vmx_vcpu_run:
>    0x0000000000007440 <+0>:     e8 00 00 00 00  callq  0x7445 <vmx_vcpu_run+5>
>    0x0000000000007445 <+5>:     55      push   %rbp
>    0x0000000000007446 <+6>:     48 89 e5        mov    %rsp,%rbp
> 
> 
> The warning only shows up in certain configs, e.g. I was only able to
> reproduce this using the .config provided by lkp.  Even explicitly
> enabling CONFIG_FRAME_POINTERS and CONFIG_STACK_VALIDATION didn't
> trigger the warning using my usual config.
> 
> And all that being said, I'm pretty sure this isn't related to the call
> to vmx_vmenter() at all, but rather is something that was exposed by
> removing __noclone from vmx_vcpu_run().
> 
> E.g. I still get the warning if I comment out the call to vmx_vmenter,
> it just shifts to something else (and continues to shift I comment out
> more calls).  The warning goes away if I re-add __noclone, regardless
> of whether or not commit 2bcbd406715d ("Revert "compiler-gcc: disable
> -ftracer for __noclone functions"") is applied.

It complained the call right here at the end of vmx_vcpu_run().

"callq 19eb6" in __read_once_size() via atomic_read()

and then jump back to vmx_vcpu_run() again.

/root/linux-debug/arch/x86/kvm/vmx/vmx.c:6650
}
   19e94:       e8 00 00 00 00          callq  19e99 <vmx_vcpu_run.part.21+0x3159>

__read_once_size():
/root/linux-debug/./include/linux/compiler.h:191
   19e99:       48 c7 c7 00 00 00 00    mov    $0x0,%rdi
   19ea0:       e8 00 00 00 00          callq  19ea5 <vmx_vcpu_run.part.21+0x3165>
   19ea5:       e9 8b dd ff ff          jmpq   17c35 <vmx_vcpu_run.part.21+0xef5>
   19eaa:       48 8b bd 48 ff ff ff    mov    -0xb8(%rbp),%rdi
   19eb1:       e8 00 00 00 00          callq  19eb6 <vmx_vcpu_run.part.21+0x3176>
   19eb6:       e9 b8 df ff ff          jmpq   17e73 <vmx_vcpu_run.part.21+0x1133>

vmx_vcpu_run():
/root/linux-debug/arch/x86/kvm/vmx/vmx.c:6621
        vcpu->arch.regs_dirty = 0;
   19ebb:       48 89 f7                mov    %rsi,%rdi
   19ebe:       e8 00 00 00 00          callq  19ec3 <vmx_vcpu_run.part.21+0x3183>
   19ec3:       e9 f1 e0 ff ff          jmpq   17fb9 <vmx_vcpu_run.part.21+0x1279>