On Wed, Jan 27, 2021 at 01:54:21PM -0600, Madhavan T. Venkataraman wrote:

Copying in Julien and Josh for the objtool stuff.  I'm replying here but
I'm not 100% up to speed yet so take what I say with a grain of salt.

> FP and no-FP functions
> =====================

This terminology is a bit confusing to me, I keep expanding FP to
floating point here.  Perhaps stack maintanance?

> I have a suggestion for objtool and the unwinder for ARM64.
> 
> IIUC, objtool is responsible for walking all the code paths (except unreachable
> and ignored ones) and making sure that every function has proper frame pointer
> code (prolog, epilog, etc). If a function is found to not have it, the kernel
> build is failed. Is this understanding correct?

Roughly, AIUI.

> If so, can we take a different approach for ARM64?

This feels like something that should just be a stack protector feature
rather than something that an individual architecture should be striking
out on its own with.

> Instead of failing the kernel build, can we just mark the functions as:

> 	FP	Functions that have proper FP code
> 	no-FP	Functions that don't

> May be, we can add an "FP" flag in the symbol table entry for this.

> Then, the unwinder can check the functions it encounters in the stack trace and
> inform the caller if it found any no-FP functions. The caller of the unwinder can
> decide what he wants to do with that information.
> 
> 	- the caller can ignore it
> 
> 	- the caller can print the stack trace with a warning that no-FP functions
> 	  were found
> 
> 	- if the caller is livepatch, the caller can retry until the no-FP functions
> 	  disappear from the stack trace. This way, we can have live patching even
> 	  when some of the functions in the kernel are no-FP.
> 
> Does this make any sense? Is this acceptable? What are the pitfalls?

One downside of this would be that we'd need some way of figuring out if
we've got enough of the kernel covered to be actually useful for
livepatch, and it does mean that we'll have some percentage of code
where the debugging uses for unwinding will be impacted.

It does seem like it's broadly the distinction between the existing
standard and reliable stacktrace interfaces but pushed into the
callback, however reliable stacktrace is going to also perform
additional checks like looking for conditions that might leave things in
an inconsistent state (eg, finding probes or interrupts being taken) so
the distinction isn't just in the binary but also other runtime
conditions.  Reliable stacktrace can also make assumptions that the task
is not currently running unless it is the task doing the trace.  

There was actually until very recently a flag on the callback that is
provided to arch_stack_walk() which indicated if things were supposed to
be reliable but there were no users and nothing ever changed the value,
there was something in a comment or changelog somewhere which said that
this had been for the benefit of printk() but the user had never been
merged.  That sounds a lot like what you're suggesting.

> If we can do this, the unwinder could detect cases such as:

> - If gcc thinks that a function is a leaf function but the function contains
>   inline assembly code that calls another function.

> - If a call to a function bounces through some intermediate code such as a
>   trampoline.

> - etc.

> For specific no-FP functions, the unwinder might be able to deduce the original
> caller. In these cases, the stack trace would still be reliable. For all the others,
> the stack trace would be considered unreliable.

I'm not entirely sure I see the distinction from the current situation
here?

> Compiler instead of objtool
> ===========================
> 
> If the above suggestion is acceptable, I have another suggestion.
> 
> It is a lot of work for every new architecture to add frame pointer verification
> support in objtool. Can we get some help from the compiler?
> 
> The compiler knows which C functions it generates the FP prolog and epilog for. It can
> mark those functions as FP. As for assembly functions, kernel developers could manually
> annotate functions that have proper FP code. The compiler/assembler would mark them
> as FP. Only a small subset of assembly functions would even have FP prolog and epilog.

> Is this acceptable? What are the pitfalls?

If we're trusting the compiler we can probably just do that without any
explicit support from the compiler - it should be doing the standard
stuff unless we explicitly ask it to and if it isn't then it might be a
result of a mismatch in assumptions rather than a deliberate decision to
do something non-standard.  My understanding with objtool is that a big
part of the idea is to provide a static check that the binary we end up
with matches the assumptions that we are making so the fact that it's a
separate implementation is important.

> This can be implemented easily for all architectures for which the compiler generates
> FP code.
> 
> Can this be implemented using a GCC plugin? I know squat about GCC plugins.
> 
> Thanks!
> 
> Madhavan
> 
> 
> 
>