Re: FAILED unresolved symbol vfs_truncate on arm64 with LLVM

[Nathan Chancellor <nathan@kernel.org>]

On Tue, Feb 09, 2021 at 12:09:31PM -0800, Nick Desaulniers wrote:
> On Tue, Feb 9, 2021 at 11:06 AM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > On Tue, Feb 09, 2021 at 05:13:42PM +0100, Jiri Olsa wrote:
> > > On Tue, Feb 09, 2021 at 04:09:36PM +0100, Jiri Olsa wrote:
> > >
> > > SNIP
> > >
> > > > > > > >                 DW_AT_prototyped        (true)
> > > > > > > >                 DW_AT_type      (0x01cfdfe4 "long int")
> > > > > > > >                 DW_AT_external  (true)
> > > > > > > >
> > > > > > >
> > > > > > > Ok, the problem appears to be not in DWARF, but in mcount_loc data.
> > > > > > > vfs_truncate's address is not recorded as ftrace-attachable, and thus
> > > > > > > pahole ignores it. I don't know why this happens and it's quite
> > > > > > > strange, given vfs_truncate is just a normal global function.
> > > > >
> > > > > right, I can't see it in mcount adresses.. but it begins with instructions
> > > > > that appears to be nops, which would suggest it's traceable
> > > > >
> > > > >   ffff80001031f430 <vfs_truncate>:
> > > > >   ffff80001031f430: 5f 24 03 d5   hint    #34
> > > > >   ffff80001031f434: 1f 20 03 d5   nop
> > > > >   ffff80001031f438: 1f 20 03 d5   nop
> > > > >   ffff80001031f43c: 3f 23 03 d5   hint    #25
> > > > >
> > > > > > >
> > > > > > > I'd like to understand this issue before we try to fix it, but there
> > > > > > > is at least one improvement we can make: pahole should check ftrace
> > > > > > > addresses only for static functions, not the global ones (global ones
> > > > > > > should be always attachable, unless they are special, e.g., notrace
> > > > > > > and stuff). We can easily check that by looking at the corresponding
> > > > > > > symbol. But I'd like to verify that vfs_truncate is ftrace-attachable
> > > >
> > > > I'm still trying to build the kernel.. however ;-)
> > >
> > > I finally reproduced.. however arm's not using mcount_loc
> > > but some other special section.. so it's new mess for me
> >
> > so ftrace data actualy has vfs_truncate address but with extra 4 bytes:
> >
> >         ffff80001031f434
> >
> > real vfs_truncate address:
> >
> >         ffff80001031f430 g     F .text  0000000000000168 vfs_truncate
> >
> > vfs_truncate disasm:
> >
> >         ffff80001031f430 <vfs_truncate>:
> >         ffff80001031f430: 5f 24 03 d5   hint    #34
> >         ffff80001031f434: 1f 20 03 d5   nop
> >         ffff80001031f438: 1f 20 03 d5   nop
> >         ffff80001031f43c: 3f 23 03 d5   hint    #25
> >
> > thats why we don't match it in pahole.. I checked few other functions
> > and some have the same problem and some match the function boundary
> >
> > those that match don't have that first hint instrucion, like:
> >
> >         ffff800010321e40 <do_faccessat>:
> >         ffff800010321e40: 1f 20 03 d5   nop
> >         ffff800010321e44: 1f 20 03 d5   nop
> >         ffff800010321e48: 3f 23 03 d5   hint    #25
> >
> > any hints about hint instructions? ;-)
> 
> aarch64 makes *some* newer instructions reuse the "hint" ie "nop"
> encoding space to make software backwards compatible on older hardware
> that doesn't support such instructions.  Is this BTI, perhaps? (The
> function is perhaps the destination of an indirect call?)

It seems like it. The issue is not reproducible when
CONFIG_ARM64_BTI_KERNEL is not set.

Cheers,
Nathan