Re: instrumentation, was Re: core dump analysis

[Eero Tamminen <oak@helsinkinet.fi>]

Hi,

On 13.4.2023 2.22, Eero Tamminen wrote:
> On 11.4.2023 11.24, Michael Schmitz wrote:
>> Am 11.04.2023 um 19:19 schrieb Geert Uytterhoeven:
>>> On Tue, Apr 11, 2023 at 6:56 AM Michael Schmitz <schmitzmic@gmail.com>
>>>> Am 11.04.2023 um 12:20 schrieb Finn Thain:
>>>>> A better solution might be be to port the existing instrumentation 
>>>>> like
>>>>> ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder
>>>>> how portable that stuff is.
> 
> Another possibility is just using manually added kernel tracepoints:
> https://www.kernel.org/doc/html/latest/trace/tracepoints.html
> 
> "git grep TRACE_EVENT" did not given any hits for arch/m68k/ though.
> 
> (I don't think those need any particular architecture support. One just 
> adds them to locations where it makes sense to trace activity that is 
> important, but which does not happen too often for the very small 
> overhead from disabled static trace point to be significant.)
> 
> 
>> Looking at a few arch implementations, I'm utterly confused. Wouldn't 
>> know where to start.
> 
> Good article: https://lwn.net/Articles/132196/
> 
> (Ignore jprobes stuff as that's deprecated.)
> 
> 
> Because probe replaces instruction at the probe point with a breakpoint:
> https://www.kernel.org/doc/html/latest/trace/kprobes.html#how-does-a-kprobe-work
> 
> It needs architecture specific code to decode and simulate the replaced 
> instruction (its side-effects) when it is later executed "out-of-line".
> 
> This also causes architecture specific limitations on where the probe 
> can be set.  Architecture may not support simulating all possible 
> instructions, just the most common ones (e.g. ones typically used on 
> function entry and exit points).

These are the kernel symbol entry point instructions that get executed 
when booting Atari Falcon from IDE (using emulator LILO) to Busybox init:
    4524 link.w
       4 movea.l
       3 move.l
       3 clr.l
       2 movem.l
       2 dc.w
       2 btst.b
       1 subq.l
       1 ori.w
       1 move.w
       1 lea.l
       1 jsr
       1 bra.b

For Linus' 6.2 kernel built with this config:
https://git.tuxfamily.org/hatari/hatari.git/tree/tools/linux/kernel.config

Using "gcc-m68k-linux-gnu" (4:10.2.1-1) in Debian Stable.


Count is for how many of the executed kernel symbol addresses had given 
instruction.

I'm a bit surprised how many functions (symbol addresses) start with 
link.w instruction, instead of e.g. move* instruction.

(dc.w is check for emulator NatFeats API.)


	- Eero

PS. Above info is produced by building & booting Linux under Hatari 
emulator as documented here:
https://hatari.tuxfamily.org/doc/m68k-linux.txt

And using:
-------------------
hatari ... --parse sym-breakpoint.ini > instr.txt
-------------------

Where "sym-breakpoint.ini" is:
-------------------
b PConSymbol > 0 :trace :quiet :file pc-disasm.ini
-------------------

And "pc-disasm.ini" file is:
-------------------
d pc-"pc+2"
-------------------

Then to get counts from the output:
-------------------
$ cat  instr.txt | grep -E -e '^\$' | sort -u |\
   cut -b46-52 | sort | uniq -c | sort -nr
-------------------

(Match disassembly lines, drop duplicate function calls, count uniq 
entry point instructions.)


> Besides that, AFAIK plain kprobes support should not have much 
> architecture specific stuff.
> 
> 
> (I have no experience with kernel code, I just remember some LWN 
> articles about kprobes during past 2 decades, and I was involved with a 
> tool that did something similar in user-space, before Linux got uprobes 
> support.)
> 
> 
> 
> As to what architecture to use as an example, I think it would be best 
> to ask tracing maintainers for advice, as there seems to be a lot of 
> code which has been copied from each other at different points in time, 
> that might be nowadays better as shared rather than under each arch...
> 
> These architectures support kprobes:
> ------------------------------------
> $ grep " ok " Documentation/features/debug/kprobes/arch-support.txt
>      |         arc: |  ok  |
>      |         arm: |  ok  |
>      |       arm64: |  ok  |
>      |        csky: |  ok  |
>      |        ia64: |  ok  |
>      |        mips: |  ok  |
>      |      parisc: |  ok  |
>      |     powerpc: |  ok  |
>      |       riscv: |  ok  |
>      |        s390: |  ok  |
>      |          sh: |  ok  |
>      |       sparc: |  ok  |
>      |         x86: |  ok  |
> ------------------------------------
> 
> But only these seem to have probing functionality isolated nicely to its 
> own directory:
> ------------------------------------
> $ ls arch/*/kernel/probes/
> 
> arch/arm64/kernel/probes/:
> decode-insn.c  decode-insn.h  kprobes.c  kprobes_trampoline.S  Makefile 
> simulate-insn.c  simulate-insn.h  uprobes.c
> 
> arch/csky/kernel/probes/:
> decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  kprobes_trampoline.S 
> Makefile  simulate-insn.c  simulate-insn.h  uprobes.c
> 
> arch/riscv/kernel/probes/:
> decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  Makefile  rethook.c 
> rethook.h  rethook_trampoline.S  simulate-insn.c  simulate-insn.h  
> uprobes.c
> ------------------------------------
> 
> 
> As to testing whether kprobes work, that could be easier with ftrace 
> support also present:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/ftrace/README
> 
> 
>      - Eero
> 
> PS. Looking at the kernel features list, m68k arch seems to be missing a 
> lot of other features too:
> https://www.kernel.org/doc/html/latest/m68k/features.html
>