arm64 execve/clone sys_exit tracepoints

arm64 execve/clone sys_exit tracepoints

[Federico Di Pierro]

Hi everyone,

While testing Falco on arm64 my team and I encountered some weird
issues; basically, it seems like execve() exit tracepoint is never
called.
Moreover, the clone() exit tracepoint referred to the child process is
also missing.
The issue is present on both the kmod and eBPF probe.

I tested on amznlinux2 with kernel 5.10.96-90.460.amzn2.aarch64, but
other team members tested on other kernel versions too (down to
4.14.X).
I was also able to reproduce the problem using bpftrace tool: hooking
on tracepoint:syscalls:sys_exit_execve; no event is received:

bpftrace -e 'tracepoint:syscalls:sys_exit_execve { printf("execve!\n"); }'

Since sys_enter tracepoints are indeed called, we'd expect the
sys_exit ones to be called too, just like it happens on x86.
The question is: are we missing anything obvious here?

Thank you very much for your time,
Regards
Federico

Re: arm64 execve/clone sys_exit tracepoints

[Mathieu Desnoyers]

----- On Mar 8, 2022, at 5:11 AM, Federico Di Pierro nierro92@gmail.com wrote:

> Hi everyone,
> 
> While testing Falco on arm64 my team and I encountered some weird
> issues; basically, it seems like execve() exit tracepoint is never
> called.
> Moreover, the clone() exit tracepoint referred to the child process is
> also missing.
> The issue is present on both the kmod and eBPF probe.
> 
> I tested on amznlinux2 with kernel 5.10.96-90.460.amzn2.aarch64, but
> other team members tested on other kernel versions too (down to
> 4.14.X).
> I was also able to reproduce the problem using bpftrace tool: hooking
> on tracepoint:syscalls:sys_exit_execve; no event is received:
> 
> bpftrace -e 'tracepoint:syscalls:sys_exit_execve { printf("execve!\n"); }'
> 
> Since sys_enter tracepoints are indeed called, we'd expect the
> sys_exit ones to be called too, just like it happens on x86.
> The question is: are we missing anything obvious here?

I'm not sure about your clone issue, but wrt execve, I know there
may be some discrepancy when exec turns a non-compat executable into
a compat one and vice-versa. Do you exec a binary with a different
bitness, and therefore a different syscall table, on return from exec ?

Thanks,

Mathieu


> 
> Thank you very much for your time,
> Regards
> Federico

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: arm64 execve/clone sys_exit tracepoints

[Federico Di Pierro]

Sorry for the late reply, I had a (very nice!) laser eye surgery last week :)

> I'm not sure about your clone issue, but wrt execve, I know there
> may be some discrepancy when exec turns a non-compat executable into
> a compat one and vice-versa. Do you exec a binary with a different
> bitness, and therefore a different syscall table, on return from exec ?

I double checked, and it seems like we are calling normal 64bit executables.
To test, I just used a simple C script that when run without arguments,
calls itself with execve + 1 argument (to avoid the recursion),
therefore a 64bit executable that calls another 64bit executable.
Unfortunately the issue is still present.
Thanks for the hint though, it was very helpful!

Federico

Il giorno mar 8 mar 2022 alle ore 16:01 Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> ha scritto:
>
> ----- On Mar 8, 2022, at 5:11 AM, Federico Di Pierro nierro92@gmail.com wrote:
>
> > Hi everyone,
> >
> > While testing Falco on arm64 my team and I encountered some weird
> > issues; basically, it seems like execve() exit tracepoint is never
> > called.
> > Moreover, the clone() exit tracepoint referred to the child process is
> > also missing.
> > The issue is present on both the kmod and eBPF probe.
> >
> > I tested on amznlinux2 with kernel 5.10.96-90.460.amzn2.aarch64, but
> > other team members tested on other kernel versions too (down to
> > 4.14.X).
> > I was also able to reproduce the problem using bpftrace tool: hooking
> > on tracepoint:syscalls:sys_exit_execve; no event is received:
> >
> > bpftrace -e 'tracepoint:syscalls:sys_exit_execve { printf("execve!\n"); }'
> >
> > Since sys_enter tracepoints are indeed called, we'd expect the
> > sys_exit ones to be called too, just like it happens on x86.
> > The question is: are we missing anything obvious here?
>
> I'm not sure about your clone issue, but wrt execve, I know there
> may be some discrepancy when exec turns a non-compat executable into
> a compat one and vice-versa. Do you exec a binary with a different
> bitness, and therefore a different syscall table, on return from exec ?
>
> Thanks,
>
> Mathieu
>
>
> >
> > Thank you very much for your time,
> > Regards
> > Federico
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

Re: arm64 execve/clone sys_exit tracepoints

[Federico Di Pierro]

Hi!

After a private conversation with an Arm engineer,
it seems like there is no real "correct" behavior, i mean there is no
default behavior among different architectures.

In the end, arm64 shows just a different behavior from x86_64. Well,
it seems like x86_64 is doing something
different from other archs here indeed:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kernel/process_64.c#n669

Given this is not a bug and there is no clear way to improve the situation,
I consider this thread somewhat solved (I only desired to understand
what was happening, and now I got it).

Thanks everyone!
Federico