Need help in bpf exec hook for execsnoop command

Need help in bpf exec hook for execsnoop command

[sunil hasbe]

Hello,
We are using ebpf hooks to get the process and its arguments when it
is calling exec. We are using ebpf execsnoop open source utility to
track all exec. Most of the time it works correctly, but in certain
cases (very less) it fails to get the argv[0] and argv[1]. E.g. in
below case, we are opening a new session into existing tmux session
which forks/exec a new process like this
"/usr/lib/x86_64-linux-gnu/utempter/utempter add tmux(1852218).%8".
However execsnopp is unable to get all the arguments which a userland
utility is able to get based on the cmdline for thar process. We have
used proc_connector as well to track all the processes which is able
to get the command line properly.


proc_connector process
FORK:parent(pid,tgid)=1852218,1852218   child(pid,tgid)=1935154,1935154 [tmux ]
FORK:parent(pid,tgid)=1852218,1852218   child(pid,tgid)=1935155,1935155 [tmux ]
EXEC:pid=1935154,tgid=1935154   [Uid:   0       0       0       0]      [-bash ]
EXEC:pid=1935155,tgid=1935155   [Uid:   0       0       0       0]
 [/usr/lib/x86_64-linux-gnu/utempter/utempter add tmux(1852218).%8 ]


/usr/sbin/execsnoop-bpfcc
bash             1935154 1852218   0 /bin/bash
utempter         1935155 1852218   0   tmux(1852218).%8


Upon debugging this further, we are suspecting if there is anything
related to how the parent process is forking/execing and updating its
arguments. As most of the times execsnoop is working perfectly fine
but only for few processes it fails to get the argv[0] and argv[1]. We
inspected the syscall__execve and found that argv[0], argv[1] is empty
and argv[2] is having correct value as tmux(1852218).%8.

We have seen this issue on kernel version on 5.15 on ubuntu20. Any
pointer would be very helpful on this.

Regards,
Sunil

Re: Need help in bpf exec hook for execsnoop command

[Andrii Nakryiko]

On Sun, Oct 22, 2023 at 10:12 PM sunil hasbe <sunilhasbe@gmail.com> wrote:
>
> Hello,
> We are using ebpf hooks to get the process and its arguments when it
> is calling exec. We are using ebpf execsnoop open source utility to
> track all exec. Most of the time it works correctly, but in certain
> cases (very less) it fails to get the argv[0] and argv[1]. E.g. in
> below case, we are opening a new session into existing tmux session
> which forks/exec a new process like this
> "/usr/lib/x86_64-linux-gnu/utempter/utempter add tmux(1852218).%8".
> However execsnopp is unable to get all the arguments which a userland
> utility is able to get based on the cmdline for thar process. We have
> used proc_connector as well to track all the processes which is able
> to get the command line properly.
>
>
> proc_connector process
> FORK:parent(pid,tgid)=1852218,1852218   child(pid,tgid)=1935154,1935154 [tmux ]
> FORK:parent(pid,tgid)=1852218,1852218   child(pid,tgid)=1935155,1935155 [tmux ]
> EXEC:pid=1935154,tgid=1935154   [Uid:   0       0       0       0]      [-bash ]
> EXEC:pid=1935155,tgid=1935155   [Uid:   0       0       0       0]
>  [/usr/lib/x86_64-linux-gnu/utempter/utempter add tmux(1852218).%8 ]
>
>
> /usr/sbin/execsnoop-bpfcc
> bash             1935154 1852218   0 /bin/bash
> utempter         1935155 1852218   0   tmux(1852218).%8
>
>
> Upon debugging this further, we are suspecting if there is anything
> related to how the parent process is forking/execing and updating its
> arguments. As most of the times execsnoop is working perfectly fine
> but only for few processes it fails to get the argv[0] and argv[1]. We
> inspected the syscall__execve and found that argv[0], argv[1] is empty
> and argv[2] is having correct value as tmux(1852218).%8.
>
> We have seen this issue on kernel version on 5.15 on ubuntu20. Any
> pointer would be very helpful on this.

Check what error bpf_probe_read_user() returns. If it's -EFAULT, then
it's probably the case that user memory is not physically present in
memory and needs to be paged in, which is not allowed for
non-sleepable BPF programs. So you'd need to make use of
bpf_copy_from_user() and use sleepable BPF programs.

>
> Regards,
> Sunil
>

Re: Need help in bpf exec hook for execsnoop command

[sunil hasbe]

> Check what error bpf_probe_read_user() returns. If it's -EFAULT, then
> it's probably the case that user memory is not physically present in
> memory and needs to be paged in, which is not allowed for
> non-sleepable BPF programs. So you'd need to make use of
> bpf_copy_from_user() and use sleepable BPF programs.

Hi Andrii,

We have tried using bpf_probe_read_user and it does not seem to be
returning any error, instead it returns 0. We are using a
non-sleepable bpf program.
This looks like a very special case where it is unable to fetch a few
arguments. This is the same
behaviour in opensnoop as well. We have tested the test on the 6.2
kernel as well and seeing the
same behaviour.

Do you suggest any alternative method to capture arguments in the ebpf
hooks? Or should we file
a bug in the kernel ebpf subsystem?

Re: Need help in bpf exec hook for execsnoop command

[Andrii Nakryiko]

On Fri, Nov 3, 2023 at 1:57 AM sunil hasbe <sunilhasbe@gmail.com> wrote:
>
> > Check what error bpf_probe_read_user() returns. If it's -EFAULT, then
> > it's probably the case that user memory is not physically present in
> > memory and needs to be paged in, which is not allowed for
> > non-sleepable BPF programs. So you'd need to make use of
> > bpf_copy_from_user() and use sleepable BPF programs.
>
> Hi Andrii,
>
> We have tried using bpf_probe_read_user and it does not seem to be
> returning any error, instead it returns 0. We are using a

if bpf_probe_read_user() didn't return an error, then read data should
be valid. If that data is all zeros (empty string?), then I guess env
is empty. I don't know why, you'd need to debug this, but this isn't
an BPF issue, most probably.

> non-sleepable bpf program.
> This looks like a very special case where it is unable to fetch a few
> arguments. This is the same
> behaviour in opensnoop as well. We have tested the test on the 6.2
> kernel as well and seeing the
> same behaviour.
>
> Do you suggest any alternative method to capture arguments in the ebpf
> hooks? Or should we file
> a bug in the kernel ebpf subsystem?