Error when loading BPF_CGROUP_INET_EGRESS program with bpftool

Error when loading BPF_CGROUP_INET_EGRESS program with bpftool

[Fejes Ferenc]

Greetings!

I found a strange error when I tried to load a BPF_CGROUP_INET_EGRESS
prog with bpftool. Loading the same program from C code with
bpf_prog_load_xattr works without problem.

The error message I got:
bpftool prog loadall hbm_kern.o /sys/fs/bpf/hbm type cgroup/skb
libbpf: load bpf program failed: Invalid argument
libbpf: -- BEGIN DUMP LOG ---
libbpf:
; return ALLOW_PKT | REDUCE_CW;
0: (b7) r0 = 3
1: (95) exit
At program exit the register R0 has value (0x3; 0x0) should have been
in (0x0; 0x1)
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0
peak_states 0 mark_read 0

libbpf: -- END LOG --
libbpf: failed to load program 'cgroup_skb/egress'
libbpf: failed to load object 'hbm_kern.o'
Error: failed to load object file


My environment: 5.3-rc3 / net-next master (both producing the error).
Libbpf and bpftool installed from the kernel source (cleaned and
reinstalled when I tried a new kernel). I compiled the program with
Clang 8, on Ubuntu 19.10 server image, the source:

#include <linux/bpf.h>
#include "bpf_helpers.h"

#define DROP_PKT        0
#define ALLOW_PKT       1
#define REDUCE_CW       2

SEC("cgroup_skb/egress")
int hbm(struct __sk_buff *skb)
{
        return ALLOW_PKT | REDUCE_CW;
}
char _license[] SEC("license") = "GPL";


I also tried to trace down the bug with gdb. It seems like the
section_names array in libbpf.c filled with garbage, especially the
expected_attach_type fields (in my case, this contains
BPF_CGROUP_INET_INGRESS instead of BPF_CGROUP_INET_EGRESS).

Thanks!

Re: Error when loading BPF_CGROUP_INET_EGRESS program with bpftool

[Andrii Nakryiko]

On Mon, Aug 12, 2019 at 1:59 AM Fejes Ferenc <fejes@inf.elte.hu> wrote:
>
> Greetings!
>
> I found a strange error when I tried to load a BPF_CGROUP_INET_EGRESS
> prog with bpftool. Loading the same program from C code with
> bpf_prog_load_xattr works without problem.
>
> The error message I got:
> bpftool prog loadall hbm_kern.o /sys/fs/bpf/hbm type cgroup/skb

You need "cgroup_skb/egress" instead of "cgroup/skb" (or try just
dropping it, bpftool will try to guess the type from program's section
name, which would be correct in this case).

> libbpf: load bpf program failed: Invalid argument
> libbpf: -- BEGIN DUMP LOG ---
> libbpf:
> ; return ALLOW_PKT | REDUCE_CW;
> 0: (b7) r0 = 3
> 1: (95) exit
> At program exit the register R0 has value (0x3; 0x0) should have been
> in (0x0; 0x1)
> processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0
> peak_states 0 mark_read 0
>
> libbpf: -- END LOG --
> libbpf: failed to load program 'cgroup_skb/egress'
> libbpf: failed to load object 'hbm_kern.o'
> Error: failed to load object file
>
>
> My environment: 5.3-rc3 / net-next master (both producing the error).
> Libbpf and bpftool installed from the kernel source (cleaned and
> reinstalled when I tried a new kernel). I compiled the program with
> Clang 8, on Ubuntu 19.10 server image, the source:
>
> #include <linux/bpf.h>
> #include "bpf_helpers.h"
>
> #define DROP_PKT        0
> #define ALLOW_PKT       1
> #define REDUCE_CW       2
>
> SEC("cgroup_skb/egress")
> int hbm(struct __sk_buff *skb)
> {
>         return ALLOW_PKT | REDUCE_CW;
> }
> char _license[] SEC("license") = "GPL";
>
>
> I also tried to trace down the bug with gdb. It seems like the
> section_names array in libbpf.c filled with garbage, especially the

I did the same, section_names appears to be correct, not sure what was
going on in your case. The problem is that "cgroup/skb", which you
provided on command line, overrides this section name and forces
bpftool to guess program type as BPF_CGROUP_INET_INGRESS, which
restricts return codes to just 0 or 1, while for
BPF_CGROUP_INET_EGRESS i is [0, 3].

> expected_attach_type fields (in my case, this contains
> BPF_CGROUP_INET_INGRESS instead of BPF_CGROUP_INET_EGRESS).
>
> Thanks!

Re: Error when loading BPF_CGROUP_INET_EGRESS program with bpftool

[Fejes Ferenc]

Thanks for the answer, I really appreciate it. I tried omitting
"cgroup/skb" to let libbpf guess the attach type, but I got the same
error. Really interesting, because I got the error
> libbpf: failed to load program 'cgroup_skb/egress'
wich is weird because it shows that libbpf guess the program type
correctly. So something definitely on my side - thank you for verifyng
that - I try to investigate it!

Ferenc
Andrii Nakryiko <andrii.nakryiko@gmail.com> ezt írta (időpont: 2019.
aug. 12., H, 20:27):
>
> On Mon, Aug 12, 2019 at 1:59 AM Fejes Ferenc <fejes@inf.elte.hu> wrote:
> >
> > Greetings!
> >
> > I found a strange error when I tried to load a BPF_CGROUP_INET_EGRESS
> > prog with bpftool. Loading the same program from C code with
> > bpf_prog_load_xattr works without problem.
> >
> > The error message I got:
> > bpftool prog loadall hbm_kern.o /sys/fs/bpf/hbm type cgroup/skb
>
> You need "cgroup_skb/egress" instead of "cgroup/skb" (or try just
> dropping it, bpftool will try to guess the type from program's section
> name, which would be correct in this case).
>
> > libbpf: load bpf program failed: Invalid argument
> > libbpf: -- BEGIN DUMP LOG ---
> > libbpf:
> > ; return ALLOW_PKT | REDUCE_CW;
> > 0: (b7) r0 = 3
> > 1: (95) exit
> > At program exit the register R0 has value (0x3; 0x0) should have been
> > in (0x0; 0x1)
> > processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0
> > peak_states 0 mark_read 0
> >
> > libbpf: -- END LOG --
> > libbpf: failed to load program 'cgroup_skb/egress'
> > libbpf: failed to load object 'hbm_kern.o'
> > Error: failed to load object file
> >
> >
> > My environment: 5.3-rc3 / net-next master (both producing the error).
> > Libbpf and bpftool installed from the kernel source (cleaned and
> > reinstalled when I tried a new kernel). I compiled the program with
> > Clang 8, on Ubuntu 19.10 server image, the source:
> >
> > #include <linux/bpf.h>
> > #include "bpf_helpers.h"
> >
> > #define DROP_PKT        0
> > #define ALLOW_PKT       1
> > #define REDUCE_CW       2
> >
> > SEC("cgroup_skb/egress")
> > int hbm(struct __sk_buff *skb)
> > {
> >         return ALLOW_PKT | REDUCE_CW;
> > }
> > char _license[] SEC("license") = "GPL";
> >
> >
> > I also tried to trace down the bug with gdb. It seems like the
> > section_names array in libbpf.c filled with garbage, especially the
>
> I did the same, section_names appears to be correct, not sure what was
> going on in your case. The problem is that "cgroup/skb", which you
> provided on command line, overrides this section name and forces
> bpftool to guess program type as BPF_CGROUP_INET_INGRESS, which
> restricts return codes to just 0 or 1, while for
> BPF_CGROUP_INET_EGRESS i is [0, 3].
>
> > expected_attach_type fields (in my case, this contains
> > BPF_CGROUP_INET_INGRESS instead of BPF_CGROUP_INET_EGRESS).
> >
> > Thanks!

Re: Error when loading BPF_CGROUP_INET_EGRESS program with bpftool

[Andrii Nakryiko]

On Mon, Aug 12, 2019 at 1:48 PM Fejes Ferenc <fejes@inf.elte.hu> wrote:
>
> Thanks for the answer, I really appreciate it. I tried omitting

Please reply inline, no top posting on kernel mailing lists.

> "cgroup/skb" to let libbpf guess the attach type, but I got the same
> error. Really interesting, because I got the error
> > libbpf: failed to load program 'cgroup_skb/egress'
> wich is weird because it shows that libbpf guess the program type
> correctly. So something definitely on my side - thank you for verifyng
> that - I try to investigate it!

What was the error message you got after you provided correct program
attach type?

>
> Ferenc
> Andrii Nakryiko <andrii.nakryiko@gmail.com> ezt írta (időpont: 2019.
> aug. 12., H, 20:27):
> >
> > On Mon, Aug 12, 2019 at 1:59 AM Fejes Ferenc <fejes@inf.elte.hu> wrote:
> > >
> > > Greetings!
> > >
> > > I found a strange error when I tried to load a BPF_CGROUP_INET_EGRESS
> > > prog with bpftool. Loading the same program from C code with
> > > bpf_prog_load_xattr works without problem.
> > >
> > > The error message I got:
> > > bpftool prog loadall hbm_kern.o /sys/fs/bpf/hbm type cgroup/skb
> >
> > You need "cgroup_skb/egress" instead of "cgroup/skb" (or try just
> > dropping it, bpftool will try to guess the type from program's section
> > name, which would be correct in this case).
> >
> > > libbpf: load bpf program failed: Invalid argument
> > > libbpf: -- BEGIN DUMP LOG ---
> > > libbpf:
> > > ; return ALLOW_PKT | REDUCE_CW;
> > > 0: (b7) r0 = 3
> > > 1: (95) exit
> > > At program exit the register R0 has value (0x3; 0x0) should have been
> > > in (0x0; 0x1)
> > > processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0
> > > peak_states 0 mark_read 0
> > >
> > > libbpf: -- END LOG --
> > > libbpf: failed to load program 'cgroup_skb/egress'
> > > libbpf: failed to load object 'hbm_kern.o'
> > > Error: failed to load object file
> > >
> > >
> > > My environment: 5.3-rc3 / net-next master (both producing the error).
> > > Libbpf and bpftool installed from the kernel source (cleaned and
> > > reinstalled when I tried a new kernel). I compiled the program with
> > > Clang 8, on Ubuntu 19.10 server image, the source:
> > >
> > > #include <linux/bpf.h>
> > > #include "bpf_helpers.h"
> > >
> > > #define DROP_PKT        0
> > > #define ALLOW_PKT       1
> > > #define REDUCE_CW       2
> > >
> > > SEC("cgroup_skb/egress")
> > > int hbm(struct __sk_buff *skb)
> > > {
> > >         return ALLOW_PKT | REDUCE_CW;
> > > }
> > > char _license[] SEC("license") = "GPL";
> > >
> > >
> > > I also tried to trace down the bug with gdb. It seems like the
> > > section_names array in libbpf.c filled with garbage, especially the
> >
> > I did the same, section_names appears to be correct, not sure what was
> > going on in your case. The problem is that "cgroup/skb", which you
> > provided on command line, overrides this section name and forces
> > bpftool to guess program type as BPF_CGROUP_INET_INGRESS, which
> > restricts return codes to just 0 or 1, while for
> > BPF_CGROUP_INET_EGRESS i is [0, 3].
> >
> > > expected_attach_type fields (in my case, this contains
> > > BPF_CGROUP_INET_INGRESS instead of BPF_CGROUP_INET_EGRESS).
> > >
> > > Thanks!

Re: Error when loading BPF_CGROUP_INET_EGRESS program with bpftool

[Fejes Ferenc]

Sorry for the late reply. I have some time to try it again.
I downloaded a fresh Ubuntu 19.10 image, which have bpftool included
from the linux-tools-generic package. Then tried to load a program
with 0 or 1 return code: both works.
After that I tried to load the original program, with return code 2 or
3 and got the same error message.

> What was the error message you got after you provided correct program
> attach type?

sudo bpftool prog loadall hbm.o /sys/fs/bpf/hbm2 type cgroup_skb/egress

libbpf: load bpf program failed: Invalid argument
libbpf: -- BEGIN DUMP LOG ---
libbpf:
; return ALLOW_PKT | REDUCE_CW;
0: (b7) r0 = 3
1: (95) exit
At program exit the register R0 has value (0x3; 0x0) should have been
in (0x0; 0x1)
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0
peak_states 0 mark_read 0

libbpf: -- END LOG --
libbpf: failed to load program 'cgroup_skb/egress'
libbpf: failed to load object 'hbm.o'
Error: failed to load object file

Same command with strace:
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SKB, insn_cnt=2,
insns=0x558889b59ba0, license="GPL", log_level=0, log_size=0,
log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
prog_name="hbm", prog_ifindex=0,
expected_attach_type=BPF_CGROUP_INET_INGRESS, ...}, 112) = -1 EINVAL
(Invalid argument)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SKB, insn_cnt=2,
insns=0x558889b59ba0, license="GPL", log_level=1, log_size=16777215,
log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
prog_name="hbm", prog_ifindex=0,
expected_attach_type=BPF_CGROUP_INET_INGRESS, ...}, 112) = -1 EINVAL
(Invalid argument)

Strace error is the same, even if I manually set the type to cgroup_skb/egress:
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SKB, insn_cnt=2,
insns=0x55ae46296ba0, license="GPL", log_level=0, log_size=0,
log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
prog_name="hbm", prog_ifindex=0,
expected_attach_type=BPF_CGROUP_INET_INGRESS, ...}, 112) = -1 EINVAL
(Invalid argument)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SKB, insn_cnt=2,
insns=0x55ae46296ba0, license="GPL", log_level=1, log_size=16777215,
log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
prog_name="hbm", prog_ifindex=0,
expected_attach_type=BPF_CGROUP_INET_INGRESS, ...}, 112) = -1 EINVAL
(Invalid argument)

I noticed you did a major restructuring in libbpf, I will try out
whether my problem still exist with the new version.

Ferenc